KDCAflyer

I'm friends with the former Lt. Gov. of VA on FB, and he's caught up in this too. Lost two days of his vacation, and had to buy walkup fares on another airline.

That's the third person I know personally who has gotten stuck in this.

BenA

This actually isn't true at all; resource scheduling for an airline can be oversimplified as a really difficult version of the knapsack problem (basically, selecting a set of items with given weights to maximize the overall benefit).

Computer scientists study algorithms to determine their "computational complexity" - basically, is there a solution that scales linearly or exponentially as the problem size gets bigger. The knapsack problem is well known to be NP (non polynomial) hard to begin with; when you scale it up to the size of an airline's operations the problem is completely intractable to solve "correctly".

You have to start making approximations and optimizations to get it done in any reasonable amount of time, and even then you'll be spending a ton of processor time to figure out a solution that is "pretty good". You can basically pour in as much time as you've got, and the quality and creativity of the software design greatly influences how accurate the results are.

Delta probably precomputes the schedules a long time in advance under normal circumstances and then only has to recalculate late breaking changes. When the whole thing is broken (and changing by the minute), the system may never be catching up because it can't spend enough CPU time to find a solution.

If I'm guessing correctly, the fix is to go to a cloud provider where extra capacity can be ordered on demand in an emergency. But that's a huge undertaking...

spongenotbob

If my sewage plant can't handle the load during a heavy storm and overflows and spews sewage into the streets, and it keeps doing so for the next 6 days, did it crash? Or was it just "taking much longer than normal to process"?

Justin026

I went and looked through the thread on the airline employee forum that we have been linked to from this thread. It has about 100 posts on it, but a lot of it is an echo chamber.

I see two early posts where employees state they were firmly in line to take assignments and the system simply didn't use them. I guess that would/could be IT. In the later half of the thread there is a discussion where two employees wonder if IT is down when they try to log on. Others say the employee apps are working, an employee tells them how to use the software differently and that seems to solve some, but not all, of their problems.

There is one complaint about the phones. Others say phones are working.

About a half-dozen posts are reasonably specific reports about flights being delayed because the right status, especially of FAs, didn't match up at the time of departure. Slow IT?

There are dozens of generalized posts about what a mess this is.

There are dozens of posts about manpower being stretched too thin, work rules, company politics, union stuff and staff volunteers helping out.

There are dozens of posts that are funny/ironic, etc.

arlflyer

Well done; I love a good sewage analogy.

We engineers will jump all over it and dive down to root causes and semantics, but the end result is that it done broke.

In any case I'm pretty sure we can agree that it isn't "weather" at this point.

Saj8986

I saw one post that said....."loyalty is earned in drops but lost in buckets"

We'll see.

Sc3148

From what I saw last night flying from ATL to MCO it was defiantly a IT issue. Gates A1, A3 & A7 were all scheduled to go to MCO. Delays ranged between 6 and 2 hours per plane. A7 had pilots but no FAs, A3 was missing 1 pilot and A1 was missing 2 FAs. The crews were trying to save at least 2 flights by consolidating crews. The crews were talking about their Skypro app being down so they could not make crew schedule changes or get updates and assignments. I finally made it out when 2 FAs on DL 2018 that were dead heading to MCO volunteered to get back in uniform and work the flight so they had a full crew.

Widgets

At ~160 DL+DC cancellations today, it hasn't changed much in the last several hours. Whoever recently mentioned that the OCC is predicting no more than 200 today looks about right at the moment.

Edit to re-word: 161 cancellations today as of 1:55pm EDT. "Only" 15 new cancellations in the past 3 hours. Still bad, but improving at an increasing rate. I still think ops can be back to normal by tomorrow afternoon.

BenA

No, local officials would call it a "disaster"; reference this recent West Seattle plant failure: http://www.cbsnews.com/news/seattle-sewage-treatment-plant-electrical-failure-pugent-sound/

um... You Can't Stop Seattle(tm)?

flyerslc

What is OCC?

sleuth

The 30 year old system didn't crash in August when the pretty new SNAPP system went down.

Widgets

OCC is the Operations & Customer Center (formerly the Operations/Control Center) at the General Offices in Atlanta. It's the heart of Delta's operations where flight dispatchers, meteorologists, etc. coordinate.

For a glimpse, here's Richard Anderson's old OCC intro for the onboard safety videos:

NoleATL

Not correct. I worked at the DAT up until about 7 years ago. Even then I can't think of any running these types of systems that were 30 years old. I'm not saying they don't have ANY but I can unequivocally say it is not true that "Many of DL's are 20-30 years old.

dinanm3atl

Which is still a Delta problem. You can't say 'well the weather was bad so our system couldn't keep up'. That means your system failed to deliver what it was supposed to do.


Very much so.


Hah!

chrislevin

I hope, at least, that delta's software and database tracking FAs, pilots is not the same one that manage their customer bookings.