FlyerTalk Forums

FlyerTalk Forums (https://www.flyertalk.com/forum/index.php)
-   Delta Air Lines | SkyMiles (https://www.flyertalk.com/forum/delta-air-lines-skymiles-665/)
-   -   Delta computers down (https://www.flyertalk.com/forum/delta-air-lines-skymiles/1783354-delta-computers-down.html)

dinanm3atl Aug 8, 2016 3:45 pm


Originally Posted by KissFlyGoodbye (Post 27034600)
Here ya go.

http://news.delta.com/delta-offers-c...temwide-outage

It really isn't much to be honest.

Does seem, stingy. Guess offer 200 and most will jump on it(or not even find out about it).

I was booked on 1230 flight out of MKE. It was delayed ~1hr. Went to SkyClub and waved moved(still in FC) to the flight at 950am. It ended up 'departing' at 315. We didn't move until about 345. We were scheduled for 220pm but sat for over an hour at the gate because of ATC in Atlanta the pilot claims. Makes sense with everything that happened.

Now in the air and the GoGo actually works "OK". Surprise!

Widgets Aug 8, 2016 3:46 pm


Originally Posted by yohanson (Post 27034738)
My Compass flight later tonight just landed at OKC 3 minutes early and is expected to take off for MSP a minute early. I guess I should be thankful not to be flying mainline this week.

This is the most surreal thought for me in this situation. Delta Connection has always been the lagging part of the operation... until today!

GRALISTAIR Aug 8, 2016 3:49 pm

Landed in ATL 5 minutes early. I used to ..... that DL gave the MAN - ATL flight to VS and then codeshare - not anymore. Of course no chance of going anytime soon ATL- DAY

fliesdelta Aug 8, 2016 3:49 pm


Originally Posted by bennos (Post 27034657)
As yet another IT person, I've enjoyed reading the conjecture in this thread, especially from a couple of seemingly particularly well-informed posters. The posts I enjoy most, however, (excluding "how can I scam something out of this?") are the ones from posters asserting that in their IT world nothing like this could ever happen. HA, BC, and DR are, like security, about risk reduction not risk elimination. No matter how impressive your system redundancy is, I can find a way (however unlikely) to defeat it.

Should someone get fired for this outage? Maybe, or maybe that person already quit years ago, or maybe that consulting firm finished the contract and left, or maybe it was a fluke.

The most enlightening posts might be those from MSP_Dave, which if true suggest DL may over time be heading the way of UA under SMI/J, and that it may be time to start thinking about a new primary airline...

I was also amused by the calls for heads rolling by self appointed IT experts.

I think I'll wait to hear more, if anything else is forthcoming from Delta. But also, keep in mind that if someone did screw up, that person probably learned a very important and valuable lesson and is unlikely to screw up like that again. Just firing that person and hiring someone new just means you will get a person who will make new mistakes.

CKDGM Aug 8, 2016 3:50 pm


Originally Posted by Zorak (Post 27034684)
Sure, this is still a process failure on DL's part, and embarrassing for them, but I do have some degree of sympathy for how it could have happened.

Agreed. Having supported an airline passenger service system myself for a few years (and currently part of a 24x7 support organization for non-airline systems), there's definitely more than a little bit of "there but for the grace of God" in my reaction to this. I just wish there'd be a published analysis of what went wrong and what's being done to prevent it in the future, the way some cloud providers have done it.

RustyC Aug 8, 2016 3:52 pm

NBCNews.com did a story about what to do if you're caught up in the mess

GRALISTAIR Aug 8, 2016 4:03 pm


Originally Posted by MasterGeek (Post 27033664)
It's payback time for Delta! Gotta pay all those hotel rooms, vouchers and $600 EU compensations. Karma is a biatch :rolleyes:

That will teach to Delta to be more transparent regarding award availability and pricing.


Originally Posted by Widgets (Post 27033678)
Of all the things Delta will look at following this disaster, unfortunately I doubt award chart transparency is anywhere on the list.

Agreed - silliest non-sequitur post of the year

teCh0010 Aug 8, 2016 4:29 pm


Originally Posted by RustyC (Post 27034751)
It's a bit unsettling to know there are vulnerabilities that could lead to the entire system basically shutting down like that, as you have to worry not only about accidents but also malice. Isn't the idea with redundancy to have data stored in several places geographically so that something like a fire or power outage in one place can't take the whole thing down? I dunno how realistic a proposition that is for an airline, but you hear about it for large Web-based businesses. Bastian should have some 'splainin' to do to investors over why there's no back-up plan.

Having multiple copies of the data replicated to multiple data centers isn't the problem. It's very easy to maintain the integrity of the data with replication, the problem is leveraging the data to make the system available.

If data center A burns to the ground it's an easy call to declare an disaster and begin recovering at data center B. If data center A is still there, there is just an issue with the ATS / switch gear, then it becomes a tougher decision. How long will it take to change IPs, DNS, start the application up at the DR site, confirm data consistency, test the application and make it available? Is that longer that it will take to just get the broken DC running?

Applications that just keep running in the event of a data center going offline are normally what would be considered cloud native, they create availability at the application layer not at the infrastructure layer. That works great for many types of business, but the types of transactions done by airlines and banking don't work well with that type of application architecture. So you need to build large monolithic applications due to the nature and volume of the business transaction. These applications can't create availability at the application layer, it must be provided by a very redundant infrastructure layer.

jdrtravel Aug 8, 2016 4:37 pm

Did you see the video apology from Bastian? Also interesting that they seem to be watching CNN or FOX or similar in the background...

http://news.delta.com/ceo-apologizes...very-continues

bennos Aug 8, 2016 4:49 pm


Originally Posted by teCh0010 (Post 27035007)
It's very easy to maintain the integrity of the data with replication

Unless your application is buggy, then you get replicated bad data :)

(But I do generally agree with most everything else you've posted in this thread.)

tennislover9 Aug 8, 2016 5:04 pm

Logistically, how do mass glitches like this ever get sorted our properly?

Won't the ramifications of re-booking and schedule changes take days to fix?

EditingFX Aug 8, 2016 5:05 pm

OT, but all this datacenter talk reminded me of a video shoot I did at a VERY large DC. During the pre-shoot, on-site briefing with the DC's architect, when asked if we had any questions before starting, I couldn't resist...
me - "So I couldn't tell when we entered, but are all the doors air-tight?"
architect - "Airtight? No, but they high-efficiency insulated, does that answer your question?"
me - "Well, no... I was wondering how you're going to keep the Cloud in."

(cymbal hit, please)

nevansm Aug 8, 2016 5:08 pm


Originally Posted by Zorak (Post 27034684)
Yeah, I've been mildly amused by the amount of forthright "this would never happen to me" stated or implied. It does seem likely they or their providers/vendors had large gaps in their continuity plan. But the defects are always easy to identify and pick apart in hindsight, and every business makes tradeoffs.

Sure, this is still a process failure on DL's part, and embarrassing for them, but I do have some degree of sympathy for how it could have happened.

In my posts that's all I've been implying is that this seems like a bad decision on delta's part. The way I read it, they don't have a secondary site, which to me is the issue (and the one I've brought up).

I don't think I ever said something like this doesn't happen or couldn't happen with the products I work with. But I sure as heck would have a better plan (or make people think I did at the least).

Rabidstoat Aug 8, 2016 5:12 pm

Ugh, I'm on a business trip and was delayed four hours, so I should be getting a $200 voucher. Since it's a business trip I'll need to use it on a future business trip under the same contract, which I'll surely have. But, we're a huge corporation and our travel book and expense system is so convoluted, I will never, ever figure out how to claim the voucher, let alone how to explain this all on my electronic expense report.

appleguru Aug 8, 2016 5:16 pm


Originally Posted by Rabidstoat (Post 27035197)
Ugh, I'm on a business trip and was delayed four hours, so I should be getting a $200 voucher. Since it's a business trip I'll need to use it on a future business trip under the same contract, which I'll surely have. But, we're a huge corporation and our travel book and expense system is so convoluted, I will never, ever figure out how to claim the voucher, let alone how to explain this all on my electronic expense report.

Why would the voucher end up on your expense report, and why would you need to use it for a future buisiness trip instead of a leisure one? It's not a refund... It's a customer service offering for your hassle.


All times are GMT -6. The time now is 12:31 pm.


This site is owned, operated, and maintained by MH Sub I, LLC dba Internet Brands. Copyright © 2026 MH Sub I, LLC dba Internet Brands. All rights reserved. Designated trademarks are the property of their respective owners.