Community
Wiki Posts
Search
Old Aug 9, 2016, 1:04 pm
FlyerTalk Forums Expert How-Tos and Guides
Last edit by: TheLifeOfA_NKCM
As most of us ...well, all of us, know, there was a power failure at ground ops in ATL on Monday (2016-08-08) at 2:39AM which lead to mass-cancellations and delays and rebookings spanning as far as next Sunday (2016-08-21) as of Aug 9. Passengers didn't hesitate to display their dissapointment, Twitter, ACS, and DL phone lines were ambushed with customers. In an effort to keep our fellow flyers informed on the incoming changes and information, this Wiki was formed.

You may all feel free to edit it as your leisure as we continue to watch this develop..

If you are currently at an airport right now, your information on the ongoing events and current situation there is equally as valuable. Thanks for your input.

If you have part of an article in your post, please cite it with a URL to it's
origin.

Airport & Support Situation.
.
NRT: Foam or Blowup Pad, Crackers, and water given out to customers for sleeping prior to hotel voucher last night
MSP: "Security Lines best since reconfiguration"
---
Note:
Some passengers are reporting long waiting queues on DL phone lines. If possible avoid contacting DL CSR by phone (DM and medallion lines are also expected to have a notably longer wait as well) and instead redirect att'n to Twitter which has been said to be notably quicker than other means of assistance, including Need Help Centers and Gate Agents. Fly Delta and Delta.com should also be permitting rebooking on to other flights at this time. If you are unable to take advantage of any of these methods, express add'l patience

Delta Related Updates:
---
August 11th, 2016 // 11:00AM ET - Flights OPs Return to Normal

Flight operations have returned to normal following Mondays outage, airline officials said at noon today. Delta started the day with roughly 25 mainline flights canceledsome due to bad weather in the Midwestand no additional cancellations are anticipated.

The airline is back on its feet, moving past this weeks disruption and operating normally, said Dave Holtz, S.V.P. - Operations and Customer Center. Running a reliable operation is something we pride ourselves on and frankly, that didnt happen this week. But our teams have worked tirelessly to bring back the dependable airline our customers have known us to be and have come to expect from us.

SLC.jpgWith the impact of the power loss is in the rearview mirror, the airline is carefully monitoring the development of a weather system that may hit the eastern third of the U.S. later today, with the possibility of air traffic control-related delays possible in the Northeast. No additional cancellations as a result of weather are expected, however.

The airline plans to fly more than 610,000 people today. By noon Thursday, nearly 2,200 flights had departed with more than 85 percent departing within 15 minutes of the scheduled departure time.

Even so, customers traveling today should check the status of their flight at delta.com or the Fly Delta App. Delta's travel waiver has been extended through Thursday.

Monday morning an uninterrupted power source switch experienced a small fire which resulted in a massive failure at Deltas Technology Command Center. This caused the power control module to malfunction, sending a surge to a transformer outside of Delta, resulting in the loss of power. The power was stabilized and power was restored quickly. But when this happened, critical systems and network equipment didnt switch over to backups. Around 300 of about 7,000 data center components were discovered to not have been configured appropriately to avail back-up power. In addition to restoring Deltas systems to normal operations, Delta teams this week have been working to ensure reliable redundancies of electrical power as well as network connectivity and applications are in place.
---
August 10th, 2016
Travel waiver impacted dates now extended to include August 11th
https://www.delta.com/content/www/en...em-outage.html
---
4:30PM ET // August 10th, 2016 - Delta has canceled 317 flights today, none in the last hour. Of 3,100 flights that have departed during the day, about 2/3 left within 30 minutes of their scheduled departure time.

Delta's travel waiver has been extended. The airline has extended its offer of compensation to customers significantly affected by lengthy delays or cancellations. Many customers have already been contacted by Delta.

Delta CEO Bastian: 'This isn't who we are'
---
2:30PM ET // August 10th, 2016 - Delta has canceled 311 flights today. About 2,540 flights have departed, 70 percent of those within 30 minutes of the scheduled departure time. Delta operates nearly 6,000 flights a day this time of year.

Delta's travel waiver has been extended. The airline has extended its offer of compensation to customers significantly affected by lengthy delays or cancellations. Many customers have already been contacted by Delta.
---
12:30 PM ET // August 10th, 2016 - Delta has added a few more cancellations, bringing the number today to 278. About 1,950 flights have departed, 75 percent of those within 30 minutes of scheduled departure time. Delta operates nearly 6,000 flights a day this time of year.

Delta also extended its offer of compensation to customers significantly affected by delays or lengthy cancellations. Many customers have already been contacted by Delta.
---
11:00AM EDT // August 10th, 2016 - Delta officials anticipate returning to a normal operation by afternoon Wednesday following an outage Monday that disrupted the flight schedule, though scattered thunderstorms expected in the eastern U.S. may slow the recovery.

By late morning, Delta had canceled 255 flights while 1,500 flights departed. About 80 percent of those flights departed within 30 minutes of their scheduled time.

Technology systems that allow airport customer service agents to process check-ins, conduct boarding and dispatch aircraft are functioning normally with the bulk of delays and cancellations coming as a result of flight crews displaced or running up against their maximum allowed duty period following the outage.

Were in the final hours of bouncing back from the disruption, said Bill Lentsch, Deltas Senior Vice President Airport Customer Service and Airline Operations. Delta employees have been working around the clock and are committed to bringing the airline back to full strength. We know this has been a rough couple of days for our customers and apologize to those who have experienced our less-than-stellar operation.

Leaders and employees continue to focus on promptly moving departures at the airline's Atlanta hub, the world's busiest. Delta sent reservations personnel from the corporate headquarters in Atlanta to help customer service agents process customers at Hartsfield-Jackson Atlanta International Airport. A cadre of additional employees were also positioned throughout the Atlanta airport to help customers get to their flights.

Customers traveling today should check the status of their flight at delta.com or the Fly Delta App. Customers can rebook their flight via the website.

Delta's travel waiver has been extended and the airline continued to provide hotel vouchers to several thousand customers, including more than 2,300 Tuesday night in Atlanta. Delta extended its offer of compensation to customers significantly affected by delays or cancellations. Many customers have already been contacted by Delta.
---
7:00AM ET // August, 10th, 2016 - Delta officials anticipate returning to a normal operation by mid-to-late afternoon Wednesday following an outage Monday that disrupted the airlines flight schedule, though a chance of scattered thunderstorms expected in the eastern U.S. may have the potential to slow the recovery.

Teams across the airline continued work during the overnight hours to bring an end to the impact of Mondays outage that disrupted operations. Delta will start the Wednesday operation with a little over 150 system-wide cancellations after roughly 800 flights were canceled Tuesday.

Technology systems that allow airport customer service agents to process check-ins, conduct boarding and dispatch aircraft are functioning normally with the bulk of delays and cancellations coming as a result of flight crews displaced or running up against their maximum allowed duty period following the outage.

Were in the final hours of bouncing back from the disruption, said Bill Lentsch, Deltas Senior Vice President Airport Customer Service and Airline Operations. Delta employees have been working around the clock and are committed to bringing the airline back to full strength. We know this has been a rough couple of days for our customers and apologize to those who have experienced our less-than-stellar operation.

Delta CEO Bastian: 'This isn't who we are'

Leaders and employees continue to focus on promptly moving departures at the airline's Atlanta hub, the world's busiest. Delta sent reservations personnel from the corporate headquarters in Atlanta to help customer service agents process customers at Hartsfield-Jackson Atlanta International Airport. A cadre of additional employees were also positioned throughout the Atlanta Airport to help customers get to their flights.

Customers traveling today should check the status of their flight at delta.com or the Fly Delta App. Customers can rebook their flight via the website.

Delta's travel waiver has been extended and the airline continued to provide hotel vouchers to several thousand customers, including more than 2,300 Tuesday night in Atlanta. Through noon ET, Delta will extend its offer of compensation to customers significantly affected by delays or cancellations to cover Tuesday. Many customers have already been contacted by Delta agents.

Unaccompanied minors will again be allowed to travel on Delta Wednesday following a temporary stay during the height of the disruption.
---
August 9th, 2016 // 6:00 PM EST - Rebooking, Refunds, and Compensation: What Customers Need to Know:

Delta people worldwide are working around the clock to take care of customers affected by the disruption of airline operations. Below are updates to assist customers with questions regarding rebooking a flight, the current travel waiver and compensation Delta is offering affected customers:

Q: What does Delta's travel waiver offer customers?

If a customer's flight is cancelled or significantly delayed, he or she entitled to a refund. Even if the flight is not canceled, a one-time change to the ticket may be made without a fee. See full details here.

Q. Has Delta's travel waiver been extended?

Yes, to offer additional flexibility to customers who wish to make changes to their travel plans, Delta's travel waiver has been extended. For the latest information, see full details here.

Q. Is Delta offering travel vouchers to impacted customers?

Yes. Delta will provide $200 in travel vouchers to all customers who experienced a delay of greater than three hours or a cancelled flight as a result of the impact to operations systemwide. The vouchers are available for travel on all Delta and Delta Connection-operated flights.

Customers affected by the outage and who have provided their email address will be contacted via email and will receive their voucher in seven to 10 business days. Customers who have not been contacted can complete a Customer Care form on www.delta.com/wecare to receive their voucher. All travel must be booked by one year from the date the voucher was issued.

Q. Do the travel vouchers also apply to customers who are entitled to a refund?

Yes. Delta will provide $200 in travel vouchers to all customers who experienced a delay of greater than three hours or a cancelled flight as a result of the impact to operations systemwide. ​
---
August 9th, 2016 - Delta CEO Ed Bastian Releases Another Statement
Today Ed released another statement regarding the ongoing system reset and outage. Claiming "This isn't who we are". It can be found here.
---
Nonrev Travel Discouraged; Buddy Passes embargoed
Delta employees and family with nonrev flight privileges are discouraged at this time from making any personal travel until further notice. Additionally, during this time travelling via the use of a buddy pass will be embargoed. An end date to this travel suspension has not been decided upon and will be released in future updates.
---
Travel Waiver in effect:
All Delta flights
August 8-10, 2016
Ticket must be Reissued On or Before, and Rebooked Travel must begin no later than:
August 21, 2016

Note: This also has been extended to include hotel rooms and other accommodations for those customers who may need to take an add'l night stay due to a flight cancellation.
---
August 9th, 2016 // 11:00 PM EST -
Delta canceled about 775 flights Tuesday as the airline worked to establish normal operations after Mondays outage that disrupted the flight schedule. The outlook for Wednesdays operation is positive, Delta officials said, with about 90 cancellations expected at the beginning of the day leading to normal operations later.

Delta continues to recover from events earlier this week and expect to be back to normal operations by mid-day Wed, with some cancellations in the morning. -summarized; news.delta.com
---
August 9th, 2016 // 2:00 PM EST - Cause of Outage Identified:
"a critical power control module at our Technology Command Center malfunctioned, causing a surge to the transformer and a loss of power. The universal power was stabilized and power was restored quickly. But when this happened, critical systems and network equipment didn't switch over to backups. Other syste​ms did. And now we're seeing instability in these systems."

Power was restored quickly however, due to some critical systems and network equipment didn't switch over to the backups leading to the outage; DL still in recovery. ACS systems are still catching up and have been reported to be running slow -summarized; news.delta.com
---
August 9th, 2016 // 12:30 PM EST - Delta has canceled about 530 flights today while more than 1,600 have departed as the airline works diligently to normalize operations after Monday's outage that disrupted the flight schedule. Leaders and employees continue to focus on promptly moving departures at the airline's Atlanta hub, the world's busiest.

Customers traveling today should check the status of their flight at delta.com or the Fly Delta App. Customers can rebook their flight via the website.

Delta's travel waiver has been extended to customers traveling on Aug. 9. Also, unaccompanied minors that have not yet begun travel will not be accepted until Wednesday. Those customers will be able to book without fee at a later date.

Delta offered compensation to customers significantly affected by delays or cancellations.​ -news.delta.com
---
August 9th, 2016 // 8:30 AM EST - What Customers Need to Know for Tuesday:
​​​Delta is cancelling more than 300 flights Tuesday morning as it works to reset the operation and get crews, aircraft and other operational elements in place to take care of customers. The airline experienced a loss of power Monday morning that impacted operations systemwide.

​Here are some things to know today:

1. Customers traveling Tuesday should check the status of their flight at delta.com or the Fly Delta App. Customers can rebook their flight via the website.

2. Delays continue. Customers heading to the airport should expect delays and cancellations. While inquiries are high and wait times are long, our customer service agents are doing everything they can to assist.

3. Delta's travel waiver has been extended to customers traveling on Tuesday.

4. Delta extended its offer of compensation to customers significantly affected by delays or cancellations to cover today.​ ​

5​. Unaccompanied minors that have not yet begun travel will not be accepted until Wednesday. Those customers will be able to book without fee at a later date.-news.delta.com
Print Wikipost

Delta computers down

Thread Tools
 
Search this Thread
 
Old Aug 9, 2016 | 11:22 am
  #586  
1M
40 Countries Visited
100 Nights
15 Years on Site
 
Join Date: Feb 2007
Location: Pensacola, FL (PNS)
Programs: Delta 1.99M, Hyatt Globalist, IHG Diamond, Marriott Gold, Hilton Diamond
Posts: 2,728
Originally Posted by Zorak
UPS systems and generators fail. In the 2000s probably the worst 2-3 days of my career happened when our colo provider lost power in their data center. They had redundant UPSes and a backup generator but something went wrong in the cutover and their entire facility lost power. We were a tiny company and did not have a lot of protection against this sort of failure -- that part is certainly our fault -- but the fact is that something that you wouldn't expect to happen, happened.

Granted, Delta is not a tiny company and has infinitely more resources to throw at the problem than we did. But the takeaway is that even redundant failover, sometimes fails.
This is why you have a plan, test, test again, and test over again.

If it was a hack or breach - and not saying it was - the new CIO and possibly Bastian would be out of their job(s).
lindros2 is offline  
Old Aug 9, 2016 | 11:23 am
  #587  
FlyerTalk Evangelist
Community Builder
Conversation Starter
All eyes on you!
15 Years on Site
 
Join Date: Oct 2009
Location: ATL Lost Luggage
Programs: Kettle with Kryptonium Medallion Tags
Posts: 12,676
Some coverage in the local ATL newspaper:
Newlyweds Nadia and Nathan Odiorne spent the first hours of their honeymoon stuck in airports following a power outage that caused Delta Air Lines to cancel about 1,000 flights Monday.

Well, their travel horrors continued Tuesday morning at Hartsfield-Jackson International Airport in the face of nearly 300 additional Delta flights canceled.

Passengers such as the Odiornes were stuck in winding, hours-long lines.
Link to the full article: AJC.com: Honeymoon-bound newlyweds stuck at airport (12:03 p.m. Tuesday, Aug. 9, 2016)

and another article:
Its really frustrating, said Tatjana Drachenberg, a traveler from Germany whose flight to Atlanta was delayed five hours, causing her to miss her connecting flight to Charleston. She arrived Monday evening at Hartsfield-Jackson International Airport and pretty much slept on the floor, got hardly any sleep.

Three subsequent flights to Charleston were canceled, she said, and it was unclear whether the next flight she was rebooked on would be canceled as well.

Its just really chaos. No one can tell you anything, Drachenberg said.

In the congested concourses overnight, she said, where thousands of travelers crowded into gate areas and walkways and stood in long lines trying to get information about their flights, they had a really long wait, just for a simple question like, What do I do next? On the concourse, you have no place to sit. There were a lot of people sitting on the floors and sleeping on the floors, Drachenberg said.
Link to the full article: AJC.com: Delta cancels 530 flights, travelers describe chaos on concourses (6:41 a.m. Tuesday, Aug. 9, 2016)
RatherBeOnATrain is offline  
Old Aug 9, 2016 | 11:26 am
  #588  
Community Builder
Community Influencer
All eyes on you!
10 Years on Site
 
Join Date: Feb 2013
Programs: Hilton Diamond, Delta Gold
Posts: 5,924
On a more serious note, as someone flying tomorrow afternoon connecting through ATL, I have been noticing that just because it doesn't say Cancelled next to a flight number does not necessarily mean its not going to eventually be cancelled.
Miesque is offline  
Old Aug 9, 2016 | 11:26 am
  #589  
Suspended
 
Join Date: Aug 2005
Location: BOS
Posts: 15,027
And why do airlines still operate in this DUMB way? Why must be there ONE centralized system?
Have at least two or three nodes across the country that kick in when the main node is down. Then, i.e. a plane in Tokyo does not have to wait for the stone-aged system in Atlanta to turn back on, as the node in Tokyo would just take over.
Dieuwer is offline  
Old Aug 9, 2016 | 11:29 am
  #590  
5M
100 Countries Visited
All eyes on you!
20 Years on Site
 
Join Date: Apr 2005
Location: New York, western US
Programs: DM/3MM
Posts: 4,315
Originally Posted by jackal
Except that in this case, they won't. As long as you have a boarding pass for that day at that airport, TSA policy is to allow you access to any checkpoint at that airport. At best, they may advise you that you're not at the correct checkpoint, but they will not deny you access.

This has been confirmed to be official policy by several TSA employees here on FT as well as a more official source (the TSA blog, IIRC).
So it's not like CDG, where you can only access the concourse from which your flight departs, but you can go through security at any terminal at the airport from which you are departing?
TrojanTraveler is offline  
Old Aug 9, 2016 | 11:30 am
  #591  
FlyerTalk Evangelist
Community Builder
Conversation Starter
All eyes on you!
15 Years on Site
 
Join Date: Oct 2009
Location: ATL Lost Luggage
Programs: Kettle with Kryptonium Medallion Tags
Posts: 12,676
Originally Posted by skyrider117
On the slightly ironic side of things..
I just popped over to Deltajobs.com and the following is posted...

Operations Manager IT - Crisis Management
The last line --"Must be performing satisfactorily in current position"-- would seem to rule out internal hires...
RatherBeOnATrain is offline  
Old Aug 9, 2016 | 11:31 am
  #592  
 
Join Date: Jan 2015
Programs: delta dm
Posts: 1,677
Originally Posted by RatherBeOnATrain
Yes delta has a bunch of clowns running the circus but ever heard of a car and driving.
dzflyer is offline  
Old Aug 9, 2016 | 11:32 am
  #593  
FlyerTalk Evangelist
50 Countries Visited
5M
All eyes on you!
15 Years on Site
 
Join Date: Nov 2007
Location: Denver DEN-APA
Programs: AF Platinum, EK Gold Royal Air Maroc OW Emerald Hyatt Globalist
Posts: 22,540
Wow, what a mess. This was 5am this morning at LAX (as I was on my way to T7 and UA). Did these folks just stay up all night??









SFO777 is offline  
Old Aug 9, 2016 | 11:34 am
  #594  
Suspended
 
Join Date: Dec 2011
Location: RDU
Programs: DL DM, HHonors Diamond, Marriott Platinum, etc etc etc
Posts: 2,341
Originally Posted by skyrider117
On the slightly ironic side of things..
I just popped over to Deltajobs.com and the following is posted...

Operations Manager IT - Crisis Management

https://delta.greatjob.net/jobs/JobD...a-c575fdf177fa

The job was posted on 7/25

HAHAHAHAHA, best post of the day.

Classic
vincentharris is offline  
Old Aug 9, 2016 | 11:36 am
  #595  
 
Join Date: Jan 2015
Programs: delta dm
Posts: 1,677
Originally Posted by SFO777
Wow, what a mess. This was 5am this morning at LAX (as I was on my way to T7 and UA). Did these folks just stay up all night??









What a cluster f. I would have found another way or canceled and beeen refunded if not connecting.
dzflyer is offline  
Old Aug 9, 2016 | 11:37 am
  #596  
Suspended
 
Join Date: Dec 2011
Location: RDU
Programs: DL DM, HHonors Diamond, Marriott Platinum, etc etc etc
Posts: 2,341
Originally Posted by RatherBeOnATrain
His wife, and him are 2 clowns, They made it 90% of the way getting to ATL. Rent a damn car, and drive the 5ish hours instead of waiting for food vouchers.

Enjoy the honeymoon, complain for compensation later.
vincentharris is offline  
Old Aug 9, 2016 | 11:38 am
  #597  
20 Countries Visited
1M
All eyes on you!
10 Years on Site
 
Join Date: Mar 2016
Location: MSP & BWI
Programs: DL PM & NRSA, AA Gold, ANA Plat, Hilton Gold, Marriott Gold, National EC Exec, GE, CLEAR
Posts: 1,227
Originally Posted by dzflyer
What a cluster f. I would have found another way or canceled and beeen refunded if not connecting.
So help me god. And to think I was going to even consider a connection through ATL. Nope.

If LAX is that bad, I can only imagine what poor Widgets is dealing with.
TheLifeOfA_NKCM is offline  
Old Aug 9, 2016 | 11:41 am
  #598  
A FlyerTalk Posting Legend
20 Countries Visited
3M
Conversation Starter
20 Years on Site
 
Join Date: Aug 2002
Programs: UALifetimePremierGold, Marriott LifetimeTitanium
Posts: 74,140
My friend's 4:40pm now delayed 2 hours. Flight aware shows inbound plane getting in an hour before, but that doesn't mean anything. Yesterday the inbound landed/was taxiing to the gate according to flightaware, & the ongoing flight was canceled btwn when inbound got to gate & outbound was due to depart.

Cheers.
SkiAdcock is offline  
Old Aug 9, 2016 | 11:41 am
  #599  
100 Countries Visited
150 Countries Visited
200 Countries Visited
20 Years on Site
 
Join Date: Sep 2004
Posts: 312
Originally Posted by DL2SXM
so, just wondering, for those that have flown since the computers went down and subsequently came back up, have the pilots/FA's been making any special PA announcements or just sort of carrying on like its business as usual?
I flew on a 4-hour delayed flight yesterday morning (well, afternoon it ended up being) but there was no notice made of the delay. I thought they would at least apologize for the delay, but they made no mention of it. I wasn't offended or anything, but it did surprise me. Just business as usual.
Rabidstoat is offline  
Old Aug 9, 2016 | 11:42 am
  #600  
 
Join Date: Feb 2016
Location: PUW, LWS, GEG
Programs: Delta FO, SW Rapid Rewards
Posts: 67
My bro was flying GEG-MSP-LGA-PWM yesterday, departing 9:45am arriving 9:36pm. He was rebooked to the 7pm GEG-SEA, Red-eye SEA-ATL, ATL-PWM. They cancelled the ATL-PWM flight and now he's been rebooked into BOS sometime today. Does he have enough reason to ask for a full or partial refund since Delta is no longer getting him back into his original destination?
Upthewazzu is offline  


Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service -

This site is owned, operated, and maintained by MH Sub I, LLC dba Internet Brands. Copyright © 2026 MH Sub I, LLC dba Internet Brands. All rights reserved. Designated trademarks are the property of their respective owners.