Air Canada experiencing nationwide network outage 12Mar18
Mar 13, 2018, 12:51 pm
#121
Join Date: Apr 2000
Location: Mississauga Ontario
Posts: 4,106
The mind boggles at how long this is going on for. Wasn't the last AC crash during another peak season, maybe their systems just can't handle the traffic.
This is why we need EU style rules, force the airline to fix it or pay for it. And I know all the counter arguments about prices increasing (despite the fact they're already higher) and limited population etc...
This is why we need EU style rules, force the airline to fix it or pay for it. And I know all the counter arguments about prices increasing (despite the fact they're already higher) and limited population etc...
Mar 13, 2018, 2:36 pm
#122
Join Date: Jan 2017
Location: Halifax
Programs: AC SE100K, Marriott Lifetime Platinum Elite. NEXUS
Posts: 4,574
Infrastructure types would like you to believe that DR is a matter of storage synchronization, rebooting, and network route changes. Which might have been true of single applications running on single mini or mainframe servers. Client/server, let alone Internet style services orchestrating dozens or hundreds of things?
Ha!
Its virtually impossible without deep architectural changes and checking assumptions. One doesn't build DR on top of a data center. One builds DR deep into the bones of applications.
If you aren't up to the tasks of turning loose chaos monkey and chaos gorilla on your production servers at any time you don't have application redundancy.
Air Canada takes 4 hours to update their website.
Ha!
Its virtually impossible without deep architectural changes and checking assumptions. One doesn't build DR on top of a data center. One builds DR deep into the bones of applications.
If you aren't up to the tasks of turning loose chaos monkey and chaos gorilla on your production servers at any time you don't have application redundancy.
Air Canada takes 4 hours to update their website.
Mar 13, 2018, 5:34 pm
#123
Join Date: May 2015
Location: Vancouver
Programs: Aeroplan, Mileage Plus, WestJet Gold, AMEX Plat
Posts: 2,026
As it turns out about a year ago there was an outage at a data centre in Victoria that houses some of our system (not related to aviation in any way). The same data center also houses some provincial government ministry systems, federal government systems and some banking systems. BC Hydro was doing scheduled work on AC mains, the maintenance company that looks after the emergency generators left the generators in the wrong mode and they did not fire up as planned. This still happened with all the redundancy and diverse backup power systems. In a QA oriented environment people get trained to be systematic, so when you have human error it occurs systematically across all the generators.
Mar 14, 2018, 1:50 am
#124
Join Date: May 2016
Posts: 2,494
Some companies actually employ staff to go around turning things off randomly and checking the redundancies work.
Complete loss of an entire location, should also have a redundancy. Like a secondary location. And a plan to bring the primary one up again. UPS isn’t new tech.
Complete loss of an entire location, should also have a redundancy. Like a secondary location. And a plan to bring the primary one up again. UPS isn’t new tech.
Mar 18, 2018, 1:39 pm
#125
Join Date: Oct 2012
Location: YYC
Programs: AC SE
Posts: 2,343
My spouse was delayed and stuck at YYC sitting around the airport for ~ 5 hours this past week due to a cancelled flight from the IT meltdown. A passenger from the same flight asked the agent at the lounge desk in YYC for a meal voucher, as it was a lengthy delay. The lounge agent answered in a condescending tone that there were no meal vouchers allowed, as this was an uncontrollable delay, because all flights were delayed, so it was equivalent to weather related IRROPS or an act of God. 
I'm sure it was a tough day for staff, but making up stuff about it not being within the airline's control and refusing to offer a meal voucher for a 5+ hour controllable delay doesn't really help AC's reputation in dealing with these sort of things.
I'm sure it was a tough day for staff, but making up stuff about it not being within the airline's control and refusing to offer a meal voucher for a 5+ hour controllable delay doesn't really help AC's reputation in dealing with these sort of things.
Mar 18, 2018, 1:42 pm
#126
Suspended
Join Date: Sep 2014
Programs: AC SE100K-1MM, NH, DL, AA, BA, Global Entry/Nexus, APEC..
Posts: 18,877
My spouse was delayed and stuck at YYC sitting around the airport for ~ 5 hours this past week due to a cancelled flight from the IT meltdown. A passenger from the same flight asked the agent at the lounge desk in YYC for a meal voucher, as it was a lengthy delay. The lounge agent answered in a condescending tone that there were no meal vouchers allowed, as this was an uncontrollable delay, because all flights were delayed, so it was equivalent to weather related IRROPS or an act of God.
I'm sure it was a tough day for staff, but making up stuff about it not being within the airline's control and refusing to offer a meal voucher for a 5+ hour controllable delay doesn't really help AC's reputation in dealing with these sort of things.
I'm sure it was a tough day for staff, but making up stuff about it not being within the airline's control and refusing to offer a meal voucher for a 5+ hour controllable delay doesn't really help AC's reputation in dealing with these sort of things.
Ah yes....
https://www.flyertalk.com/forum/air-...-re-wrong.html
Mar 18, 2018, 1:42 pm
#127
FlyerTalk Evangelist
Join Date: Jun 2003
Location: YYC
Posts: 23,816
My spouse was delayed and stuck at YYC sitting around the airport for ~ 5 hours this past week due to a cancelled flight from the IT meltdown. A passenger from the same flight asked the agent at the lounge desk in YYC for a meal voucher, as it was a lengthy delay. The lounge agent answered in a condescending tone that there were no meal vouchers allowed, as this was an uncontrollable delay, because all flights were delayed, so it was equivalent to weather related IRROPS or an act of God.
Still, my bet would be that the agent made that rule on the spot.
If you feel you might need a 10% discount code, you know what to do, right?
Mar 18, 2018, 2:01 pm
#128
Join Date: Oct 2012
Location: YYC
Programs: AC SE
Posts: 2,343
Yes, all very true!
Mar 18, 2018, 3:37 pm
#129
A FlyerTalk Posting Legend
Join Date: Sep 2012
Location: SFO
Programs: AC SE MM, BA Gold, SQ Silver, Bonvoy Tit LTG, Hyatt Glob, HH Diamond
Posts: 44,390
Mar 18, 2018, 5:04 pm
#130
Join Date: May 2013
Location: YYT/YYC/TPE
Programs: AC SE, UA, National Exec Elite, Nexus, GE
Posts: 1,810
Perhaps RES III is also used to issue meal, hotel, and taxi vouchers? When RES III went down, AC stopped operating.
At least their payroll system isn't tied in to RES III. Or is it?
At least their payroll system isn't tied in to RES III. Or is it?
Mar 19, 2018, 11:24 am
#131
Suspended
Join Date: Oct 2009
Location: Kan@da
Programs: Anything with sweet spots
Posts: 1,790
Are people getting compensations from Air Canada due to the delays caused by the outage. Delta proactively offered $200 vouchers to anyone affected for a similar case in 2016.