Go Back  FlyerTalk Forums > Miles&Points > Airlines and Mileage Programs > British Airways | Executive Club
Reload this Page >

27 May BA IT outage miscellaneous discussions thread

27 May BA IT outage miscellaneous discussions thread

Old May 27, 17, 5:55 pm
  #76  
 
Join Date: Jun 2014
Location: Heathrow
Posts: 124
A direct copy/paste from Alex Cruz on Yammer - our internal social network.

He is responding to concerns raised by frontline staff in T5. I really don't know where to start, so I won't say anything at all.

Guys, either you are part of the team working to fix this or you aren't. We are not in the mode of "debriefing on what happened" but rather "let's fix this" mode.

I have been answering some emails from colleagues (thank you for the support) and I just finished a video for media and another one for all of us. I suppose you will have seen our Twitter account and BA.com.

Now, your interest on today's events is well noted. I am wondering if you would like to help out? Either LHR or LGW or anyone of our stations could use you, now. If you indeed can, drop me a message and I will connect you or go straight to the airport and make yourself available. I am sure they will appreciate your involvement.

In the meantime, if you don't want to get involved or you cannot get involved, I would kindly ask you to refrain from live commentary, unless it is a message of support to the thousands of colleagues that love BA as much as you do. Thanks.
alextheengineer is offline  
Old May 27, 17, 6:00 pm
  #77  
 
Join Date: Jul 2016
Location: London
Posts: 251
I have given BA a lot of leeway in the past. I have worked as a BA employee, and still undertake work on behalf of BA. I have been GGL since the age of 20. I don't care particularly about food provisions, lounges, or any sort of special treatment, and don't expect much from the airline even though I am fortunate to always fly in the highest class of travel.

Today, however, BA cost me and my company an inordinate about of money by being (a) unable to provide a back up system for missed flights, and (b) any sort of appropriate alternative provision.

I am deeply sorry for all of those within BA trying to make up for these shortcoming, but enough is enough.
ShuttleRunner is offline  
Old May 27, 17, 6:02 pm
  #78  
 
Join Date: Apr 2014
Location: London
Programs: Don't even mention it. Grrrrrrr.
Posts: 950
Originally Posted by alextheengineer View Post
A direct copy/paste from Alex Cruz on Yammer - our internal social network.

He is responding to concerns raised by frontline staff in T5. I really don't know where to start, so I won't say anything at all.
And that is fair enough comment IMHO. Get stuck it or shut up (for now)
Banana4321 is offline  
Old May 27, 17, 6:04 pm
  #79  
 
Join Date: Aug 2009
Posts: 321
Originally Posted by alextheengineer View Post
A direct copy/paste from Alex Cruz on Yammer - our internal social network.

He is responding to concerns raised by frontline staff in T5. I really don't know where to start, so I won't say anything at all.
Guys, either you are part of the team working to fix this or you aren't. We are not in the mode of "debriefing on what happened" but rather "let's fix this" mode.

I have been answering some emails from colleagues (thank you for the support) and I just finished a video for media and another one for all of us. I suppose you will have seen our Twitter account and BA.com.

Now, your interest on today's events is well noted. I am wondering if you would like to help out? Either LHR or LGW or anyone of our stations could use you, now. If you indeed can, drop me a message and I will connect you or go straight to the airport and make yourself available. I am sure they will appreciate your involvement.

In the meantime, if you don't want to get involved or you cannot get involved, I would kindly ask you to refrain from live commentary, unless it is a message of support to the thousands of colleagues that love BA as much as you do. Thanks.
This is really quite special. Time to fall on his sword.
polochick is offline  
Old May 27, 17, 6:11 pm
  #80  
 
Join Date: Apr 2004
Location: Scottsdale, AZ
Programs: AA 2MM - UA 1P / Hyatt Diamond - SPG Plat / Hertz 5* - Avis 1st
Posts: 3,502
Apparently the temptation at BA to outsource the systems of record, combined with the projected savings, were too large to resist.

Long ago, far away in the world of the Y2K concerns, we assessed some portions of the BA infrastructure as a part of our job as a supplier to them. At the time, BA had an in-house IT department that struck me as competent and organized. Our evaluation indicated that they would handle Y2K with sufficient grace, which in fact they did.

Granted this was two decades ago, but I am strongly amazed at the deterioration of the BA IT organization, given the results we have seen this weekend.

You can't outsource IT and "throw it over the wall." Someone has to actively and intentionally monitor and manage the processes and key resources, but based on today's results, it certainly doesn't appear to be the case at BA.
Wilbur is offline  
Old May 27, 17, 6:32 pm
  #81  
 
Join Date: Jan 2000
Location: SoCal to the rest of the world...
Programs: AA EXP with lots of BA and CX. (Disgruntled UA Lifetime Plat) - No hotel loyalty anymore
Posts: 6,345
Originally Posted by Wilbur View Post
You can't outsource IT and "throw it over the wall." Someone has to actively and intentionally monitor and manage the processes and key resources, but based on today's results, it certainly doesn't appear to be the case at BA.
I sold to Speedwing EONS ago (like 20 years ago) - didn't that group run most of the IT platform before the sell off to Amadeus. Boggles the mind when an airline takes their in house staff and moves them to the supplier vs keeping a core competent group to run backup provisions and keep the supplier "honest"
NickP 1K is offline  
Old May 27, 17, 6:59 pm
  #82  
 
Join Date: May 2015
Location: Vancouver
Programs: Aeroplan, Mileage Plus, SPG Gold, AMEX Plat
Posts: 1,324
Originally Posted by Worcester View Post
Aren't airports classed as critical infrastructure by the government? Perhaps they should force this Spanish airline (BA) to have higher standards of IT infrastructure if they have so many airport slots.
Perhaps this is internal to the UK subsidiary. Given the Spanish and Irish operations not affected.
Fiordland is offline  
Old May 27, 17, 8:06 pm
  #83  
FlyerTalk Evangelist
 
Join Date: Nov 2007
Location: The Hague, NL
Programs: GMLFL, Life 2.0 - Mucci Premiere Classe & des Chevaliers Toulousiens
Posts: 22,850
Originally Posted by Banana4321 View Post
If you started in 2017 then yes it's easier. Try walking into a company with legacy systems dating from the 1960s (which most large companies do); add in a few systems acquired as part of a takeover/merger; a few rationalisations of suppliers; a few upgrades here and there of operating systems; a decision based on a dogmatic CIO to change database suppliers.
Meh... BA seems to be hit harder than most. That is also because they are often launching customers for new technology. If you are, you need to make even more sure that you have redundancies etc. It is never going to be 100% uptime though.

Not going to type an essay about it. Someone already did that. And here is the other side.

It happens to all airlines at some point but I believe BA had it multiple times now. Expensive problem though. Didn't Delta lose around $100M?
henkybaby is offline  
Old May 27, 17, 8:32 pm
  #84  
 
Join Date: Apr 2014
Location: London
Programs: Don't even mention it. Grrrrrrr.
Posts: 950
Originally Posted by henkybaby View Post
Meh... BA seems to be hit harder than most. That is also because they are often launching customers for new technology. If you are, you need to make even more sure that you have redundancies etc. It is never going to be 100% uptime though.

Not going to type an essay about it. Someone already did that. And here is the other side.

It happens to all airlines at some point but I believe BA had it multiple times now. Expensive problem though. Didn't Delta lose around $100M?
Maybe they (BA) are more vulnerable because they are advancing through a huge set of changes in order to deliver better value to the shareholders and better service to their customers. As the saying goes, if you are in control then you are not going fast enough.

If BA is hit "more than most" that is not surprising to me. This is not a tiny company, this is one of the world's largest airlines and companies - a large company with global operations and a huge legacy infrastructure that have suffered many takeovers and mergers. Of course it is much more likely to be hit than (say) AirBNB who have had the luxury of creating a modern technology infrastructure. As for "Multiple times" this issue has only happened once...
Banana4321 is offline  
Old May 27, 17, 9:37 pm
  #85  
 
Join Date: Dec 2001
Location: China
Posts: 1,500
That message from Cruz is spot on for internal comms I thought. Either help out, or at least don't be a distraction. Just bringing sandwiches in for the front line staff would be appreciated.

Pretty much all airlines outsource 'systems of record'. Delta and ANZ the exceptions I think. As do most companies- who now builds a proprietary/ in house general ledger or email software for example?
peasant is online now  
Old May 27, 17, 9:49 pm
  #86  
 
Join Date: Apr 2004
Location: Scottsdale, AZ
Programs: AA 2MM - UA 1P / Hyatt Diamond - SPG Plat / Hertz 5* - Avis 1st
Posts: 3,502
Originally Posted by peasant View Post
That message from Cruz is spot on for internal comms I thought. Either help out, or at least don't be a distraction. Just bringing sandwiches in for the front line staff would be appreciated.

Pretty much all airlines outsource 'systems of record'. Delta and ANZ the exceptions I think. As do most companies- who now builds a proprietary/ in house general ledger or email software for example?
No one is going to build their own accounting system, I agree. However, there is a difference between maintaining custodianship of your instance of SAP and outsourcing the custody and operation of the instance of SAP that holds your critical financial records. As an executive, you have to answer the question of, "What is the maximum pain and maximum frequency of my outsourcer losing or failing to maintain this data?"

Even MORE important, however, is your operations application stack. If you outsource that, and the outsourced service provider loses it, you don't operate. As BA is discovering. And the costs of "failure to operate" for a lot of companies is quite high.
Wilbur is offline  
Old May 27, 17, 9:58 pm
  #87  
 
Join Date: Jan 2000
Location: SoCal to the rest of the world...
Programs: AA EXP with lots of BA and CX. (Disgruntled UA Lifetime Plat) - No hotel loyalty anymore
Posts: 6,345
Originally Posted by Wilbur View Post
No one is going to build their own accounting system, I agree. However, there is a difference between maintaining custodianship of your instance of SAP and outsourcing the custody and operation of the instance of SAP that holds your critical financial records. As an executive, you have to answer the question of, "What is the maximum pain and maximum frequency of my outsourcer losing or failing to maintain this data?"

Even MORE important, however, is your operations application stack. If you outsource that, and the outsourced service provider loses it, you don't operate. As BA is discovering. And the costs of "failure to operate" for a lot of companies is quite high.
This single event likely strikes the "cost reductions" gained from 5 years of an outsourced service provider.
NickP 1K is offline  
Old May 27, 17, 10:42 pm
  #88  
 
Join Date: Dec 2001
Location: China
Posts: 1,500
True but if an in house data centre loses control it is equally nasty see delta for example
peasant is online now  
Old May 28, 17, 12:05 am
  #89  
 
Join Date: Jan 2016
Location: LON
Programs: BAEC, Accor
Posts: 1,381
Originally Posted by drb1979 View Post
Yes, so the network was up. But why couldnt others log in? that points to an authentication or DNS issue which doesnt make sense if the root cause is a data centre power issue hours earlier. Both systems should be distributed. Unless I'm missing something!
Originally Posted by peasant View Post
True but if an in house data centre loses control it is equally nasty see delta for example
Agree, there are many aspects of running a data centre and the looking after the Base level equipment that an internal IT team never have enough day to day exposure of to be well practiced in all of the eventualities that might present themselves. There are economies of scale and expertise here.

So moving your tin and getting an outsourcer to be responsible for operations and maintenance up to a point for the Infrastructure makes a lot of sense and maybe some common off the shelf low importance applications. Let your internal IT look after the key business applications - this is where they can add value to your operation, as the outsourcer usually struggles to understand what is important to the culture and frontline of the company.
plunet is offline  
Old May 28, 17, 12:29 am
  #90  
 
Join Date: Feb 2010
Location: London
Programs: BA GGL (98% to Lifetime Gold), SPG convert from IHG (and Marriott fan now!)
Posts: 4,645
Originally Posted by alextheengineer View Post
A direct copy/paste from Alex Cruz on Yammer - our internal social network.

He is responding to concerns raised by frontline staff in T5. I really don't know where to start, so I won't say anything at all.
even for internal comms that is incredibly poorly presented. It says something about his mindset that he jumps straight to the "if you're not part of the solution you're part of the problem" position. Could be rewritten in 2 minutes, but shouldn't have been so poorly phrased in first instance.
lorcancoyle is offline  

Thread Tools
Search this Thread