IT outage - anyone know the root cause ?
Apr 2, 2022, 7:05 am
#31
Join Date: May 2014
Posts: 7,218
I think Altéa is a red herring here, as far as I know it didn't fall over this week or February. It has done so in the past, and there is a backup system for Altéa. Altéa has failed with other airlines, there was a bad failure back in 2017 that took out much of Europe's airlines for a couple of hours. But as I say, I don't think it was directly involved with either of the two events of this year. As far as I can tell the impacted systems were some interconnected operational pieces of software that control crews, gates, IDAHO, the absence of which makes the basic management of LHR T5 almost impossible. This is presumably software/middleware controlled by IAG's IT team in KRK. The recent one, giving a 2 hour outage, is probably within the tolerance of fallback/recovery mode but if you are extremely short of staff then the dominos keep on falling. It also came on top of the February hardware problem which means the pain of an unrelated incident tends to mess together in any discussion - as we can see upthread.
One point I'd like to make is that, to the best of my knowledge, there's no IT managed in Krakow. When IAG GBS was created, the plan was for it to be a service provider for Finance, Procurement, IT and HR. Then HR wriggled out (never say that a reorg touches it!); Finance and Procurement moved to Krakow, but IT didn't - apart from the buyers. IAG GBS IT head honchos are all in London.
Apr 2, 2022, 5:59 pm
#32
Join Date: Jan 2007
Location: Channel Islands, UK
Programs: BA Blue
Posts: 556
I think that BA's IT system has become like Railtrack. Railtrack outsourced most of its engineering and maintenance capabilities to other companies as it was "cheaper" and as a result, lost the collective knowledge of managing the systems.
And by laying off all their staff over the last 2 years to save money, they lost the collective knowledge of their operations, like Railtrack!
We all know what happened to Railtrack in the end!
And by laying off all their staff over the last 2 years to save money, they lost the collective knowledge of their operations, like Railtrack!
We all know what happened to Railtrack in the end!
Apr 3, 2022, 2:58 am
#33
Fontaine d'honneur du Flyertalk
Join Date: Jul 2001
Location: Morbihan, France
Programs: Reine des Muccis de Pucci; Foreign Elitist (according to others)
Posts: 19,167
Fascinating though it is to know why this fiasco came to pass; I am far more interested to know what the hell they are doing to prevent this reoccurring. How often are we faced with cancellations and delays. I’m fed up to the back teeth of reading about how people are seriously inconvenienced by flights cancelled in a heartbeat and all the chaos amd stress that it causes.
In the name of God, is this a business or a circus? I’m really losing patience with this.
I have wondered at times maliciously and without a shred of evidence that when Cutback Cruz binned the IT department and handed the job to cowboys whether a few little bugs were put into the system to snooze and come back and bite BA’s bottom on a monthly basis as a goodbye basis. It’s is far from impossible to know or to prove but I do not seem to hear of this nonsense at other airlines.
This has gone beyond a joke. Burnt Discs, Backups be dratted. This is going from bad to worse.
There, now I feel much better. Time to put on my slap and take HI out for lunch.
In the name of God, is this a business or a circus? I’m really losing patience with this.
I have wondered at times maliciously and without a shred of evidence that when Cutback Cruz binned the IT department and handed the job to cowboys whether a few little bugs were put into the system to snooze and come back and bite BA’s bottom on a monthly basis as a goodbye basis. It’s is far from impossible to know or to prove but I do not seem to hear of this nonsense at other airlines.
This has gone beyond a joke. Burnt Discs, Backups be dratted. This is going from bad to worse.
There, now I feel much better. Time to put on my slap and take HI out for lunch.
Apr 3, 2022, 3:25 am
#34
Join Date: Apr 2005
Location: UK
Programs: IC Hotels Spire, BA Gold
Posts: 8,667
I think that BA's IT system has become like Railtrack. Railtrack outsourced most of its engineering and maintenance capabilities to other companies as it was "cheaper" and as a result, lost the collective knowledge of managing the systems.
And by laying off all their staff over the last 2 years to save money, they lost the collective knowledge of their operations, like Railtrack!
We all know what happened to Railtrack in the end!
And by laying off all their staff over the last 2 years to save money, they lost the collective knowledge of their operations, like Railtrack!
We all know what happened to Railtrack in the end!
With Hatfield specifically, the dangerous flaw in the rail had been identified and the replacement rail even ordered and was lying between the tracks waiting to be fitted when the crash happened - it had been there over 6 weeks!! But compared to the old BR, a completely separate company now had to fit the new rail so this involved lengthy discussions as to whether this was renewal or wear-and-tear, then negotiations, T&Cs, possession booking, agreement with the train companies to be able to close the line etc etc. The inquiry noted with a similar incident (under BR) on the West Coast Main Line 7 years earlier, the replacement rail had been fitted within 6 days of the serious flaw being identified. Shocking stuff!
Railtrack went into receivership around 2002/3 and the new Network Rail was formed, with key routine Maintenance and many services being taken back in-house. Some interesting parallels with BA and IT outsourcing I think!
Apr 3, 2022, 3:26 am
#35
Join Date: Nov 2015
Location: London
Programs: BA Gold
Posts: 1,680
Fascinating though it is to know why this fiasco came to pass; I am far more interested to know what the hell they are doing to prevent this reoccurring. How often are we faced with cancellations and delays. I’m fed up to the back teeth of reading about how people are seriously inconvenienced by flights cancelled in a heartbeat and all the chaos amd stress that it causes.
In the name of God, is this a business or a circus? I’m really losing patience with this.
I have wondered at times maliciously and without a shred of evidence that when Cutback Cruz binned the IT department and handed the job to cowboys whether a few little bugs were put into the system to snooze and come back and bite BA’s bottom on a monthly basis as a goodbye basis. It’s is far from impossible to know or to prove but I do not seem to hear of this nonsense at other airlines.
This has gone beyond a joke. Burnt Discs, Backups be dratted. This is going from bad to worse.
There, now I feel much better. Time to put on my slap and take HI out for lunch.
In the name of God, is this a business or a circus? I’m really losing patience with this.
I have wondered at times maliciously and without a shred of evidence that when Cutback Cruz binned the IT department and handed the job to cowboys whether a few little bugs were put into the system to snooze and come back and bite BA’s bottom on a monthly basis as a goodbye basis. It’s is far from impossible to know or to prove but I do not seem to hear of this nonsense at other airlines.
This has gone beyond a joke. Burnt Discs, Backups be dratted. This is going from bad to worse.
There, now I feel much better. Time to put on my slap and take HI out for lunch.
I suspect BAs only real option is to keep picking off the most problematic parts of the system, and overtime develop more robust solutions, but on a piecemeal basis. Oh, and don’t get rid of any more experienced staff if there any left.
Apr 3, 2022, 4:06 am
#36
Fontaine d'honneur du Flyertalk
Join Date: Jul 2001
Location: Morbihan, France
Programs: Reine des Muccis de Pucci; Foreign Elitist (according to others)
Posts: 19,167
If you run a complicated mishmash of system developed over a long period of time, you don’t need staff with malicious intent, the complexity/wouldn’t start from here nature of your IT will soon enough bring problems by the bucket load.
I suspect BAs only real option is to keep picking off the most problematic parts of the system, and overtime develop more robust solutions, but on a piecemeal basis. Oh, and don’t get rid of any more experienced staff if there any left.
I suspect BAs only real option is to keep picking off the most problematic parts of the system, and overtime develop more robust solutions, but on a piecemeal basis. Oh, and don’t get rid of any more experienced staff if there any left.
Apr 3, 2022, 4:38 am
#37
Join Date: Aug 2015
Location: Effectively grounded
Programs: BA GGL for a little while longer
Posts: 844
There are parallels here (mentioned somewhere above I think) with the major High St banks.
Years of mergers, crises and stakeholder return-driven economies stripped out a huge amount of organisational memory and expertise.
Some years ago, Mr OtH was on a project for a large well-known bank. It became necessary to know, for reasons of regulatory compliance, the precise order and timing of some account operations.
They asked the product managers - nobody knew
They asked the operations teams - nobody knew.
They asked IT - nobody knew
They asked senior management to pose the question again, rather more forcefully - still, nobody knew.
But down the pub, the IT guys closest to the (core) systems in question said "we avoid touching them if humanly possible: all the expertise got laid off last year, there's no documentation and a good chunk of the components are out of support, like years out of support..."
Mr OtH moved on a little while later for unrelated reasons so I assume they found a way to fudge it, but fixing errors in that kind of environment is a nightmare at the best of times...
Years of mergers, crises and stakeholder return-driven economies stripped out a huge amount of organisational memory and expertise.
Some years ago, Mr OtH was on a project for a large well-known bank. It became necessary to know, for reasons of regulatory compliance, the precise order and timing of some account operations.
They asked the product managers - nobody knew
They asked the operations teams - nobody knew.
They asked IT - nobody knew
They asked senior management to pose the question again, rather more forcefully - still, nobody knew.
But down the pub, the IT guys closest to the (core) systems in question said "we avoid touching them if humanly possible: all the expertise got laid off last year, there's no documentation and a good chunk of the components are out of support, like years out of support..."
Mr OtH moved on a little while later for unrelated reasons so I assume they found a way to fudge it, but fixing errors in that kind of environment is a nightmare at the best of times...
Apr 3, 2022, 4:42 am
#38
Join Date: Oct 2014
Location: Krakow
Programs: BAEC Silver, Miles and More(FTL), IHG(Platinum), Accor, HHonors(Diamond), SPG, Hertz Five Star
Posts: 5,879
Apr 3, 2022, 4:59 am
#39
Join Date: May 2014
Posts: 7,218
Fascinating though it is to know why this fiasco came to pass; I am far more interested to know what the hell they are doing to prevent this reoccurring. How often are we faced with cancellations and delays. I’m fed up to the back teeth of reading about how people are seriously inconvenienced by flights cancelled in a heartbeat and all the chaos amd stress that it causes.
In the name of God, is this a business or a circus? I’m really losing patience with this.
I have wondered at times maliciously and without a shred of evidence that when Cutback Cruz binned the IT department and handed the job to cowboys whether a few little bugs were put into the system to snooze and come back and bite BA’s bottom on a monthly basis as a goodbye basis. It’s is far from impossible to know or to prove but I do not seem to hear of this nonsense at other airlines.
This has gone beyond a joke. Burnt Discs, Backups be dratted. This is going from bad to worse.
There, now I feel much better. Time to put on my slap and take HI out for lunch.
In the name of God, is this a business or a circus? I’m really losing patience with this.
I have wondered at times maliciously and without a shred of evidence that when Cutback Cruz binned the IT department and handed the job to cowboys whether a few little bugs were put into the system to snooze and come back and bite BA’s bottom on a monthly basis as a goodbye basis. It’s is far from impossible to know or to prove but I do not seem to hear of this nonsense at other airlines.
This has gone beyond a joke. Burnt Discs, Backups be dratted. This is going from bad to worse.
There, now I feel much better. Time to put on my slap and take HI out for lunch.
In the aftermath of the May Bank holiday outage I remember Alex saying, very frankly and rather frustratingly, that he didn't have the control of an IT department. It was in the WTS theatre and I remember saying "I'm a CEO and I don't have an IT department". In fact Teflon Bill, the man who ran IAG GBS, reported to Willie and not to Alex.
BA overcooks its IT. Has always been the case, sadly.
Apr 3, 2022, 5:17 am
#40
Join Date: Oct 2009
Location: Dundee
Programs: BA Plastic. HH Diamond. Speedwell Bar Lifetime Platinum.
Posts: 1,424
I think that BA's IT system has become like Railtrack. Railtrack outsourced most of its engineering and maintenance capabilities to other companies as it was "cheaper" and as a result, lost the collective knowledge of managing the systems.
And by laying off all their staff over the last 2 years to save money, they lost the collective knowledge of their operations, like Railtrack!
We all know what happened to Railtrack in the end!
And by laying off all their staff over the last 2 years to save money, they lost the collective knowledge of their operations, like Railtrack!
We all know what happened to Railtrack in the end!
If you look back to the early stages to privatisation, the IMCs (Infrastructure Maintenance Companies) didn't feature and Railtrack was to be the operator and maintainer.
Only later did the IMUs (Infrastructure Maintenance Units) come out of BRIS (British Rail Infrastructure Services), before being sold off to various outfits to become separate IMCs.
Railtrack didn't have a big say in the initial tranche of contracts, they just went with the legacy IMUs. Only the subsequent round of bids changed much.
Apr 3, 2022, 5:19 am
#41
Fontaine d'honneur du Flyertalk
Join Date: Jul 2001
Location: Morbihan, France
Programs: Reine des Muccis de Pucci; Foreign Elitist (according to others)
Posts: 19,167
Alex did notbin the IT department. Much as I don't like the man, much as I think he was a mediocre leader and not the one BA deserved, it's important to underline it wasn't him. BA's IT department moved under IAG GBS at some point in 2013-14. I think that was the time when the then head of BA Procurement also left, and that was the first time when BA IT people were being told that they'd be losing their job. The decisions on the gigantic cock-up that has been IT were done at IAG level.
In the aftermath of the May Bank holiday outage I remember Alex saying, very frankly and rather frustratingly, that he didn't have the control of an IT department. It was in the WTS theatre and I remember saying "I'm a CEO and I don't have an IT department". In fact Teflon Bill, the man who ran IAG GBS, reported to Willie and not to Alex.
BA overcooks its IT. Has always been the case, sadly.
In the aftermath of the May Bank holiday outage I remember Alex saying, very frankly and rather frustratingly, that he didn't have the control of an IT department. It was in the WTS theatre and I remember saying "I'm a CEO and I don't have an IT department". In fact Teflon Bill, the man who ran IAG GBS, reported to Willie and not to Alex.
BA overcooks its IT. Has always been the case, sadly.
I think that one need now to ask - what are they doing to prevent this happening again? Where are they going with all this as it will kill BA if this goes on month after month.
Apr 3, 2022, 5:25 am
#42
Join Date: May 2014
Posts: 7,218
Fine so it wasn't him, and it is unfair of me to blame him for the sins of the others. It must be the only thing that he didn't cut
I think that one need now to ask - what are they doing to prevent this happening again? Where are they going with all this as it will kill BA if this goes on month after month.
I think that one need now to ask - what are they doing to prevent this happening again? Where are they going with all this as it will kill BA if this goes on month after month.
Apr 3, 2022, 5:27 am
#43
Join Date: Aug 2014
Posts: 2,657
I’m led to believe that the recent outages have been caused precisely because BA has recognised its IT infrastructure is crap and is trying to do something about it. The problem is, the company they brought in to do it has turned out to be even worse than they are, and almost every change they make breaks immediately and has to be rolled back. (And no, before the xenophobic idiots jump in, it’s not an Indian company this time, it’s a supposedly blue-chip British supplier.)
Apr 3, 2022, 5:33 am
#44
FlyerTalk Evangelist
Join Date: Mar 2013
Location: London
Posts: 17,007
To middle management, it's very hard to define these positions, and so it seems there are a bunch of long serving highly paid folks in department code X who surely can be replaced by an overseas contractor to perform the equivalent function more cheaply?
Apr 3, 2022, 6:08 am
#45
Join Date: Nov 2017
Posts: 3,059
These ancient interconnected systems usually only work because a very small number of people take huge pride in their jobs, know everything about everything inside out, and go above and beyond to fix problems before anyone else notices them.
To middle management, it's very hard to define these positions, and so it seems there are a bunch of long serving highly paid folks in department code X who surely can be replaced by an overseas contractor to perform the equivalent function more cheaply?
To middle management, it's very hard to define these positions, and so it seems there are a bunch of long serving highly paid folks in department code X who surely can be replaced by an overseas contractor to perform the equivalent function more cheaply?
The ba.com redesign is apparently close to being signed off. But it’ll take 5 years to deliver… and the manager who designed it has just left. I can see now how well that’ll go…