Originally Posted by flatlander

This sort of incident needs to be investigated in the manner of a flight operations incident, so analysis of what happened, direct causes and contributing factors, and analysis of how to stop these same causes acting again to cause another incident. Taking a first action of deploying either the HR disciplinary procedure (on internal staff) or lawyers with damages briefs (on external suppliers) will achieve little for better reliability. I encourage anyone who investigates this sort of IT failure, or wants to know how to do it better, to read AAIB accident reports and books on Just Culture in technical operations.

Originally Posted by DesertNomad

How many empty positioning flights are running? If I were BA, I would be really tempted to just board whoever wants to go anywhere. 400 seats available to Japan. Show a passport and go. If the gate agents could not determine if a visa was required, then don't board them. Maybe have them prove that they have a return ticket (printout or such). If they did this, I doubt many people would game the system... e.g. if I had a round trip LHR-ZRH, I am not going to jump on a plane to Japan since getting back would be a problem.

It'd be a way to clear it out. Put Richard Branson in charge of things and there would be a solution. I have boarded third world airlines with a hand written ticket and BA is looking fairly third world at the moment. Time to break out the hand-written tickets.

Originally Posted by lorcancoyle

unfortunately I don't think many other countries would permit flights to arrive unless the manifest had been provided in advance - Spain and US are 2 with API requirements that spring to mind.

Originally Posted by Oaxaca

The reason is understandable, but this forum is the busiest I've seen it in the couple of years I've been on here. I opened the forum homepage and thought there was something wrong with the formatting on my iPad - but no, it was just a massive list of active users. As I write there are 1322 active users on the BA FT forum, over 300 registered users and over 1000 lurkers.

I hope Alex Cruz sent out a few hi-viz jackets for the mods to get them through the night, though FT's IT seems to be coping admirably...

Originally Posted by techie

While this will is and will be an expensive lesson for BA, I hope that Heathrow airport will learn a thing or 2 as well.

Originally Posted by Lewis King

Seeing a few flights scheduled to depart at 1-2am. Guessing Heathrow have allowed BA to break the curfew?

Originally Posted by flatlander

There have been a number of declarations of how it should be done better with independent data center locations and so on (many in the main thread about this event). Those points are valid, but in orchestrating the failover or load distribution between the datacenters is complex. Knowing what services are provided from where, avoiding split brain situations (ensuring that authoritative data sources are in one place or the other but not both) and so on is not easy.

In particular, while you can have two isolated copies of a system, two sets of services, etc, in each datacenter, the system that orchestrates which is running at any time has to have knowledge and connections in both datacenters. The fail-over mechanisms are not simple twins where if one dies, the other lives, but siamese twins, connected to both datacenters and subject to problems in either datacenter - if one dies, the other can get sick.

Comments so far indicate that BA has several locations, and uses a private cloud infrastructure. That private cloud will need a cloud resource manager, job scheduler, service availability manager, etc, and that cannot be isolated to one datacenter or another. If an unexpected failure occurred these cloud management systems then the entire cloud can stop working. This sort of failure has been the root of several Amazon Web Services outages, as well as other well-known outages, and the possibility of it happening is impossible to remove because the resource manager must have a connection with all datacenters, and so a vulnerability in all datacenters, to function.

In summary, I am afraid I am tempted, on a bad day, to answer those blunt comments in the other thread with "I think you'll find it's a bit more complicated than that."

Meanwhile, obviously such an outage should be quicker to repair in BA, and this is where most of the learnings will lie: questions of how to restart systems, investigation into previously unknown dependencies, reducing complexity to improve separability and therefore reliability in case of partial outages, and so on.

Additionally, those who call for 'heads to roll" and for people to be sacked over simply because it happened, are also calling for the wrong thing and acting in a way that will not improve reliability. Personal vengeance rarely makes complex systems any better.

This sort of incident needs to be investigated in the manner of a flight operations incident, so analysis of what happened, direct causes and contributing factors, and analysis of how to stop these same causes acting again to cause another incident. Taking a first action of deploying either the HR disciplinary procedure (on internal staff) or lawyers with damages briefs (on external suppliers) will achieve little for better reliability. I encourage anyone who investigates this sort of IT failure, or wants to know how to do it better, to read AAIB accident reports and books on Just Culture in technical operations.

First discover what went wrong, why it went wrong, and then if, and only if, there was gross negligence or malfeasance do you fire or sue people.

Originally Posted by Genius1

HAL have head office staff on standby to assist in the terminals when things go wrong like today, as well as the flexibility to deploy additional operational staff - both of these things happened today and presumably will continue for the next day or two. HAL have contingency stocks containing items such as water and snack bars which helps to a degree when passengers are queuing, as well as the ability to hand out refreshment vouchers.

Originally Posted by bioyuki

This will be one epic postmortem. I agree with your premise that an investigation needs to happen before heads roll, but I disagree with your fundamental argument that building a stable, resilient architecture is difficult. It's 2017 for goodness sake;

Originally Posted by Banana4321

If you started in 2017 then yes it's easier. Try walking into a company with legacy systems dating from the 1960s (which most large companies do); add in a few systems acquired as part of a takeover/merger; a few rationalisations of suppliers; a few upgrades here and there of operating systems; a decision based on a dogmatic CIO to change database suppliers.

As has been said: "I think you'll find that it's a bit more complicated than that".

As the joke goes.
Driver stopping for directions: "Which is the way to <my destination> please?"
Pedestrian: "Well, I wouldn't start from here if I were you"

Originally Posted by Banana4321

If you started in 2017 then yes it's easier. Try walking into a company with legacy systems dating from the 1960s (which most large companies do); add in a few systems acquired as part of a takeover/merger; a few rationalisations of suppliers; a few upgrades here and there of operating systems; a decision based on a dogmatic CIO to change database suppliers.

As has been said: "I think you'll find that it's a bit more complicated than that".

As the joke goes.
Driver stopping for directions: "Which is the way to <my destination> please?"
Pedestrian: "Well, I wouldn't start from here if I were you"