Calchas

Not that I am authorized to speak for jerub but I think she/he is contending that the root cause occurred earlier than the power surge, i.e., it was a failure to arrange or maintain the backup systems properly that is the root cause. Such a failure makes an outage an inevitability, it is then a simple matter of timing.

T8191

Yes, I was obviously being simplistic. I recall my sense of panic when my Mac backup started making protest noises, but at least I was able (as an old git) to carefully work my way around that.

My feeling, from several posts, is that something went horribly wrong during the recovery/reboot process. Which, in an ideal world, it shouldn't!

Calchas

Actually this is a point, in that if you have a very reliable system, it is hard to test the backup system regularly because it is never needed. But if your backup system doesn't actually work, that is not very far away from not having one.

Banana4321

If you re-read jerub's post you will see that he did not contend that at all. Rather jerub referred to the power issue as a "precipitating event".

As for "inevitability", everything happens in the end.

contractor

yeah, root cause of the service failure will be something like process failure to ensure backup systems were functioning, or process failure to implement correct backup procedures.

You could try to trace those back further to either negligence or some other item such as incompetence but going further will be difficult. Also it depends who is doing the investigation too...

Root cause can't be the power failure as that's a trigger for a known failure mode.

MPH1980

But thats also why you perform a scheduled DR event every so often. Prove out the backups and fail overs.

Of course - it wont always protect you. For example - a company I know had an outage due to a timezone issue. They had tested the DR when the UK and US were aligned on non DST (literally the week before a DR event). Failover failed when US was on DST and UK wasnt.

So you cant account for every variable but the failover should have been tested regularly.

Assuming it was tested - then the question is why did it fail THIS time?

T8191

True. Since that hiccup some months ago, I now check (occasionally) to ensure the silent background backups are happening as they should. A while later, the Mac died, and restoration to the new machine was painless!!

Stez

I think you need a backup keyboard.... or fingers.

T8191

You know my typong too well And for some reason iPad is having a go-slow weekend!

Calchas

^ You know, I hear BA may have an opening in their IT department for a redundancy engineer, if you could be tempted out of retirement.

T8191

hahahaha.

if that was the case, I could stop this iPad forever having to reload pages. I'm sure it's FT's endless adverts, for bloody Condor Ferries, combined with a 10mbps connection, that's causing it

Every time I try to expand the text so that I can read it ... "Reload"!

LadyShiva

Um, no. Properly designed redundancy can be performed rather easily--I have several groups that do weekly maintenance on the entire infrastructure. However, the funding to properly design seems not to have been provided by BA. Limitations to one datacenter/SPOF make me twitch.

The importance of chasing 5 nines seems to be lost on BA until this situation. Fifteen minutes of downtime a year seems to be outside of their previous understanding; I hope the lesson has been learned fully as a result of WHY funding 5 nines is an organizational imperative.

V10

I wouldn't bet your house on that possibility under the current management regime, to be honest.

LadyShiva

*laughing* We are agreed on your point. It usually takes a brutal financial slap to wake large companies up to how insane they've let their IT get via primary usage of duct tape and spit fixes.

subject2load

BBC Radio 5 Live have devoted a lot of time to BA during the course of this afternoon, with their coverage having now shifted firmly to issues surrounding handling & communications, rather than the specifics of cancellations/ delays.

Also included clips from Alex Cruz interview earlier today.

There have been contributions from affected passengers, BBC presenters, guest journos, and consumer/aviation 'specialists'.

The criticism throughout has been unfailingly relentless, and all centred on the standard of communication - described in one piece as 'just appalling.'

I think it is this matter of their communication & overall response to affected passengers which will prove most damaging - rather than the outage itself (the latter being something that many - perhaps even most - people might have been willing to accept)