Sorry for the cuts and reordering but trying to put a bit of order in the criticisms levelled at the airlineratings.

First, let's start with the obvious. Of course any evaluation, league table, or rating is, by nature, "unscientific" by nature. The media, websites, and even regulatory authorities are not about science. There are others of us who are and there is good reason why we are not typically producing the stuff you read in papers or indeed regulatory processes which, whilst they always claim to be "embeded in science" are not aiming to be generated by it. Now, of course, it would be quite easy to get on my scientific high horse, poo poo everything the FT, Times, NYT and others produce on the grounds on being "unscientific", but those, invariably motivated by a desire, on the part of the authors to generate interest and thus money, play a role in society too, so let's agree that if any of you has consistently ignored every university league table when you or your kids decided where to study, hospital league tables where family members are facing challenging life circumstances, area ratings when you have decided where to buy your house and car (or tv or washing machine or laptop or phone or whatever) ratings when you have bought one, well done to you and you can stop reading now: you are officially immune to the league table frenzy and quite frankly, your objections are valid towards the whole of it rather than this specific one.

For the rest, let's try and be a little bit more specific because "unscientific" would mean pretty much nothing in an exercise which is by nature located outside of the scope and purpose of science, which is really to understand and explain and not to give gold stars and the likes. There are four big types of objections which I have seen either expressed or implied in this thread.

The first relates to aggregation. How do you bring together various heterogeneous data to come up with some sort of index of safety. This is the one I have the most sympathy for, and that is why from the start, I have largely ignored the "headline" stuff to try and move to one of the specific criteria mentioned which I found startling: the suggestion by the website reps answers to the media that BA effectively has a lot more non-life threatening incidents compared to other airlines of similar size.

The second relates to the face validity of the operationalisation. That's where that choice of criteria matters. Now of course, everyone is entitled to their own opinion about what they care about, for instance one could perfectly decide that they only care about fatalities, something the entire airline industry typically does to assert that virtually the entire sector worldwide is perfectly safe and welcome to your Air Koryo flight. Now don't take me wrong, I actually agree that it is good that airlines check planes, spot problems, and report them to get them fixed, but ultimately, in a framework where air travel indeed represents an incredibly safe form of transportation (and I say that entrusting my life to 200 flying things a year and having no issue travelling on TK, QR and tiny regional turboprops alike), I personally do find that the occurrence of non-life-threatening incidents is a useful criterion to look at.

The third is the question of comparability of metrics across cases. That, to be honest is something someone will invariably raise about any comparative dataset. I mean that literally, whether you talk of airline safety, personality traits, lung cancer occurrence or people's height, there is always some reason to suspect some level of uneven playing field. While the knowledge of such differences is useful, experience tells me that their effect is almost always much less than people suspect and that the vast majority of comparative research conclusions remain valid when you change or harmonise measurement, particularly if you focus on a relatively homogeneous sub-universe (ie forget about the comparison with Qatar or Emirates, let's look at claimed differences with other European airlines). That problem in one of the comments above is also levelled at whether any descriptive differences are effectively significant. Fair question, and indeed, if anyone worries about that it would strike me as perfectly fair to write to airlineratings and ask them the question (or ask them for the data) but it would not make sense to assume insignificance by default. In fact, from experience, mechanically and based on sheer numbers and the number of cases, one can almost guarantee that whether one deems those really meaningful or not, those differences will almost certainly tick the boxes of significance so that objection, whilst worth checking, is unlikely to carry much weight in practice.

The fourth is the notion of bias particularly due to the idea that there would be a fundamental difference in what looks like a non-life threatening incident on BA vs all other airlines. Now, there would be a major paradox in accusing the overall model of lacking scientific systematicity and synchronically accepting some anecdotal impression (however competent) of such difference as scientific disproof. I really think that any suggestion that somehow, a LH, KL, or SK pilot will likely go for some sort of "oh well, who cares, que sera sera" approach where the BA ones would systematically call maintenance not very intuitively convincing. Again, maybe I'm naïve, but my impression is that those BA, LH, KL and SK pilots are overwhelmingly competent professionals who, in my experience, systematically call maintenance every time an indicator flashes that should not and so on even when they are certain that this won't be a problem and certainly, my experience on board those airlines is of fairly similar messages with identical proportion of clearance about 5-10 minutes later vs need to call for additional tests which would not suggest vastly different threshold.

There is also the question of why the situation would be described as worsening on this specific ground for BA over the years if BA's position is due, as argued, to outlier reporting standards compared to all other European airlines in particular.

Now don't take me wrong, if we are talking on the basis of intuition, then everything goes and people are obviously free to consider that BA is the safest airline in the world and everything just as good as it was 2 or 3 years ago. One alternative is that something has systematically changed in counting or reporting instructions over time that could explain apparently worsening non-life-threatening incidents. The alternative is that somehow, there is a valid reason for such change - for instance that the progressing ageing of some of the fleet does indeed logically result in some maintenance issues becoming a little bit more regular than before, or that the change in some maintenance contracts has been unfortunate, or that changing behaviour on passengers' side - such as increased plugging of lithium battery powered electronics - generate additional challenges on board, or that increased average plane use over time has a toll in terms of the likelihood of minor incidents occurring. Personally, while I think that BA is absolutely a safe airline as are all other European airlines and most world airlines and once again, I fly them all without feeling worried, if I were BA, I would certainly consider those questions rather than just dismiss as a "the methodology is against us" which I think would be unwise, and I do not think that it is unreasonable to raise those questions on FT either.