A lot of us have anecdotes about terrible FF award availability, and they have seeded widely held beliefs: CO is awful, UA is pretty good, AA is getting worse, etc. But so far as I know there's no authoritative data-gathering exercise which measures the situation on a per-airline basis. I propose inventing one.

I would like to develop a fair and rigorous survey methodology to poll major US airlines' FF award availability. The goal is to publish a quarterly (or even monthly) index/ranking of airlines, showing how accessible their "standard" awards really are.

Look at gas prices. Everyone has anecdotes about how it's $2.55 here, $2.25 there, etc., but they don't add up to a market survey. For that you need Dan Lundberg, who brings a proven methodology to the price watch. We don't have a Dan Lundberg in the FF field... so maybe we can become the equivalent.

Some of the factors that would go into designing a survey methodology:

** How many airlines would we survey? I'd propose ranking all the US carriers offering online award search tools.

** How many random city pairs per airline would constitute a fair "core sample"? I'd propose 100.

** What timeframe between search date and travel date should we specify? Because anecdotes suggest the availability pictures changes as a travel date nears, I propose trying at least two timeframes: maybe 10 days out and 120 days out.

** Would we attempt to weight our "test city pairs" so we manufacture a mix of leisure and business destinations, foreign and domestic destinations, hub and multi-segment trips? Or just let random probability work its will?

** I propose we have non-elites survey standard award availability in all classes.

I think an index/ranking would interest not just FTers but the press and maybe universities. Lots of schools issue vague customer satisfaction rankings of airlines based on soft factors. This index would be about data.

I'm an ex-journalist, now a writer and information designer, and I'd like to have a crack at this -- but I need some help and would like the community to comment on/propose revisions to the methodology ideas above.

Anybody want to play?

Why eliminate elites? I suspect in general they have more miles to burn than non-elites and therefore would be more likely to go after international awards and J and F awards that require higher mileage.

If you are worried that elites get to pull from an "extended" pool of award seats and therefore would skew the results, I wouldn't worry. The reality seems to be we have as much trouble getting awards as non-elites.

Other than that, I think it is a good idea if the data is gathered in a standardized way.

It might be interesting to somehow track award availability in comparison to empty seats. Also, any interest in extending the study to hotel awards?

Incredible! Count me in. I was thinking the exact same thing this morning and was trying to gather my thoughts for a post. I planned on calling it the "Great CO Award Availability Study" in deference to the pathetic availabilty of standard award seats on that airline, but clearly other airlines should be included.

My view is that a random city pair sample is not the best approach. For any particular airline, we would want to focus on "prime" frequent flyer award redemption routes. I'm sure most people are much more interested in CO award availability from EWR to HNL than they are for EWR to Cleveland. I propose we poll people on popular routes for each airline and go with the top hitters. I would also suggest that we not limit the study to non-elites, but that we attempt to secure volunteers at all levels so we can see the impact being elite has/does not have on award availability.

The most troublesome issue is how far out to check. Again, I believe we need a range of times starting at 330 days and moving down to 1 day in advance at 60-90 day intervals (less as you get closer in). We also need to establish guidelines for how the availability check is to be performed and when, i.e., all checks done on-line at Noon each day, so we have a rough comparison standard across airlines.

Also, we will need to have multiple and staggered start dates as we are all aware that availability varies significantly by day of week. For example, if someone started checking 330 days out from today (somewhere around May 21, 2005 which is a Saturday, we would also need someone to check identical routings for Mondays and maybe a mid-week day as well. The logisitics of how this works will need to be agreed upon, but I believe we can come up with a workable solution. Thoughts?

100 city pairs might be a way to do the survey, but I suppose that we would find some airlines opening many seats for awards on unpopular routes, when they no longer hope to sell them. If this routes are unpopular, companies shouldn't be given many points for making them available for award. Besides, most of us, I think, are saving our miles for C or F class tickets, or long-haul tickets.

Perhaps we should survey (instead of or as well as your initial idea of 100 US city pairs) the availability of tickets in first or business class between major cities around the world. Even if we lack a way to know the real number of available seats, we could check if there is at least one seat available for a given city pair 331 days before departure, then checking for steadily nearer dates. It would answer the question : how long should I plan a trip in order to redeem successfully my miles. Besides, airline websites do not generally allow to book many seats in one go, so we wouldn't know really how many seats are available, while when a flight has no more available seats, it is no longer possible to book even one seat.

I love the idea and would be willing to help on any of the airlines I'm a member of -- AA, CO and NW.

I think you'd want to pick a variety of cities/routes and then check a number of days in advance (for example: 1, 7, 21, 42, 90, 180 and 330) for each pair -- as well as availability on some routes around specific holiday periods.

If you wanted to build a specific rating, I think you'd want to weight it in favor of economy awards as those are the ones which are most often used.

Greg

I second the idea of targeting only a few routes. Here are the important variables: route choice, lead time for the reservation, class of service, what airline, and all other factors.

Just having an analysis, by airline, of lead time required for regular-mileage economy class awards would be great. Route selection is the hardest variable to target in a survey like this for many reasons.

Different airlines have different routes available. It's hard to compare across airlines, considering that direct competition (many airlines flying the exact same routes) is so rare these days. Maybe the thing to do is take each airline's three most frequent long-haul routes. That gives us a good guess as to which long-haul routes are popular and diminishes the possibility of weird artifacts that might happen if only one flight a day is running a particular route.

This could be really very interesting...

The biggest problem, I think, is the one GradGirl and others touch on -- the apples-to-apples problem. To create a meaningful index for ranking, we need a level playing field. If we find excellent availability on numerous F9 city pairs, for example, the index should give F9 credit for that even though there's no chance to redeem for HNL or Europe.

OTOH if the survey reveals CO regularly offers standard award seats to SYR or DTW, the index should still penalize CO if we find no standard seats to LAS, HNL, MCO, etc. I hypothesize that the airlines point to sheer numbers of available award seats as proof the programs work, but many are to Flint or Wichita. (OTOH again, those secondary-market seats aren't valueless... I myself fly free to MHT or PWM a lot and appreciate it.)

Perhaps three diverse "market baskets" of business, leisure and secondary destinations would work well. Business: DCA, LGA, LAX, ORD, etc. Leisure: LAS, MCO, etc. Secondary: MHT, SYR, etc.

The other thing we'd have to manage is feasibility. I don't think we could try ten different timeframes, from two to 330 days out, for 100 different city pairs, and live to compile the results. I think three or four timeframes might make a sufficient data map: maybe 7 days out, 30 days out, 90 and 300?

I had suggested non-elites chasing standard awards because there are more non-elites than elites out there and the results should have the broadest meaning possible. Do we think it's worth it to have an elite and a non-elite query the system the same way, the same day and check for variance?

We would want to pick polling days offline and privately because I wouldn't put it past airlines to read these boards and stuff the system temporarily. Call me suspicious.

Keep the discussion going...

I think you have a good idea. I would recommend two categories, the 25 biggest US cities and the 10, 15 or 20 top US vacation destinations from each of the airlines hub cities. Who really cares if you can get to the hub on a free ticket if you can't get out to your destination.

You could then add the 10 biggest cities outside the US and the top 10 Vacation destinations outside the US.

I think we should limit it to US airlines (and their foriegn partners) that can be checked on-line. I know on NWA.com it is hard to check KLM flights out of AMS. So you can check to AMS or other cities in Europe that NW flies to directly, but not connections on KLM.

I think if you add to many connections in you will get availability on one leg, but not another.

I can help if you need another "tester".

As far as routes go, I recall that the now defunct Consumer Reports Travel Letter did a survey/study several years ago on the most popular frequent flyer award routings in the U.S. That study was very extensive, was based on actual data from the airlines (thru DOT if I remember correctly), and to this day is the best I have ever seen. Would love to get a copy of that for use in this project. Randy, if you are reading I bet you have this study squirreled away somewhere....care to share?

My view is that routings from each airlines' hubs would be the logical starting point as most FFs would either be coming from or transiting through the hubs. Popular destinations in the U.S. would include HNL, MCO, SFO, LAX, NYC, and ORD. So, for example, if we were going to study CO, I would suggest evaluating all routings from IAH and EWR to HNL, MCO, SFO, LAX, and ORD and IAH to/from NYC. We can probably reduce that list somewhat based on feedback.

My suggestions for advance time would be the following: 330, 270, 180, 90, 60, 30, 14, and 1 day(s) out.

To this point - absolutely. In fact, I think there should be two types of "buckets" - Domestic and International, further clarified by foreign "mini-hubs" and final destinations.

The only issue here is that most airlines make extensive use of their alliances for international travel - so perhaps the international "bucket" should be delimited by Alliances, not individual airlines.

This does have its drawbacks, however as evidenced below:

Here's an example - I'm trying to book a first class award to Rangoon, Myanmar in November. This flight would use UA for the segment JFK-LHR and THAI for the segments LHR-RGN (LHR-BKK-RGN) for the outbound and THAI RGN-NRT (RGN-BKK-NRT) and UA for the NRT-SFO-JFK on the return. On my first two choices of dates, all of the outbound segments are available, but the NRT-SFO on the return is not available in F, but is available in Y.

How would you treat this? Would the unavailability on ONE segment attribute to Star Alliance, lowering the overall *A rating, or would it attribute to UA specifically - and if it did attribute only to UA, how would you handle the fact that the award was only unavailable on one segment - and to even further delineate it, the award was available on that one segment, just not in F.

I think it's a great idea, but you will need to do quite a bit of planning to make sure that it is indeed a true objective system and can take into consideration the above example.

Good thoughts, but I don't see this as a big issue. If I have UA miles and UA can't get me to my desired destination, whether on UA metal or some combination of *A partners, then UA (or more specifically Mileage Plus) is dinged. I think we need to treat first class, business class, and Y separately to avoid the complications you note and make this whole thing manageable.

One problem I see is that many airlines do not permit booking international award travel on-line. That certainly limits what we can do unless we take to the phones. To get us started, I recommend we limit ourselves to U.S. destinations. After all, I believe some 75-80% of redeemed awards are the 25K domestic variety.

Excellent question. My knee-jerk response: the testing model specifies X days' trip length (maybe X = 10 for leisure cities, 3 for business cities) and a given class of service and if the return's not available, the whole award is recorded as unavailable. In your Myanmar case three-quarters of an award does you no good. Repeat that one rigorous thing enough times, and you'd emerge with a fair picture of an airline's award availability.

Model-planning is everything. If we run a survey, publish an index and get some press, someone isn't going to like it and we have to be able to defend the methodology.

A previous poster suggested we test for travel from hub cities to the top 25 business and leisure destinations. I seem to recall talk on the DL board that availability was thought to be worse when starting from a DL hub... you could get SEA-ATL-TPA but not ATL-TPA. So you'd probably want to have a balance there.

I would very much like to see that Consumer Reports travel letter alluded to above.

Which US carriers do not permit international award booking online? Having lived in the CO/NW/DL shadow for some time I plead ignorance...

Right on! It's very important to rate in all 3 categories: If a FF has the opportunity to use miles to fly to Syracuse, NY, Birmingham, AL, Winnipeg, MB, or Billings, MT, for business or a cousin's wedding, our system should tell him whether s/he has a realistic hope of using the same miles toward Hawaii or Paris next year OR if s/he should grab that ticket to Winnipeg while the grabbin' is good.

Niether DL or AA currently permit booking of international award travel via their website.