The methodology has been questioned in the past as being too susceptible to manipulation by folks who want to "game the system".

Because it is not the numbers of folks who vote for a given program as the "best" that is determinative, but rather the "value" the voters who selected that program give it, a program that a few think is truly "best" can win if those few give it automatic "10s" while the more popular programs are honestly graded. This has led in the past to "organized" attempts to vote a particular program "in" by giving it inflated values, or to punish a program by selecting it as the "best" but awarding value grades of "1" (one year Randy threw out a bunch of such votes on the grounds that someone who honestly thought a program merited a value of "1" could hardly honestly think it was the "best").

I understand Randy's concern. If you're truly trying to recognize quality, you can't simply count noses; that translates into a popularity contest and you might as well tally memberships and give the award to the program with the most active members. On the other hand, the current methodology does indeed lend itself to gaming.

And it is difficult for the travelers to have the base to honestly determine how a given program compares to all others out there, since a frequent traveler usually doesn't survey all of the programs every year, but rather concentrates on a few in order to achieve elite status.

I'm not sure where the answer lies, but it may be in some combination of the "users" verdict (votes of the travelers) and the expert panel evaluation, perhaps where experts evaluate (say) the five programs that the voters choose as their most favorite in each category.

I'm not sure what the answer is, but I do agree the question merits consideration.