Bad Surveys
#1
Original Poster
Join Date: May 2003
Location: GEG
Programs: Motel 6 Club Avoir Le Cafard
Posts: 5,027
Bad Surveys
I'd like to see an outright ban on any surveys that consist only of radio buttons and do not provide reasonable means of free-form text input. Surveys of this type don't allow people to tell the stupid corporations that commission the surveys what it is we would really like to tell them.
#2
FlyerTalk Evangelist




Join Date: Mar 2004
Location: SGF
Programs: AS, AA, UA, AGR S+, Choice Platinum
Posts: 23,317
I'd like to see an outright ban on any surveys that consist only of radio buttons and do not provide reasonable means of free-form text input. Surveys of this type don't allow people to tell the stupid corporations that commission the surveys what it is we would really like to tell them.
Based my [admittedly very little] experience gathering data for empirical research papers, adding the ability to put free-form text adds a large degree of complexity to the research process and moves the poll from a measurement of hard data to subjective interpretation of what people meant.
A well designed poll should cover all of its respondents' points of view without leaving them feeling like they wished they could answer a different way. For example, in the project currently posted on FlyerTalk, I was satisfied with my answers to most of the questions, but questions 7 and 8 left me wanting--I wished there were an option for $0 (I wouldn't pay for a "health food & drink menu") or "I don't need this service" (I never use, much less would pay for, the ability to do airport check-in, except when I am required to by the airline's own backwards policies), and I think the spectrum of miles should realistically be wider (would an airline really offer an upgrade to business class for 1,000 miles? I wish they would, but 10,000 or more is much more realistic).
And, of course, the preamble to question 7 was misleading, given that Southwest is actually an odd duck among its low-fare companions in that it does not charge for most additonal services (opting instead to simply not offer them at all). I chalk that up to the researcher's UK background.

So, is this a poorly designed survey? Maybe; maybe not. Of course, the researcher is at a disadvantage, since he/she is subjecting himself/herself to scrutiny by a panel of several hundred thousand experts.
The lesson to future researchers should be to, in order to save face and also get the most accurate data possible, perhaps do a test run with a few dozen people from the forums here to get input on the poll itself before releasing it to the general population. But I don't think an outright ban is truly necessary.

Edit: There is some discussion of this particular topic in ORP: http://www.flyertalk.com/forum/only-...er-survey.html
Last edited by jackal; Jul 18, 2010 at 8:12 am Reason: Link to ORP

