Originally Posted by
The Situation
The interesting tidbit that I just remembered from one of the articles or press releases (can't remember) was that prior to making these changes they did a focus group study. When DL told the focus group the initial qualification requirements people were very turned off to the program. Once DL told the focus group all of the new ways to earn MQD, people became happy with the program. To DL, hearing that feedback told them the changes were exactly what was needed. The initial reaction should have been the red flag.
Having been part of a focus group and also leading a focus group study at work, I can confidently say the study was designed to tell the DL team what they wanted to hear. Otherwise why would you start the goal posts far away and move them closer - of course people are going to warm up to whatever you say in that situation. The end result should have been the only thing presented to the focus group if they wanted honest feedback.
That's such a good point. So many organizations misuse focus groups.
A great story I heard once was when Sony was designing the Walkman, years ago. As I recall, they did a focus group where they asked the people in the group whether they preferred a black walkman design (more "serious" looking), or a bright yellow one (more "fun"). The participants overwhelmingly said they wanted the black one. But then they put a bin outside of the meeting where participants would be able to pick up a free prototype Walkman, and they had both black and yellow ones. Almost everyone actually chose the yellow one.
"Design by focus group" is fraught with danger, because focus group participants are often going to tell you things in the abstract that, when actually faced with a real choice in real life, they would absolutely not choose. People often don't know what they want until they see it in reality and actually have to make a real choice. If you're smart, you do your market research not by asking people whether they "like" things, or what things the think they want, but by observing their actual behavior when their choices make some sort of concrete difference to them. Clearly Delta just did this very simplistic level of research where there's no skin in the game and people were just abstractly answering focus group questions without actually thinking about and working out how various program designs would really function in practice for them. You're gonna go way off course designing things that way.