...
In my mind, if a feature pleases 10% of our members and doesn't bother the other 90%, it adds value to FlyerTalk. If a feature improves post quality without bothering members, it adds value. If a feature improves post quantity without degrading post quality, it adds value. These are the three ways in which I expect reader feedback to improve FlyerTalk. This is why I want to test it.
...
Am I correct to read this as your goal is to add value to Flyertalk and to determine if value is added the criteria are
a) at least a small minority like it and majority of users are not bothered
and/or
b) increased post quantity with post quality unchanged
and/or
c) increase post quality without bothering members
and the suggested way to do it is by adding a "like" button with the resulting like-count attached to posts and/or a list of members liking the post?
You said that users sometimes does not know what they want until you show it to them. That is very true. But it is also true that users may not know and understand the long term effects of what they are shown. They see only he surface of what you show them. Continuing on the Jobs analogy, when shown new laptops without anti-reflex coating, they users loved the new high contrast displays but didn't understand it would be difficult to use them in a lit setting. When shown the original ipod with polished steel backs they adored the new shiny toy without understanding how easy it was to scratch it.
Some things demos very well. It doesn't mean they are certain to add value.
One recent example is the new premium deals forum. The major change to the structure has apparent and immediate positive effects and many like what they see. Others are very bothered with it. As for the large majority of members I'd guess they are not bothered with the change as they either never saw it or never used it. By your evaluation this is a success and even though some major issues have been raised, it is clear that TB won't even listen to a remodel/take down call for several months, if ever.
My personal view is that quality of posts in that forum have gone down considerably and that the long term effect on the forum, despite all the positive effects, is declining relevance when fewer good deals get posted. If that is what really is happening long term, did that change really add value to FT?
This is why I have an issue with the criteria "doesn't bother others". Sure, I am not bothered with the like-button itself. I can overlook it and never click it. But I
am bothered with the long term effects.