This is part 3 of my series on writing a good user survey. If you're just joining us, you may want to start with part 1.
Step 4: Don't tip your hand.
When creating a survey, it's very important to remember that users don't always know what they want. Everybody has heard Henry Ford's famous quote about his customers wanting a faster horse (which it turns out he probably never actually said, but the point still holds). Ford's apocryphal quote doesn't mean that customer feedback isn't useful -- it should simply serve as a warning about asking your users the wrong questions.Customers usually aren't good at imagining innovative and financially-viable solutions to life's problems -- that's your job as the businessman. So why would you ask customers what solutions they want? What they are good at is... complaining. Which isn't a bad thing! It just means that when you ask them questions, sometimes you need to be careful to ask only about problems -- not solutions. And that can be tricky since customers are good at reading between the lines.
Is this hole noticeable?
Never ask this. If you have a hole in your jeans and you want to know if it's noticeable, why would you point it out to someone in the question? You've just ruined their objectivity. They can't un-notice it, so whatever they tell you is untrustworthy before they even say anything. If you really want to know if the hole is noticeable, create a test and measure the results: make sure the hole is visible to your friend, but instead ask "What do you think of these jeans? Does anything about them look unusual?"
Fundamentally, the difference between these two approaches is whether you're asking for an opinion or measuring results. It's easy to imagine ten friends saying "Nah, I don't think it's very noticeable." They might be wrong or trying to make you feel good, but you just don't know. Opinions may be well-meaning, but they aren't very trustworthy.
What is trustworthy, however, is measurable and repeatable results. By keeping your friend in the dark about your motivations, you don't have to wonder how accurately he is able to judge his own observational skills. You can find the answer -- the correct answer -- just by asking your question a little differently. Instead of saying "Do you think this is noticeable?" you are secretly asking "Do people notice this?" If you can remove "do you think" from your questions, your results will be infinitely more trustworthy.
Sometimes you'll find that measuring results obviates the need for a survey question in the first place. Which is fine -- sometimes surveys aren't the best way to find out what your users want. Before you go to a user survey to plan your roadmap for the next six months, ask yourself if there's a way to solve your problems by measuring results. You can learn hugely valuable things (and have a lot more confidence in your data) from AdWords tests or A/B Testing. (Multi-armed bandit is even better if you have the setup.)
Check back tomorrow for part 4, where I'll talk about another kind of question that can mess up your survey results.
--
This is part 3 of my series on writing a good user survey. If you want to start at the beginning, head back over to part 1.
Hire me, and then I'll write your next survey! I'm available for full-time or consulting work -- email me.
No comments:
Post a Comment