How to write great user surveys - Part 4

This  is part 4 of my series on writing a good user survey. If you're just joining us, you may want to start with part 1.

Conflicts of interest

Imagine the following question (which I just made up):

Do you agree or disagree with the following statement? Munchery provides a good value for the money I spend.

• Strongly agree
• Mostly agree
• Neither agree nor disagree
• Mostly disagree
• Strongly disagree

When that question is written, the writer thinks "Customers will tell me what they think of Munchery's value." When the customer reads that question, however, sometimes he thinks "Munchery's value is okay. But if I say that I disagree, maybe Munchery will lower its prices." Some customers will do this and some won't -- it depends on your customers and your product. But it will skew your results, and it might skew them in a way that costs you money you didn't need to spend.

Essentially, customers answering that question have a conflict of interest -- their incentives are not aligned with yours. But if you need that information anyway, you can usually still get it if you're sneaky -- you just have to hide your motives a little bit. In my made-up Munchery example, that question has some business mystery behind it. Maybe it's "should we lower or raise our prices?" or "do our customers consider us a luxury?" or "would we get more orders if we gave out more coupons?" There's almost certainly some business need as the motivation, and it's probably not "The CEO was just wondering how users perceive the service's value on a five-point scale."

So think about what you really want to find out. Is there a way you can get that information from the user without asking him to think about it? Like the hole-in-the-jeans example from part 3, there's probably another question you can ask which will allow you to measure results instead of asking for opinions. And the right answer might not always be a survey question. But whether it is or not, it's important to avoid these conflicts of interest if you care about getting the most accurate data.

Come back tomorrow for part 5, where I'll talk about why the answers you provide are just as important (and can be just as harmful) as the questions you ask.

--
This is part 4 of my series on writing a good user survey.  If you want to start at the beginning, head back over to part 1.

Hire me, and then I'll write your next survey! I'm available for full-time or consulting work -- email me.