How to write great user surveys - Part 5

This  is part 5 of my series on writing a good user survey. If you're just joining us, you may want to start with part 1.

Step 5: Allow for the entire spectrum of possible answers.

One of the most common mistakes I see in surveys is that the test-maker didn't give the user enough options to express his opinion accurately. This happens constantly. Consider the following examples:

Did your experience with the site today make you more or less likely to recommend us to a friend?

• More
• Less

Hypothetical user Bob: "The site is great, but I practically never recommend sites to people."

or

Hypothetical user Jay: "I don't know; the site is fine. It doesn't really affect my odds of recommending. Whatever, man."

or

Hypothetical user Randall: "I recently had crappy customer service from you and I'm taking this survey with plans to express that in one of the questions. Your site is fine, but I'm not recommending you to anyone. You guys suck."

Now think about those users. They're probably in the middle of this survey. They've already invested some time in this, and they want to finish it so you can hear their opinions. They all think "I don't know what to put here... I wish there were more options." Let's look at how each user can affect your bottom line:

Bob: "The site is great, but I practically never recommend sites to people."
Bob isn't too bothered -- this question doesn't really apply to him. If he's smart and is thinking about how you'll perceive his answers, you may still get valid data out of him. He could think "Well, I don't recommend sites very much, but this site is pretty good. I'll tell them 'more'." He's not too risky, because he probably won't give you a wrong answer.

Jay: "I don't know; the site is fine. It doesn't really affect my odds of recommending."
Jay is ambivalent but dangerous -- whichever answer he picks will be actually wrong. And if that's the case for a substantial percentage of your users, just think what an expensive error that could be! If a bunch of those ambivalent users choose "less," that could translate to thousands of hours of work by your software engineers and designers -- all to fix a site that is already fine. That's not a cheap mistake!

Randall: "I recently had crappy customer service from you and..."
Randall isn't going to do you any favors either. He's already unhappy with you, and this survey is giving him an outlet to express that. He may or may not have an opinion about the quality of your site, but without a neutral option he's far likelier to give you negative feedback because of his state of mind. Like Jay, he can contribute to a very costly snipe hunt.

(Of course, if Randall thinks that taking a survey is the easiest way to express his dissatisfaction to you, you're already doing it wrong. He should be swimming in opportunities to tell you -- in your product itself (if possible), on the main page of the site, in your promotional emails, etc. But that's a subject for another article.)

It's worth noting here that there are some situations where you do only want one bit of information from your users. Check back tomorrow for part 6 where I'll talk about that special case, what's special about it, and how to do it right.

--
This is part 5 of my series on writing a good user survey.  If you want to start at the beginning, head back over to part 1.

Hire me, and then I'll write your next survey! I'm available for full-time or consulting work -- email me.