How to write great user surveys - Part 2

This is part 2 of my series on writing a good user survey. If you just got here, you may want to start with part 1.

Step 3: Expect your questions to expand.

Let's rewrite my question from step 2 with some better language. Just as a reminder, that question was:

I think lots of people are bothered by redesigns. Is that true? How many people are?

 First I'll define all my squishy terms:

• "lots of people" - For my post on site redesigns, I care about users who are likely to be current or future customers of Silicon-Valley-type startups.
• "bothered" - I'm going to leave this term pretty broad, because I don't really know what kind of data to expect. However, I'm mostly interested in knowing if a redesign made people actually unhappy, so I'm going to use something slightly stronger: "frustrated." Then I'll user a 1-to-10 scale to get some more data on that afterwards.
• "redesign" - It would take a lot of time to figure out what users think qualifies as a redesign. Since their initial understanding of the term isn't really important to the study, I'm just going to define it for the user with my own made-up definition. It doesn't have to be the most elegant or concise definition -- it just has to be clear and unambiguous.
• "how many" - I'm mainly interested in the opinions of people who are irritated, because I expect that to be the majority. However, I could be wrong. If I am wrong, my post on redesigns will have a lot less value, and that is important information to have. (Primarily because it will save me many hours of writing several lengthy articles.) So even though I don't really care about the opinions of people who aren't usually irritated by redesigns, I need to find out how many there are. So in this case my "how many" will grow into two questions: whether people are usually frustrated by a redesign, and then a second question asking them to think of any time they have been frustrated by a redesign.

So! The results:

Before:

 I think lots of people are bothered by redesigns. Is that true? How many people are?

After:

For this survey, a "redesign" means a major change in the look, feel, location of features or actions, or other major functionality changes.
Are you usually frustrated when a website or app you use gets a redesign?

• Yes. Usually some parts of a redesign have made me frustrated or annoyed. 
• No. I don't usually mind when an app or site is redesigned.

If you answered Yes above, how unhappy does that frustration make you?
On the scale below, 1 is slightly unhappy and 10 is extremely unhappy.  

Of all the redesigns you encounter, how many of them make you frustrated?

• All or almost all of them are frustrating.
• Most of them are frustrating.
• 50-50. Some are, some aren't.
• Not that many. Most redesigns don't frustrate me.
• None or almost none of them frustrate me.

Boy, that one thing I wanted to find out turned into quite a lot of questions, eh? This is actually very common. When you’re interviewing a user in person, a small question can produce paragraphs of results. For multiple-choice survey questions, however, you’re only getting one or two bits of information each time. The simple question “is my product good?” has answers in dozens of dimensions -- price, durability, value, convenience, appearance, utility, quality, enjoyability, etc. And if you want to capture all of that information, you’re going to have to ask about it all explicitly. Because each of your users has a unique collection of dimensions in his head for what constitutes product goodness, and he’ll answer your question according to his definitions.

So be prepared to ask a lot of questions, and try to keep things focused. Nobody likes twenty-minute surveys, so it's important to keep your goals narrow. You can take shortcuts if you want by getting rid of questions, but that loss will carry over into the quality of your results.

Check back tomorrow for part 3, where I'll cover some common errors that can skew your survey results.
--
This is part 2 of my series on writing a good user survey.  If you want to start at the beginning, head back over to part 1.

Hire me, and then I'll write your next survey! I'm available for full-time or consulting work -- email me.