I see quite a few teams who are validating their market with surveys. Unfortunately, a large majority of them seems to fall for common traps or might not attract the type of answers they expect. So instead of trying to help critique each survey one by one, I’m going to identity six important tips you can apply and pitfalls to avoid that will get better survey results.
Tip #1: People have better things to do
Most people would rather not spend their free time answering your survey questions. Your objective when making a survey above all else should be making it easy for them to give you a response without using too much of their free time. All of the other points support this one, but it should be at the front of your mind as you add questions to your survey.
Questions should be short and survey takers should be done answering within 15-30+ seconds of finishing the question. If they have to really think about the question, find another way to ask.
Set the survey taker’s expectations so they know approximately how much time they will invest to respond to your survey or how many questions they can expect. If they are unprepared for a 15 minute session, they might leave the session early without submitting or lose interest as they don’t see a conclusion on the horizon.
Tip #2: Always ask important questions
Before you begin your survey, you should have a clear idea of what knowledge you hope to gain from its responses. Many of us will be trying to validate our value proposition. For this objective, each question should be chosen to validate as much of your value proposition as possible without being too broad. After you write your question, ask yourself the following: Has this question moved me toward validating my idea? Can I rewrite it so that it’s simpler and gets me closer to idea validation?
For example, when asking someone “How important is XXX to your business?” you could give them the standard “Very Important -> Not Important” scale, or you can find a more clever way to ask that removes subjectivity from the question. “If you had to remove XXX from your business, what percentage of revenue would you lose?” and provide them with a “0% -> 100%” scale. In this example, we assume that revenue is an important metric for the survey taker. With the new question, you now have an objective and (more importantly) quantifiable answer.
Tip #3: The order of your questions matter
There are two scales which will help you determine the order of your questions.
Most Insightful <—–> Least Insightful
Easiest to Respond <—–> Most Difficult to Respond
With these scales in mind, I would prioritize them like so: Easiest to Respond first, Most Insightful, Least Insightful, and Most Difficult to Respond at the end. Anyone who comes to your survey will undoubtedly give a worthwhile shot at answering a few questions. Giving them easy questions at first builds up their momentum into the survey and provides a sense of accomplishment (that we all love).
You’ll have to carefully balance how soon you ask questions that are difficult to respond as each one will discourage the survey taker from continuing. Sometimes you have to ask difficult questions up front. In these cases, offer something simple they can answer just before a difficult question. Even something innocuous and irrelevant like “Did you have a great day today?” with a Yes/No would likely get more responses for your difficult question than otherwise.
Tip #4: Avoid open-ended questions at all costs
When faced with the option of how to present your question to the survey taker, there are three primary question types: Multiple Choice (choose one), Checkboxes (choose as many as needed), and Free-form (blank text box). Each has an influence in the type of answer you will receive and how you will process that data later.
Multiple Choice is almost always the ideal format as it gives the viewer the benefit of fitting their answer into a predefined choice. They typically require little thought and the fastest time to complete. However, offering the correct choices that remove bias, self-selection, and covers enough range in potential answers without adding unnecessary noise to your results is very difficult. For the majority of the questions, worrying about bias is not an issue. (I’ll touch on bias in a bit.)
Checkboxes are a great way to ask “coverage” questions. Coverage means broad questions that (usually) have a fixed answer set and removes the burden from the user to think of every possible answer. Questions like “What social network do you use?” or “How many of these competitors have you tried?” are perfect for this. You may have to do a little extra work to make sure that you have a good sample of your answer set available for selection. If you believe your sample answers are not representative of the total set of possible answers, you can usually get away with throwing an “Other…” option in the mix to cover your bases (but only do this if it’s necessary).
And then there’s the dreaded free-form answer. These questions literally suck the life out of your survey taker. Not everyone wants to formalize their answer into words, the answers will probably have a lot of unnecessary information which you’ll need to scrub through later, and you force the user to think a lot more. But there are some scenarios where they are unavoidable.
I usually try to form the question in one of the above formats first, but if the answers are too restrictive for multiple-choice or checkboxes, free-form text is typically what you’re left with. ALWAYS ask a direct and simple question. The survey taker should not have to think more than a few seconds and their answer should fit into a sentence or two. If your answer needs to be more elaborate, consider breaking the question down into parts.
Tip #5: Pagination can be your friend
Facing the user with many questions will likely turn them away before they ever start. Some survey tools will allow you to break up questions among a series of pages. When used sparingly, they can be a great tool for getting your survey taker to complete their response. Putting 3-5 questions per page gives the survey taker that sense of accomplishment they secretly love without the discouragement of a visibly long survey.
As an added bonus, some survey tools will “record” the progress as they move from page to page. If they abort half-way through, you still have partial data. (A half-win is better than no-win.) Avoid too many pages and give the survey taker an idea of how much further they have to go before they finish. You can also consider putting individual free-form questions on their own page so the survey taker isn’t distracted or overwhelmed.
Tip #6: How to avoid bias and self-selection
Some questions will lead the survey taker to the answer you want instead of getting an answer that’s honest. Here are a few examples:
Ex 1: Do you think the King Penguin is harmful strangling the Antarctic’s natural resources?
Ex 2: The King Penguin is…
- beneficial to Antarctic’s natural resources
- neutral to Antarctic’s natural resources
- harmful to Antarctic’s natural resources
In the first version, bias is introduced by offering a starting position for the survey taker to agree or disagree with. In the second version, the survey taker can assert their own opinion without being “primed” or influenced. When asking a question that is subjective, leave the subjectivity completely out of the question part and put it in the answer. Some more advanced survey tools will randomize the order of the answers on each load to further reduce perceived bias by the survey taker.
Ex 1: Were you satisfied with product XXX?
- OH YES! A LOT!
- Oh, it’s okay.
- Uh, I loathe it.
Ex 2: Rate your satisfaction of product XXX?
- 1 – Very Unsatisfied.
- 5 – Very Satisfied.
In the first version, you have a few problems. You automatically remove the “loathe” option for anyone who doesn’t understand the word. Focus on words that you’re certain the population taking the survey will understand. This also includes avoiding slang and informal answers (which a user might have a negative reaction to for a completely unrelated reason). In the second version, numbers typically won’t carry any extra baggage in their meaning but still convey an appropriate amount of information to all parties.
What are some other tips for writing effective surveys?
Making a good survey is tricky, but applying some or all of these tips will give you a survey that’s easy to finish and leaves the survey taker feeling happy they could help. The Adventure Lab applied these ideas to its own market validation survey and will be testing them with individuals who have very little spare time and little opportunity to try again. I’m sure there are many other tips and suggestions that could improve a survey’s effectiveness and I’m curious if you have any that you’ve personally had success with. What are some of the ones you’ve tried?
Image courtesy of 89studio.