Survey Design: Structured Questions

This is part 2 of our series on Survey Design, previous segments are available in our archives.

There are two kinds of questions you can ask in a survey: structured questions, where there are a fixed set of possible answers and open questions, where the respondent can write whatever they want. Today we’ll cover the design of structured questions and tomorrow delve into open questions.

The most common form of structured question is multiple choice, where the respondent is asked to choose a single response from a few different options. This works well if you can capture every possible response in the options you provide. For example, in the Data Driven Survey we asked “How large is your company?” and the options included every possible company size, which makes the data easy analyze and visualize:

companysize

There are many traps to avoid when designing structured questions to ensure you get reliable and useful data:

  • The options can affect the results. If the multiple choice options you provide don’t capture every possible answer, you risk having respondents pick a wrong answer just to answer the question. If necessary, provide an “Other” option but realize that it will make your data harder to process. You might be surprised at how often it will be used, in the Data Driven Survey we asked “What industry does your company work in?” and 17.4% of respondents selected Other!
  • The order of the options matters. Respondents will be lazy, and they will have a bias for choosing answers that are at the top of the list instead of reading every option. Randomize the ordering of your options (most survey software can do this) to eliminate option order as a possible factor in responses. Of course, there are some options (like numeric scales and ordinal sets) which you don’t want to randomize so use your judgement.
  • Clarity is critical. While it might be tempting to ask long and detailed questions, customers will only answer questions to the extent they understand them. The easier a question and its options are to understand, the more likely you are to get the true answer instead of a best guess.

The great part of structured questions is that the data is very easy to work with and gives you immediate insights. For example, we know that the majority of readers review their metrics on a weekly basis without having to do any additional data processing:

frequency

Tomorrow we’ll review how to handle open questions where the responses are more interesting but much harder to analyze.

 

Quote of the Day: “Sometimes the questions are complicated and the answers are simple.” ― Dr. Seuss