Survey Design: Open Questions

This is part 3 of our series on Survey Design, previous segments are available in our archives.

Multiple choice questions are easy to create and easy to understand, but sometimes you need to know more about your customers than a few options can represent. In these cases you should ask open questions, where the respondent can write as much as they like in response to the question.

Unfortunately, the price for this deeper feedback is a lack of consistency in the answers and a much more difficult problem in analyzing them. For example, in the Data Driven Survey we asked “What is your title or role at your company?” and made it an open question. As a result, of the 46 responses the most popular answer was Product Manager with 3 responses! In fact, only 3 answers (Product Manager, CEO, Account Manager) had more than one response. That means most answers included a job title that was unique across all other responses.

Does this mean that everyone has a different title? No, it just means that different people in the same job might describe it differently given the chance. We had respondents refer to their job as “Product Owner” and “Product Manager – Software” at Software companies and as “Digital Manager” at media and energy companies. Those are likely similar jobs but they way they are described in different industries and companies are different.

Designing open questions is not very hard, but there a few traps to avoid:

  • Avoid questions that can be answered with a “Yes” or a “No”. Instead of expecting the respondent to elaborate on their answer, make sure the question requires a detailed answer. Instead of asking “Do you like our product?” ask them “What do you wish we would improve about our product?”.
  • Explain why you are asking. If possible, explain why you are asking a question. This will help the respondent understand what you are looking for in their response. If you ask “What problems have you had with our product recently?” and then say “We will use this feedback to improve the next version.” they know you are looking for product feedback instead of customer support feedback.

When you do get your results, analyzing them can be a challenge because the answers can vary so widely. In most cases you’ll need to manually process the results, but there are some shortcuts you can use:

  • Word frequency analysis. By simply counting the frequency that certain words appear in results you can learn some interesting things. For example, we asked “What Key Performance Indicators (KPIs) do you use to run your business?” and it was an open question. The most frequently used words in the responses were “Revenue”, “Rate”, “Conversion” and “Margin”. That doesn’t tell us which KPIs were popular, but it does tell us that you care a lot about revenue, conversions and your margins.
  • Word count analysis. By looking at how many words respondents use to answer questions, you can learn some interesting things. For example, when we asked “What KPIs do you use to run your business?” the average response used 7.3 words, while when we asked “What data do you look at EVERY DAY?” the average response was 5.3 words. This means that companies are likely not looking at all of their KPIs every day or else the answers would be the same!

Still, there is no substitute for having a person read through the open answers and interpreting them. This means you should be careful about the number of open questions you ask and the sample size of your survey because you need to ensure you have enough time to review the answers. Otherwise you may have a lot of responses that you don’t have time to understand.

Why did we ask about KPIs as an open question instead of just giving you a multiple choice list of options to choose from? We were interested in how companies describe their own KPIs, just like we were interested in how people described their job title. Sometimes you care more about how customers describe themselves than having structured data that is easy to analyze.

Tomorrow we’ll talk about how to avoid bias in the responses by thinking critically about who is filling out your survey.

 

Quote of the Day: “I refuse to answer that question on the grounds that I don’t know the answer” Douglas Adams