When to ask open vs closed questions in employee surveys

When to ask open vs closed questions in employee surveys-01.png

The traditional employee survey has a selection of closed or scale-based questions. These might be asked as a statement which the respondent has to agree to – usually on a Strongly Disagree to Strongly Agree scale, or a numerical score such as the 0–10 scale of an employee Net Promoter Score question.

We believe that in the vast majority of times such closed questions are used not because they are the most effective or appropriate way of gathering information, but because they historically have been the easiest to analyse.

With a scale-based question the respondent provides a response by selecting an option from a list. With most questions they are able to select a single answer however it could be possible to select several items from a list of alternatives. Because of the way the way the surveys are administered it is easy to count the number of people who answer “Strongly agree”, “Agree” etc and calculate a percentage.

Open questions however provide text data. This is typically described as ‘unstructured’ and requires effort to transform it into ‘structured’ data before it could be analysed. Until recently the only reasonable way of doing this was for a skilled researcher to do it manually. It is a long and slow process and is subject to a number of biases and issues with consistency (both between researchers and over time).

We have now reached a stage where computers are able to perform the coding task. The cutting edge algorithms, such as the ones we use in Workometry, are able to out-perform humans at coding text responses to open questions. They’re also able to do it at a tiny fraction of the cost of doing it manually meaning that cost and time becomes less of a deciding factor between using open questions and closed questions.

When you do this you get ‘count’ data plus the option of various other elements of metadata (data about data) such as whether the comment is offensive, is an outlier (some of the most important comments are often outliers) or even something simple like word count.

The choice of which question type to use therefore depends much more on why you are asking the question and what you want to do with the answers

Objective

The first question to ask yourself is why are you collecting the data.

Scale questions are best for providing a quantitive response from all respondents about a particular topic. They are therefore best for things such as KPIs which you need to track accurately on an ongoing basis.

The other way you might use a closed scale question is to test the strength of feeling on a particular topic, for example to provide as accurate value as possible for use in a business case.

If you need to identify how strong a feeling is you should use a scale question. However if your aim is to understand what people are thinking, why they are thinking it or how it affects them (or your business) you probably want to use an open question which provides text.

Open questions provide information which enable you to understand how to act. Closed questions might indicate you might need to act but won’t tell you how to do that.

MECE

A key principle when thinking about collecting information via closed questions is that you’re ensuring that the topics or reasons are Mutually Exclusive and Completely Exhaustive. Of this I think the most important is that the categories are Completely Exhaustive. Mutually Exclusive is important but having a hierarchical structure – ie that a category can be a sub-category of another – can be useful.

In some instances having completely exhaustive categories is easy. I could ask people which is their favourite Shakespear play as the number of plays is finite and reasonably small. My list could quite easily be Completely Exhaustive.

An alternative way of thinking of categories is at the question level. With an engagement survey historically we’ve asked a set of questions that are used to create an engagement index and then a large number of questions that are used to understand which factors, or categories, correlate with engagement. You can think of all those questions – ‘is communication good?’, ‘does your manager support you?’ etc. – as covering all the categories. The reason these surveys are so long is that there are so many possible categories.

If I want to know why someone did something it is impossible to build a Completely Exhaustive list. Some closed questions on survey might have an ‘other’ choice where the respondent then writes in their answer. Alternatively there might be a single open question at the end for the user to add anything that hasn’t been asked. Really this is saying ‘we don’t have a completely exhaustive list’. Unfortunately we see that these uses of ‘other’ will provide different (lower quality) responses than if you just ask an open question.

Open questions are, by their nature, exploratory in nature. This means that when you ask them you’re open to the possibility that the answers are outside the group of categories you could initially identify. When we ask open questions in an engagement type survey we find that about 30% of categories that employees mention are ones that we’ve never seen on a commercial engagement survey. We see a difference between two companies, even in the same sector. The reasons are very personal and business specific.

Another way of thinking about closed vs open questions is that with closed questions you have to ensure you’re Completely Exhaustive before asking the questions; with open questions your answers are Completely Exhaustive automatically. This makes survey design much simpler and removes many issues with ‘validation’.

How much signal is enough?

Fortunately the topics and their frequency identified during a coding are not randomly distributed. With our clients, a typical open question will generate about 250,000 words which result in the region of 100 different themes. The most common theme might appear 10% of the time where the smaller themes might appear less than 1%.

As the data size increases two things happen: first, the number of statements where we can identify a meaningful topic increases. The first or second time the algorithm spots something could be an outlier but after a few more instances we start to have enough signal to determine that this is a meaningful topic.

The second is as you get more and more data the confidence that you can safely assign to any ‘answer’ increases. You can start to consider tracking usage of topics over time. You can start to see which parts of your employee population are far more or less likely to talk about a particular topic.

Sometimes the topics are tightly distributed. With one client we saw a few people raising issues about bad contracts. Whilst in many organizations this might be ‘noise’ in this organisation the topics were all from one group and about one contract. By highlighting this statistically the management team could investigate and de-risk the situation.

What open questions don’t do is find a quantitive score against all potential categories – they don’t allow you to understand what each person thinks about each category. Instead they identify what is ‘top of mind’.

Respondent burden

As I’ve written about before, with data and analytics you need to think about what’s in it for the employee. Surveys and questionnaires are just the same.

There are three aspects for improving the respondent experience that I think you need to consciously try to improve:

  • The technology interface of the survey tool – how easy is it to provide the answers, especially across different devices including mobile
  • How long the respondent will have to devote to providing feedback
  • Whether the respondent will be able to tell you exactly what is on their mind.

On the first of these points we did worry about whether respondents would find it difficult to respond to open questions on a mobile device or whether responses would be shorter. At the moment we’ve found little evidence (though the 500+ word answers are mostly done with a proper keyboard).

For the second, to collect the richness of data a questionnaire which is based on closed questions inevitably needs to ask at least one question for each topic. Hence we either have the traditional long surveys, or we are forced to abandon data quality to provide a shorter experience. With a 4 question open question survey we find the average time to complete is less than 5 minutes.

Finally, open questions are the only way of ensuring that all issues are captured. Ideally with closed questions you’d want to ask about each category ‘how do you rate this’ and ‘is this important to you’. For example you might ask all employees about whether they think the firm offers family-friendly policies or benefits, but if a respondent doesn’t have a family they might not care (but could rate it as true). Many surveys assume that each category is equally weighted where this assumption is highly unlikely to hold.

As previously noted, when we’ve used open questions instead of scale questions in firms we’ve found that only about 70% of the important topics were typically part of a traditional employee survey.

Conclusion

Although we’re very strong believers in the importance of open questions linked with AI-linked analysis it’s clear that the best solution is a balance between open and closed questions, using both for their strengths.

In terms of information-gathering the vast majority of questions should be through open questions as the principle aim should be to identify issues that need fixing, or ideas that will be implemented. However, it’s important to have a quantitive measurement to capture your KPI on each survey. This data is very useful not only for tracking trends, but also for analysis.

The key point is that if your aim is to take effective action you should only use closed questions where absolutely essential. After all, if you want to really understand what people think you don’t ask a series of closed questions.