How to ask for and receive better employee feedback

Over the last 18 months that clients have been using Workometry we’ve collected hundreds of thousands of rows of employee feedback, for numerous large organizations. Although we haven’t been able to randomly assign different questionnaires to different employees to run a ‘scientific’ test we are seeing some clear patterns into what makes a successful request for feedback or ideas.

Show that you’re listening

Responding back to employees quickly, even if it is a short summary communication, with an overview of the results and how you’re going to act makes a big difference to the effort people will take to respond the next time.

When you’re using open questions it’s useful to summarise the messages that you were provided. Many employees don’t expect management to read their comments – we even have seen employees start comments with ‘I don’t expect anyone to read this but…’

Show that you’re the capability to understand what they’ve said even if you’ve asked thousands of people. Avoid using a simplistic approach like you’d get on a wordcloud, but instead summarise the comments (think ’shortage of career opportunities’ not ‘career’ or ‘opportunities’).  You should be able to understand the context. Use poignant examples of quotes where possible.

Try and communicate this within a day or two of the feedback period closing. Of course ensure that you’re giving an honest overview of the responses.

Ask as few questions as possible – you’ll get better data

Nobody likes filling in long surveys and we certainly see an inverse relationship between the length of the questionnaire and the quality (measured by word / sentence length and topics identified). The more qualitative questions you ask the less information people provide in the open questions.

We believe that when you really want to understand what someone thinks you ask open questions. Furthermore our algorithms learn the topics from the text provided meaning you don’t need to pre-determine what the possible answers will be. Hence, we believe that text data is richer data and more accurately reflects the provider’s views.

If we consider each topic identified as replacing a quantitive question then one open question can ‘replace’ up to 50 quantitive questions. With this in mind you need to ensure that you only ask quantitive questions where a quantitive question is essential.

As an example, on a recent questionnaire we had a median answer length of 16 words with the maximum answer to a question being over 500 words. For a different questionnaire for the same client which was more ‘traditional’ we had a median answer length of about 7 words.

Lead with the open questions

Most of the time people design questionnaires with the open questions last. In the worse case the question might even be a ‘catch-all’ such as ‘Is there something we didn’t ask which you think is important?’

Recently we had a client who turned this approach on it’s head. They asked 4 great open questions FIRST and then asked a few scale questions after. The quality of the responses that they got was superb. It sent a message that these were the most important questions.

Ask for the good and the bad

Text algorithms have progressed considerably over the last couple of years however you always can build more accurate models if your constrain your text. One way of constraining the text is asking employees what’s they think is good and what could be improved as two different questions.

We code each question individually which gives Workometry its remarkable accuracy (but also means we need decent data sizes hence working with large organisations). What we see is if we’ve asked for positive and negative separately, and code the answers as two different questions we get a better model than if we combine the text and build a single model.

Consider breaking from the 5 point Likert scale

Employee surveys have almost always used Likert scale questions, usually with 5 points ranging from ’Strongly disagree’ to ‘Strongly agree’. Whilst there are benefits from using them – not least familiarity – we think you should question whether they’re always right.

Increasingly, as an analyst, I’m enjoying working with data from other question types, especially when we use Semantic Differential scales. These are questions where two polar-opposite adjectives are used and in our implementation the 11 points between the extremes are numbered.

The data we get from such questions is richer than from Likert questions. We treat Likert data as strictly ordinal (so no calculations like taking a mean) whereas given the alternative uses a numbered scale we believe you can assume the data is interval.

What we certainly see is that the data to most scale questions are not normally distributed.

• Wikipedia’s post on Semantic  Differential scales
• This post compares Likert vs Semantic Differential Questions

Make sure you have at least one target variable (which isn’t qualitative)

Asking open questions is great but don’t abandon all scale questions. The really powerful feedback is when you can combine quantitive and qualitative data.

Each time you ask for feedback there is probably one or two things that you really want to understand how strong the respondents feel. For example if you’re asking for the quality of service you’re providing you might ask:

• ‘On a scale of 0-10, how well do you think we’re performing?’
• ’During the time you’ve been dealing with HR has the service we’ve provided got better or worse?’

With these two variables you’ve got the ability to understand the (maybe 100 or so) different themes from your open questions by things such as the average or variance of people providing certain responses. Alternatively you can look at the top / bottom quartiles on each scale question and identifying which themes these groups are more likely to use.

Adding a small number of scale questions gives your analysis and interpretation of the results purpose.

Give people results directly

One of our clients, after using Workometry on a monthly basis for a year, decided to ask their employees for feedback on the feedback approach.

One of the clearest, and most surprising results was that employees wanted to be sent the relevant results directly. Whilst there was some evidence to suggest that this was particularly strong for teams where the manager didn’t take time to discuss the results the feedback was pretty consistent across all parts of the organization.

These employees wanted to have the results emailed to them directly.

Embrace mobile devices

Several of our clients work in environments where a large proportion of employees don’t have access to computers. In these environments employers should embrace the opportunities that communicating to employees using their smart phones.

We’ve found that such employees, who are often at lower levels in the corporate hierarchy, not only have universal access to smart-phones but also are likely to acquire the newest phones the quickest. Maybe it’s because they aren’t in front of a computer all day that they invest in good mobile technology.

We’re currently not finding much difference in the quality of answers people provide on mobile phones which had concerned us.

At the other hierarchical extreme, executives like giving feedback on iPads.

Think of your questionnaire as an interview, not a survey

Twenty years ago I used to head recruitment for a fast-growing consulting organization.  What I realised as I created and used structured interviews for the firm to use was just how much you can learn from a single, well thought-out open question. It’s rare that you need to ask more than a couple of follow-up questions.

We’re learning that you should think of your feedback questionnaire like you do an interview. Ask a small number of open questions. Give the respondent the opportunity to present their ideas in their own words and ensure that you have enough information and context so you feel confident acting on what you hear.

With feedback often the most valuable is when you get information you’d never have expected. This has always been a problem with survey design – though if you don’t ask open questions you might never realise it. With recruitment interviews the best people I met were often those that I didn’t expect to be exceptional.

Only by starting with an open mind and asking well-thought questions do you really get the best information.