How text analytics is changing the employee survey

In my last post I discussed the process by which employee engagement increases and decreases and therefore what are the most effective interventions leaders can use to make a long-term difference. In this post I explore the related question – what do we need to change?

As People Analysts OrganizationView have conducted a lot of surveys over the last 6 years. One thing we’ve come to appreciate is how important employees’ open text comments are to driving action from management. The issue has usually been how to deal with tens of thousands of comments, provided in multiple languages. This was one of the driving forces behind our decision to develop Workometry, our employee feedback platform.

Before the advent of reliable text analytics, analysing these comments at scale was time and cost prohibitive. Much of the way employee surveys have developed is because of this constraint. However, just in the same way that statistics has changed by the availability of large, easily available computing power and cheaper acquisition costs for data we predict that understanding employees will change by the availability of text analytics.

With text there are two main tasks that we want to do: we want to categorise the text into one or more topics and we might want to score the text on some sort of scale, for example a level of sentiment.

When categorising we want to go further than just words. We look at two dimensions – the topic (eg ‘career development’) and the context (eg ‘shortage’). This has to be more than just keyword as we’ll want to link together the multiple synonyms – it’s the meaning not the words that they’ve chosen which is important.

Doing this is adding metadata to our text. We can then apply various statistical techniques to the metadata. Typically we’re identifying in the region of 40 – 60 different topics for any text question. We can think of this as equivalent to adding another 40–60 scale questions to a survey. Therefore we can ask the short surveys that are needed to maintain response rates when you’re doing them frequently whilst capturing very rich data. We use an unsupervised learning approach meaning that the topics are suggested by the data, not precoded based on generic results.

One of the reasons that we do analysis is to draw attention to the parts of the information that managers need to focus on. We’re currently doing that by combining two techniques.

One of the joys of working with Employee data is that we often have extensive records about the employee – who they are, what role they’re doing, who they work with, how they’ve performed and the key events (job changes, pay rises etc). Linking this data to the perception data lets us provide much greater insight than if we just analyse the much smaller survey data on its own.

With Workometry we use probabilitic models to identify which groups are most likely to discuss a certain topic. We can incorporate information from HR systems, business systems and even other survey answers provided at a different time – for example looking at their perceptions to their onboarding process.

The results from these models can then be used within interactive visualisations to guide the user to the most interesting groups / results. The interactivity lets the user drill down into the data in new ways, guided by the models and ultimately lets them see the underlying text comments.

One very simple, but from our experience very powerful, way of looking at text data is to identify which topics are most likely to be discussed by engaged / disengaged employees. We see clear patterns that demands for work become far more transactional as a result of people moving from engaged to disengaged. This fits with information we get from exit interviews. We can think of a two stage process – first they become disengaged & then some leave. This supports the systems dynamics model I discussed in the last post.

Finally, what we’ve learnt from looking at vast quantities of text in this manner is that in a survey information in text comments seem to change much more quickly than scale-type questions. For one client we saw a jump in comments about technology 3 months before we saw changes in a related scale question. This ability to provide information to enable managers to fix issues whilst they’re still emerging should be seen as a key benefit of collecting and analysing text.