Sentiment analysis is frequently used on employee text but how useful is it in practice? If it is to be used what do you need to think about?
Employee-sourced innovation can deliver significant economic value for firms. In this article I describe how technology and analytics can support these activities
Nick Lynn’s new book ‘Employee Experience Leadership: build trust through employee experience and engagement’ is a fantastic read. Nick’s background as a practitioner and consultant, working with senior leadership team to implement employee insight and insight-driven changes is apparent yet this not a book designed to sell more consulting assignments but a comprehensive and information-dense guide to what is for many a new topic.
Over the last 2 years we’ve seen a big shift in HR conversations from a narrow focus on employee engagement to a wider one about employee experience. Whilst this change has been widely accepted the implication for what this means in terms of measurement hasn’t been considered in such detail.
More qualitative data is needed
Unlike employee engagement, employee experience isn’t grounded around such a formal, psychology-based concept. Understanding experience demands a much more exploratory-based approach which of course demands much more qualitatively based measurement.
When I first got involved in what would now be called ‘employee experience’ almost 15 years ago as a quantitive analyst I was initially surprised not only about how my colleagues in customer experience used a much-more qualitative based research approach but also how effective this qualitative data was at driving effective action.
A simple example were the user tests we’d conduct on the careers site and tool that we designed / implemented. I was advised by our researchers that large numbers of different individuals weren’t necessary to run an effective study. Of course we ensured we recruited well and tried to reduce bias but I was surprise at how quickly the results started to converge. In terms of identifying key areas to change we probably had found most before the 10th person.
Another thing that I rapidly realised was that the issues that we identified were rarely the ones which we had expected. We went in to the testing with a set of ideas to test, often building multiple versions to compare, then found that something completely unexpected needed fixing more urgently.
The data that we captured was typically much richer than a traditional quantitive approach. Quantitive analysis can be very good at identify what, when and how but it’s very poor at helping understanding why - i.e. the underlying causes. From the perspective of improving experience the critical part is to understand ‘why’.
Measurement and service design
Within service design, which we believe employee experience aligns closely with, an approach of going in with an open mind and asking simple, open questions will typically outperform analysing more structured, quantitive data.
We’re trying to understand the events - often called ‘moments that matter’ - that create the positive or negative experiences so that we can systematically improve these touch-points (or reinforce the good ones). Understanding experiences is inherently an exploratory approach. We need to start with an open-mind to understand the experiences from the users’ perspectives.
Traditional surveys are ineffective
Traditional surveys used in HR (and old-school customer surveys) take a constrained view of what could have happened and then offer each possibility as an option.
We all know not only how ineffective these can be and how frustrating. I guess everyone has seen a customer survey which asks question after question about an experience (eg a hotel stay) but fails to ask about the one event that you want to comment on.
Not only do these designs fail to capture the relevant information, they also provide a poor usability and make a poor experience feel worse.
Over the last 15 years these types of surveys have become less common as firms have transitioned to simpler feedback methods, usually with a small number of scale questions and the opportunity to provide open feedback. Think NPS-type surveys or online rating platforms such as Expedia or Glassdoor. The common part of all these is that they use open-text instead of closed questions to capture the majority of the information.
Quickly addressing issues
If we look at mature customer experience efforts we see a strong (often company-wide) desire to close the loop quickly with raised issues, regardless of the channel they come in from.
In the hotel industry a range of platforms have been developed to enable managers to capture feedback regardless of where it is left (guest surveys, social media or one of the many hotel-review sites) and ensure that the guest is acknowledged quickly and any issue addressed. Many hotels will have daily ’meetings for management to review issues and identify solutions.
This approach has two key benefits, both of which apply to employee experience. First, the issue can be addressed in a timely and effective manner. Second, responding quickly minimises the chance that issue becomes bigger than necessary via word-of-mouth.
Multiple levels of action
With employees action often needs to come from various sources. Some interventions can be dealt with at the manager-level. Others might require escalation to various internal teams, eg IT. Finally there are a class of issues which are most effectively dealt with at a senior level.
If HR want to effectively manage employee experience I believe that they need to build an approach similar to the management meetings in hotels. They need to build cross-organization groups to ensure issues are addressed quickly and effectively.
When I managed the new employee experience program at UBS we created governance groups to ensure a multi-stakeholder approach. IT created a special team to manage global technology provision and we co-designed processes to bridge the gap between HR and IT and ensure that even though managers were still responsible for ordering, if IT hadn’t received the order in sufficient time they could proactively contact the manager. We also created technology that could ensure that different members of our cross-business governance group could be responsible for their content whilst providing one simple coordinated communication.
With employees it’s not always a good idea to devolve action to managers. Employees will often choose to remain silent rather than providing voice if they don’t feel safe. With the manager so important in creating, and destroying, employee experience there is an opportunity for a employee experience team to act as the filter, viewing the feedback and allocating it to the correct individual, whilst preserving confidentiality if appropriate.
On a regular basis there is a need to take a top-down view of all feedback. Statistical analysis and ‘AI’ can provide badly needed objectivity to understanding how themes are distributed, where outliers lie and how feedback is linked. This objectivity is vital to ensure that decisions are effective. The technology ensures that such work can be done in a timely and cost-effective manner.
Ask for ideas and issues
Many in the customer experience space suggest asking a simple ‘why did you provide that answer’ type question as the open question. After studying millions of answers to employee questions we believe that there is a better way.
Some of your most engaged employees will be the ones who identify the most important things to change. Conversely it’s important to understand what those who are disengaged or unimpressed value.
Our experience is, just like how good performance feedback can be given, it’s best to ask everyone ‘what is good about x?’ and ‘what could we do to improve x in the future?’
This effectively doubles the amount of feedback you receive whilst reducing bias in the data. It makes analysis easier and adds little additional burden on the user.
The advantages of asking many people open questions
At the beginning I suggested that qualitative studies typically required less participants. The key reason was that the ideas tend to converge quite quickly.
The main reason for not asking lots of people is the cost of doing so. Manually themed text is time and resource-intensive so it made sense to do as little as is needed to get good results.
However there are advantages of getting more data. Themes and ideas have a long-tail distribution and the more data you have the more you can look down that tail.
At the same time if you want to understand the distribution or relationship between feedback and ideas you can extract more insight with more comments.
Modern text algorithms dramatically reduce the time and resources needed to get human-level accuracy. They therefore offer more opportunities to collect and comprehend ideas at scale.
In July 2018 the Financial Reporting Council published its long-awaited update to the UK’s Corporate Governance Code. We’ve seen considerable coverage of this in the legal press but very little for HR. This seems strange given that those in HR are likely to have to implement it.
The new regulation applies to firms with over 250 employees in the UK.
What the regulation states and what it doesn’t
The new UK Corporate Governance Code states:
For engagement with the workforce, one or a combination of the following methods should be used:
a director appointed from the workforce;
a formal workforce advisory panel;
a designated non-executive director.
If the board has not chosen one or more of these methods, it should explain what alternative arrangements are in place and why it considers that they are effective.
The challenge with all of these methods is how to capture the views of the employees, to synthesise them and provide them to whichever party that has been designated to include them in the board decision process.
Of course the workforce is highly unlikely to have an uniform voice. Therefore it is essential to capture this diversity of ideas. The formal approach used to decide how to handle potentially conflicting ideas isn’t part of this article.
The challenge of upward communication
As an executive you’d like to hear about all the issues that need your attention. By ‘your attention’ I mean that either they’re things that you don’t want to be surprise with when presented from another source or things that for whatever reason need you to take action (maybe because they are cross-organisational or need your level of authority).
The problem that you’re faced with is twofold:
Only a proportion of the topics you need to hear about will reach you ‘undistorted’
Due to communication windows it will take time for these topics to reach you.
Let’s take the example of a business with 8 layers between the executive team and the folks who deal with the customers.
If a high 80% of topics that you need to hear about get through each level then this implies you’ll only hear about 21% of topics
If a more realistic 50% get through each level you’ll hear less than 1% of all issues
Of course you’ll probably hear about the the wrong topics. Because many of the issues might seem ‘too petty’ they won’t be raised. However if 500 people across the organization have this issue it probably demands your attention. Small and widespread often doesn’t get through, whereas big and niche often does. Often the real value is in fixing small and widespread issues.
The other key issue - one that we frequently see and one that is well-researched in the academic journals - is that employees often don’t raise issues - so called ‘Employee Silence’. It can be more difficult raising issues up through the chain, where it might threaten their managers than to raise it confidentially to a central point. The sort of issues here might be to do with cultural risks such as poor incentives or behaviours.
Executives often think about this as needing to facilitate ‘whistle-blowing’ or raising of issues. This is, of course, important but these instances are rare and high value - the ‘rare and niche’ that I mentioned earlier. In truth these cases need special channels and need to be treated differently than other forms of employee voice.
The real challenge is not how to find the rare event with a clear channel, it’s finding the diversity of opinions and ideas about something widespread.
Often the true value in integrating employee opinion into decision making is to understand the distribution of ideas and opinions from a large population. It’s about asking all of your employees about a decision and understand the breadth and distribution pattern of the responses.
Of course some of these ideas will be truly innovative and potentially super-interesting but often you’re trying to get a good grasp of the top ideas.
The need for qualitative information
There is a time for getting quantative information - mostly when you already know the problem and are looking to understand the scale of it. If you know all possible options, a closed question might be the best way of putting a number on it.
In most instances however you don’t know all aspects of the problem, or haven’t identified all the possible solutions. When you’re at this exploratory stage then it’s best to ask open questions.
Sufficient, reliable, relevant and useful
Adopting a concept that comes from internal audit:
Information is sufficient when there is enough of it to come to an unbiased and dependable view of the topic. If you’re asking all your employees about a subject and a meaningful number of employees raise an issue then it’s probably sufficient.
Information is reliable based on the source of the information. Customer perception data from people working day-in and day-out with customers is potentially more reliable than information that comes indirectly. Information that can be validated is more so.
Relevant information is information that relates directly to the question explored. When we ask open questions to employees relevant information is that which relates to the topic. We need to down-weight information that we’ll receive but doesn’t help us answer the question (some people have something that they will raise regardless of the question asked).
Useful information is information that helps an organization meet its goals. If the information was about an old reporting system which has since been phased out, it probably wouldn’t be useful (because the issue has already been dealt with.)
Employee feedback provided via free text can be all of these things, though it might gain extra reliability if linked to existing data. It’s important when reviewing the summarised feedback that managers assess it against these 4 lenses.
The need to synthesise
Once you’ve decided to ask your employees a series of open questions about a key topic or decision, what do you need to do?
For many of our clients if we ask a couple of open questions we’ll get on average 20 words per question per employee. This means for each question with 50,000 responses you’ll be faced with 1 million words.
The problem is that you probably need to present less than 1,000 words to your executive team, ideally 1 side of A4 paper.
How do you do this?
Technology to the rescue
Historically, analysing a large amount of free text was a long, expensive process. The quality that you’d get out of this also was probably lower than you’d imagine. Getting 80% agreement between reviewers is pretty good. It’s really hard for reviewers to be consistent throughout a review. Identifying a new category means having to start again.
There are several different capabilities that you need to have, most importantly the need to understand themes - context rich descriptions - rather than topics. Historically this level of understanding has been hard yet with the progress of text analytics over the last couple of years the best machine learning approaches can match human performance in many tasks.
What you will need your technology to do
When considering an algorithmic support to enable you to collect and understand Employee Voice you need to ensure that your text analysis tool can deliver several key capabilities
1) Ability to ask any question, and to analyse the responses
We do not believe that Employee Voice - the ability to let employees contribute to decision making - is possible without being able to ask and understand questions about the decision you need to make.
The first, and arguably the most important requirement for any Employee Voice technology is that you can ask any question, and the system can theme the answers with a decent level of accuracy.
This might seem obvious but it’s not. The best text analytics approaches work on very narrow types of text. A system might be great at parsing a CV but couldn’t understand a shopping list for example. To get the level of accuracy that you need we think you probably need to have a model fine-tuned at a question level.
2) Themes not topics
As mentioned before it’s important to understand the themes - how people describe their views - not the topics. So ‘more transparent communication’ instead of simply ‘communication’.
The algorithms should provide summaries the answers. Only if you can understand the underlying meaning just from reading the theme label then it’s probably good enough.
3) Identify notable and unusual answers
Another key aspect is the overall pattern of themes, both in terms of overall distribution and ‘hot-spots’ of feeling across the organization and employee population. Often you’ll need to identify the rare comments that might bring genuine innovative ideas (or tell you about a problem you really need to deal with).
4 Track asking and answering of questions
For compliance purposes you will want to be able to show who was asked, when they were asked, how the feedback was analysed and how the information was integrated in the decision making process. Technology is well-suited to this task.
A process for working with Employee Voice
We do not believe that technology alone will enable firms to meet the requirements of using Employee Voice in decision making. However whichever way (or ways) businesses decide to use to bring voice into business decisions it’s clear that technology can significantly improve the cost, responsiveness and quality of the process needed to maximise benefits and demonstrate compliance.
What we constantly hear from Workometry clients is that when executives experience the benefits of being able to consult their organizations quickly and effectively they want to use it more and more. We hope that this new regulation helps elevate employee voice into a standard business practice.
Taking it further
Earlier this year we published a Guide to Employee Voice. It includes a set of useful resources and documents for anyone trying to understand best practice in this area.
If you do have thousands (or even hundreds of thousands) of free text answers from your employees let us show you what is possible.
Large organizations have vast amounts of employee text feedback that so-far they’ve done little with. Often this information is the most valuable part of a questionnaire or survey yet understanding it at scale has historically been hard and time consuming.
The process of understanding this feedback is known as ‘coding’. What a good qualitative researcher is trying to do is to identify a set of categories that both cover as many of the comments as possible yet not so many that they become hard to navigate and understand.
The importance of the question and organization context
Feedback does not necessarily come in beautifully crafted sentences. The understanding of any answer to a question is highly dependent on the question asked. People write answers assuming that you know the question asked and view the feedback through the lens that this context provides.
Given this it makes sense that to develop a set of themes from the answer to a specific question it has to be done in the context of the question asked. General models are hard to make accurate. It’s why great text analytics models typically have narrow domain use.
In organizations you also need to understand the context of the organization. Organizations develop their own language use. Many times this is internal jargon, project and technology names. Sometimes it’s re-purposing words to have a different meaning. The employees of one of our clients talks about ‘payroll’ when most people talk about ‘headcount’ (because if they have more money for pay they can afford to take on more staff). Really good models need to learn this language use.
Themes not topics
A good set of categories should be understandable without needing to read the underlying comments (though of course you will likely want to let users drill down to these if they wish). The theme labels need to be sensible answers to the question asked.
If we look at typical categories that survey tools are starting to provide in their text analytics, if the themes are simple, one-word nouns then users will almost certainly have to read the comments to understand what it is about that topic that they are discussing. Noun-only topics are good for navigation - guiding the user to read a subset of comments - but poor at providing insight.
For helping drive decision making what is needed is contextual understanding about what it is about that topic which is of interest. So instead of ‘Communication’ we need to describe that it is ‘more regular communication’, ‘less communication’, ‘more transparent communication’ etc.
It is quite common in our analyses that we’ll find conflicting themes within a particular topic. We’ll find that some employees want ‘more hours’, some will want ‘less hours’ and another group will talk about ‘more consistent hours’. If you’re just looking at the topics - ‘hours’ - all of these themes will be grouped together. To take action you need to understand the context.
Semantic understanding rather than keywords
Early text coding systems used keywords to identify topics. You can even do this with a simple spreadsheet application like Excel. Wordcloud applications typically use this approach.
What is important when creating good themes is to use a semantic understanding of the comment, not just look for certain words. So an employee might ask for ‘more pay’ or ‘better wages’ or a ‘salary increase’. To a human reader these would all fall into the same theme. Our algorithms even recently allocated ‘ha ha, a raise’ to be about improving pay.
The bar for usefulness is quite high
When creating a good set of categories of it’s hard to be useful until you get to a stage where you’re able to identify most of the useful themes and that these themes are summarisations of the text they contain.
We’ve found from clients that the level of coding accuracy has to be high before it’s seen as valuable. In our instance, where we’re trying to replace quantitive, closed scale questions with open-questions it’s important to have good quality themes that accurately reflect the semantic and contextual meaning of the answers.
Whilst providing a list of topics might seem a good first-stage approach, the reality is that it doesn’t replace reading the comments and therefore doesn’t add much value.
Deductive and Inductive coding
There are two general approaches to doing a coding - deductive and inductive.
Deductive coding starts with a predefined set of codes. It’s what most employee engagement apps and even consultants typically use. You start with a model of what people will likely say and then you allocate their topics to these categories.
There are a few advantages with deductive coding:
- You know that you will identify comments in the themes in which you are interested
- It’s easier to compare results across different organizations
However there are several key issues:
- You might miss key themes. From our experience, about 30% of topics from a typical employee question might not fit into a general model
- The coding model has bias as the themes are based on an analyst’s model, not the views of the employees
- It takes more time / resources up-front to decide what themes to look out for
- A model really only performs well with one specific question
The alternative to deductive coding is inductive coding. With inductive coding you start from scratch, using the data as the source of the themes. With inductive coding you can understand the answers to any question.
The typical process for creating an inductive code manually is:
- you read some of your responses
- you identify themes (codes) that cover this sample
- you code your all of your responses to these themes
- you identify the comments you weren’t able to code and identify new themes
- you then recode all your responses again against the new full set of codes
- this whole process is repeated until it’s not possible to identify new, useful themes.
This is, at a high level, how we create the themes with Workometry, apart from instead of doing it all manually a set of algorithms helps with the coding.
Balancing accuracy and coverage
In any predictive model the analyst has a choice between false positives and false negatives. In the case of text classification, if you want to increase the accuracy of your coding (reduce false positives) you increase the risk that you’ll miss some comments in your themes (increasing false negatives).
Our experience of text coding is that readers are much more sensitive to a comment being misclassified than they are about being said that it hasn’t been coded. As a human we tend to be much more understanding if someone says ‘I’m sorry I don’t understand’ to them answering the wrong question. The inductive coding process is iterative to try and create the needed balance.
Whilst coding might seem to be a task which everyone can perform even humans find it difficult. Several studies have shown that when you get several skilled coders to theme the same piece of text you'll only get agreement about 80% of the time. Consistency is even problematic at the individual coder level as their opinions will change as they review more feedback. AI-based solutions will be more consistent than a skilled human.
With volume you get granularity
We often get asked ‘how many different topics will you find?’ The answer is almost always ‘it depends’.
However there are two key factors that it depends on: the question and how many answers that you receive.
With the question one of the big determinants is whether you’re asking a ‘what is good’ or ‘what could be improved’ type of question. Typically you’ll find less different themes in the former than that latter.
To understand the likely number of themes it’s worth understanding how the volume of responses matching a theme tend to be distributed.
If you plot the frequency of mentions for the different themes in your question and order the themes largest to smallest you tend to have an exponential-type distribution.
If instead of plotting the number of mentions for each category you plot the logarithm of the number of mentions the curve becomes very close to a straight line. The area under the curve represents the total number of mentions.
As long as the number of themes per answer stays the same as you increase the volume of answers then the effect of this is that the curve representing the relationship moves out as shown. The implication of this is that both the mentions of any theme increases and the number of themes identified increases.
Another way of thinking about this relationship is that as you get more comments you start being able to see further down the ‘long tail’ of theme groups.
Combining with other data
Understanding the themes within the data is important but where it starts to really come alive is when you start looking at where the use of certain themes is particularly high or low.
We have an automated detection approach which scans through groups looking for unusual incidents of particular themes.
One example of this would be to understand which themes were unusually likely to be used in various countries or functions. Often the theme might be expected (eg finance people talking about the finance system) but in other instances it might reveal areas for further exploration. We believe that managers should be managing exceptions.
Another use is to identify which themes are closely related to each other - ie co-occurrence of themes. For example you’d want to know that if the temperature in the stores was closely related to comments about customer complaints.
If you’ve been asking open question feedback as part of a mixed question type survey you might want to build a model which links the provision of your themes with the scores on various questions. This can help demonstrate what is linked to engagement, for example.
Finally when text is structured it can be included in any other predictive model. You might want to understand the themes that people talk about which later drives attrition for example. Our experience is that these data points are often some of the most predictive variables, and the easiest to action.
How Workometry codes answers
Our Workometry solution uses an inductive approach to coding feedback themes meaning that it can be applied to the answers of any question. We learn the themes for a specific question in a specific organization and can understand multi-language feedback. Regular clients benefit from our ability to learn organization-specific language and embed earlier knowledge in themes.
As an augmented intelligence solution, Workometry scales very well and many clients are dealing with tens, or even hundreds of thousands of comments to any question. Our key differentiator is to be able to build highly accurate, inductive models in a very short time.
One of the greatest benefits of using a text-centric, exploratory approach to employee feedback is that you will identify issues that you’d never have thought about asking on a traditional survey. In many instances these are ‘low-hanging fruit’ - easy to fix when you know what they are.
OrganizationView, the firm that I run, is based in the world-famous Swiss resort of St. Moritz.
One of the key trends that we’ve seen with Workometry clients over the last 18 months is the increasing use of Employee Voice as a contributor to enable innovation across global organizations. Leading firms are increasingly actively involving their employees in contributing ideas and identifying issues that need addressing, effectively using the huge cumulative insight as large-scale collaboration.
To understanding this trend in more detail I interviewed Yves Zieba, a prominent advocate of open innovation here in Switzerland. I hope that you enjoy it.
Hi Yves, could you please give me us an introduction to who you are?
I am a typical ‘slasher‘. I share my time between co-animating an open innovation space, facilitating workshops on strategic agility, design thinking or fablab experience. I am also coaching successful SMEs in the European Union Horizon 2020 program. We apply leading edge practices in facilitation with Seethrough and with business models for the open economy workshop from Without Model. It is our way to combine open innovation principles, facilitation skills on emerging HR trends or peopletech topics.
What is ‘open innovation’? How are firms using it?
It is primarily a mindset. It is often confused with open source. There are similarities, although it is not exactly the same thing. The idea is to co-create with the right customers, partners and providers. It is the exact opposite of what is happening in closed and highly secretive R&D departments. It can take several formats. What I see most these days are idea box, hackatons, public private partnership, problem solving contest, UX beta testing communities, or corporate venturing (i.e. large group working with small startups in different partnership formats).
What is the employees role in open innovation ?
It is to adopt this open mind, this equal-to-equal relationship, and to develop their ability to co-create, to work in flexible teams co-develop modular solutions with external stakeholders in a similar way than with internal stakeholders. Most of the time the open innovation transformation is taking place at the same time as the digital transformation. So the employees role is to drive this change, or at least to understand and contribute positively to this change. To some extent, taking some risks is also part of the game. Employees can now take advantage of the emergence of new management technics and latest organisation design, such as holacracy.
When we facilitate open innovation or buisness models for the open economy workshop, I am surprised to realise how little most people know about these principles or about creative commons licences.
What have been the challenges to using employee voice in this manner ?
In open innovation as well as design thinking the prerequisite is to adopt an equal to equal relationship. I typically ask everyone to adopt a beginner ‘s mindset and to start to « think together ». In reality, it takes time as existing hierarchies continue to carry their weight, and are making use of their business stamina in many organisations. So when we are using employee voice, typically in an « idea box » context, it is crucial to have transparent criteria for selection. We can also introduce elements of collaborative and participative approach to project selection and investment decision making. We plan ahead to be clear about the rules of the game (which idea will be funded, how, what will happen to the team, who owns the IP…). Companies sometimes fail, it typically occurs when they ask their employees to contribute and then fail short when it comes to implementing great ideas.
Who would you say has done this well? What benefits did they realise ?
There is a long list of organisations who have been using open innovation extremely well and successfully for years. To name a few, Quirky, for collaborative inventions, Samsung, Lego, GE with their ergonomic design challenge, Philips, on crowdsourcing in the healthcare and well being industry, Baiersdorf with the Nivea campaign, Procter & Gamble Connect & Develop campaign or Mozilla toolkit are some of my favorite examples. The University of Geneva Quality of Life Lab and the Geneva Lab togther prepare a great program with EnoLL, there should be lots of best practices being displayed at OpenLivingLabDays2018 during the week of August 23rd
These players have adopted what they like about open innovation to meet their needs.
The traditional employee survey has a selection of closed or scale-based questions. These might be asked as a statement which the respondent has to agree to – usually on a Strongly Disagree to Strongly Agree scale, or a numerical score such as the 0–10 scale of an employee Net Promoter Score question.
We believe that in the vast majority of times such closed questions are used not because they are the most effective or appropriate way of gathering information, but because they historically have been the easiest to analyse.
With a scale-based question the respondent provides a response by selecting an option from a list. With most questions they are able to select a single answer however it could be possible to select several items from a list of alternatives. Because of the way the way the surveys are administered it is easy to count the number of people who answer “Strongly agree”, “Agree” etc and calculate a percentage.
Open questions however provide text data. This is typically described as ‘unstructured’ and requires effort to transform it into ‘structured’ data before it could be analysed. Until recently the only reasonable way of doing this was for a skilled researcher to do it manually. It is a long and slow process and is subject to a number of biases and issues with consistency (both between researchers and over time).
We have now reached a stage where computers are able to perform the coding task. The cutting edge algorithms, such as the ones we use in Workometry, are able to out-perform humans at coding text responses to open questions. They’re also able to do it at a tiny fraction of the cost of doing it manually meaning that cost and time becomes less of a deciding factor between using open questions and closed questions.
When you do this you get ‘count’ data plus the option of various other elements of metadata (data about data) such as whether the comment is offensive, is an outlier (some of the most important comments are often outliers) or even something simple like word count.
The choice of which question type to use therefore depends much more on why you are asking the question and what you want to do with the answers
The first question to ask yourself is why are you collecting the data.
Scale questions are best for providing a quantitive response from all respondents about a particular topic. They are therefore best for things such as KPIs which you need to track accurately on an ongoing basis.
The other way you might use a closed scale question is to test the strength of feeling on a particular topic, for example to provide as accurate value as possible for use in a business case.
If you need to identify how strong a feeling is you should use a scale question. However if your aim is to understand what people are thinking, why they are thinking it or how it affects them (or your business) you probably want to use an open question which provides text.
Open questions provide information which enable you to understand how to act. Closed questions might indicate you might need to act but won’t tell you how to do that.
A key principle when thinking about collecting information via closed questions is that you’re ensuring that the topics or reasons are Mutually Exclusive and Completely Exhaustive. Of this I think the most important is that the categories are Completely Exhaustive. Mutually Exclusive is important but having a hierarchical structure – ie that a category can be a sub-category of another – can be useful.
In some instances having completely exhaustive categories is easy. I could ask people which is their favourite Shakespear play as the number of plays is finite and reasonably small. My list could quite easily be Completely Exhaustive.
An alternative way of thinking of categories is at the question level. With an engagement survey historically we’ve asked a set of questions that are used to create an engagement index and then a large number of questions that are used to understand which factors, or categories, correlate with engagement. You can think of all those questions – ‘is communication good?’, ‘does your manager support you?’ etc. – as covering all the categories. The reason these surveys are so long is that there are so many possible categories.
If I want to know why someone did something it is impossible to build a Completely Exhaustive list. Some closed questions on survey might have an ‘other’ choice where the respondent then writes in their answer. Alternatively there might be a single open question at the end for the user to add anything that hasn’t been asked. Really this is saying ‘we don’t have a completely exhaustive list’. Unfortunately we see that these uses of ‘other’ will provide different (lower quality) responses than if you just ask an open question.
Open questions are, by their nature, exploratory in nature. This means that when you ask them you’re open to the possibility that the answers are outside the group of categories you could initially identify. When we ask open questions in an engagement type survey we find that about 30% of categories that employees mention are ones that we’ve never seen on a commercial engagement survey. We see a difference between two companies, even in the same sector. The reasons are very personal and business specific.
Another way of thinking about closed vs open questions is that with closed questions you have to ensure you’re Completely Exhaustive before asking the questions; with open questions your answers are Completely Exhaustive automatically. This makes survey design much simpler and removes many issues with ‘validation’.
How much signal is enough?
Fortunately the topics and their frequency identified during a coding are not randomly distributed. With our clients, a typical open question will generate about 250,000 words which result in the region of 100 different themes. The most common theme might appear 10% of the time where the smaller themes might appear less than 1%.
As the data size increases two things happen: first, the number of statements where we can identify a meaningful topic increases. The first or second time the algorithm spots something could be an outlier but after a few more instances we start to have enough signal to determine that this is a meaningful topic.
The second is as you get more and more data the confidence that you can safely assign to any ‘answer’ increases. You can start to consider tracking usage of topics over time. You can start to see which parts of your employee population are far more or less likely to talk about a particular topic.
Sometimes the topics are tightly distributed. With one client we saw a few people raising issues about bad contracts. Whilst in many organizations this might be ‘noise’ in this organisation the topics were all from one group and about one contract. By highlighting this statistically the management team could investigate and de-risk the situation.
What open questions don’t do is find a quantitive score against all potential categories – they don’t allow you to understand what each person thinks about each category. Instead they identify what is ‘top of mind’.
As I’ve written about before, with data and analytics you need to think about what’s in it for the employee. Surveys and questionnaires are just the same.
There are three aspects for improving the respondent experience that I think you need to consciously try to improve:
- The technology interface of the survey tool – how easy is it to provide the answers, especially across different devices including mobile
- How long the respondent will have to devote to providing feedback
- Whether the respondent will be able to tell you exactly what is on their mind.
On the first of these points we did worry about whether respondents would find it difficult to respond to open questions on a mobile device or whether responses would be shorter. At the moment we’ve found little evidence (though the 500+ word answers are mostly done with a proper keyboard).
For the second, to collect the richness of data a questionnaire which is based on closed questions inevitably needs to ask at least one question for each topic. Hence we either have the traditional long surveys, or we are forced to abandon data quality to provide a shorter experience. With a 4 question open question survey we find the average time to complete is less than 5 minutes.
Finally, open questions are the only way of ensuring that all issues are captured. Ideally with closed questions you’d want to ask about each category ‘how do you rate this’ and ‘is this important to you’. For example you might ask all employees about whether they think the firm offers family-friendly policies or benefits, but if a respondent doesn’t have a family they might not care (but could rate it as true). Many surveys assume that each category is equally weighted where this assumption is highly unlikely to hold.
As previously noted, when we’ve used open questions instead of scale questions in firms we’ve found that only about 70% of the important topics were typically part of a traditional employee survey.
Although we’re very strong believers in the importance of open questions linked with AI-linked analysis it’s clear that the best solution is a balance between open and closed questions, using both for their strengths.
In terms of information-gathering the vast majority of questions should be through open questions as the principle aim should be to identify issues that need fixing, or ideas that will be implemented. However, it’s important to have a quantitive measurement to capture your KPI on each survey. This data is very useful not only for tracking trends, but also for analysis.
The key point is that if your aim is to take effective action you should only use closed questions where absolutely essential. After all, if you want to really understand what people think you don’t ask a series of closed questions.
Depending on which data you look at employee engagement has tended to be flat or declining over the last 15 years. There is obviously something wrong with the way that firms have been attempting to manage engagement within their organizations. Our view is that the core issue is that engagement typically suffers flawed analysis leading to ineffective changes.
A simple way of thinking about the way most traditional surveys are designed is by considering the questions in two groups: a set of questions that make up the engagement index and another set, usually larger, which is used to identify factors that are correlated to that engagement index.
If we consider how this works in more detail we can see that what we’re doing is looking at the differences in views between engaged and disengaged employees.
Most engagement survey analysis is static – i.e. no knowledge of each respondents’ previous states or answers are considered when conducting the analysis. Where history is included it is usually at the sub-organization (eg team, department) level where there can significant differences in the underlying respondents from survey to survey.
During the analysis stage of a survey typically variables that are most closely linked with engaged / disengaged employees are identified. Unhelpfully these are often called ‘drivers’ implying, at least to the lay-person, that they are causal. They are not, they’re simply correlations.
What these analyses do is identify symptoms of being engaged rather than what causes the employee to become engaged in the first place.
There are several reasons this matters. The first and most obvious is that any post-survey action is likely to be sub-optimal. I can take a flu-remedy which prevents my nose running and tempers my headache but I still have the flu. As soon as the medicine wears off the headache returns.
The second reasons is that our sensitivity to certain issues changes as a result of our overall view. We probably all know people who cited reasons such as ‘they left the toothpaste lid off’ when explaining why they changed partners. When love disappears these things become more grating, but overall they aren’t the reasons why love is lost.
How to fix this
To understand the issues that are linked to employees moving from an engaged to a disengaged state (or vice versa) we need to include how any individual changes over time in our model.
There are two key ways of doing this. My view is that the ‘gold standard’ is to do both:
- We can look at the transition between engaged & disengaged at an individual-level and understand what happened to them. One way of doing this is to link employee background (education level, function, tenure etc.) and event data (eg pay rises, job changes, performance reviews) and build models which identify which combinations are linked to changing individual-level engagement. We can look at which responses change and explore whether there were underlying reasons why this might have occurred.
- We can ask people if they feel more / less engaged (or even identify it from their history) and if they feel a change what events, experiences or factors have influenced their changes in feeling.
What we learn
When analysing data in this way we see significantly different issues that are cited as reasons for a change in engagement (or recommendation when using eNPS).
Our employee feedback tool, Workometry, uses open questions instead of traditional scale questions. It then uses highly sophisticated text analytics approaches to summarise the statement and group them into meaningful groups (with meaningful descriptions such as ‘More clarity on the future direction’). The summaries and groups the algorithms find will depend on the answers to the question asked (and often be organization-specific).
Many of the reasons cited for changes in engagement would not be part of a typical engagement survey, meaning that there is no way they’d be able to identify them, and by implication that you wouldn’t be able to act upon them.
Reasons may seem at first glance far more personal but there are a few key themes that we typically find:
- Environmental issues are often the largest group. These will include the team dynamics, toxic colleagues and ineffective management. I’d suggest that the single most effective way of improving a disengaged employee’s engagement level is changing their role
- Barriers to getting the job done is a significant issue, especially amongst high performers
- Measurement and incentive related issues are common. These could include targeting at the general level or where targets are badly aligned to things that are important to employees values, such as delivering great customer experiences.
- Various events that could all be categorised in a general ‘Employee Experience’ group can have significant changes. For individuals where we see sharp drops in engagement over two, close time periods there is usually an event which has caused the change.
- New senior leadership and refocussed strategies can increase engagement for certain groups.
Changes to action planning
Given the reasons we see most of the action that is likely to be effective would be bottom-up rather than the typical long, top-down approach. There will be some systematic issues that will need to be addressed but in general a principle of ‘decisions / action should be made at the lowest possible level’ is probably correct.
For organizations which really want to get this right will be building teams to address more structural issues. I’d imagine such teams would have experience in areas such as process and job design, technology enablement and incentives. Such changes would need to be formally costed / valued and prioritised.
As an extension of this organizations need to have a serious debate about how to ‘close the loop’ with employee issues in a timely manner. Whilst customer feedback can trigger this easily, in organizations employee feedback is usually provided with a promise of confidentially. Technology offers opportunities here.
‘Low hanging issues’ need to be fixed quickly. We always see issues such as staff facilities that are broken or policies, designed for one situation which are inappropriate in another area. Fixing these things quickly will likely not address fundamental reasons (they’re the toothpaste-lid sort of reasons mentioned above) but rapidly taking action sends a signal to employees that feedback is being listened to and providing it is worthwhile.
Overall we believe that much of the failure to improve engagement in organizations is due to using static analysis to understand dynamic systems issues. Businesses address the causes rather than the symptoms will realise significant benefits.
Andrew Marritt reflects on a People Analytics Podcast episode, a fascinating conversation between Laurie Bassi and Max Blumberg about the myths of Employee Enegagement, and looks into the real impact of Employee Engagement in business performance and value.
A large part of economics is concerned with understanding and modelling individuals’ behaviours when faced with various incentives. Whilst some believe that economists are only interested in money as an incentive, in truth incentives can take many forms.
George Akerlof co-wrote a fascinating book – with Rachel Kranton – on how identity is an incentive (Identity Economics: How our identities shape our work, wages and well-being). There is even a sub-discipline of economics called ‘Personnel Economics’ which is dedicated to understanding how organisations work.
Ten years ago, when I was part of the HR Analytics team at UBS, I was the only economist in a team of psychologists. Looking at People Analytics from the perspective of an economist forms a significant part of this podcast between Laurie Bassi and Max Blumberg.
One of the things economists spend a huge amount of time learning at university is calculus. Calculus is useful to tackle problems involving optimisation. Given we are always dealing with limited resources almost all problems in organisations are optimisation problems. I would argue that one of the key historical basis of modern people analytics in organizations comes from the discipline of operations research, like economics an applied mathematics discipline concerned with optimisation.
The reason calculus is useful is that rarely is there a straight-line relationship between a variable and its desirability. For many firms it would be beneficial to reduce employee turnover, however I’ve worked with a couple of organisations over the last 18 months where more attrition would be beneficial. There is an optimal level, past this the benefits decreases. You can have too much of a good thing.
Our organisations work as systems. There is rarely a clear, unique, optimal solution. Invest money in one area and – given limited resources – you can’t invest the same money elsewhere. Much of management is about prioritising resources to maximise returns.
This brings us onto a key point in Max & Laurie’s conversation – how much resource should one allocate to improving engagement. As the resources are limited what should we reduce spend or effort on? New production investments? Training?
In a system, many inputs depend on each other. In a dynamic system they often depend on the levels of the previous state. Rarely is it true that you provide the optimal solution for the system by optimising each individual input. Furthermore, optimisation doesn’t mean maximisation.
Unfortunately this is often the belief of many in the HR profession when considering engagement. Grand action plans are built without considering the alternative uses of the needed time or money. As Laurie so eloquently mentioned ‘Engagement is a necessary but not sufficient condition’.
Tom H. C. Anderson of OdinText has been writing about some research that he has run comparing Likert and Open Text survey responses.
Tom used Google Surveys to ask two samples of 1500 people a question about perceptions to Donald Trump’s controversial executive order banning people from certain muslim countries. To one group he asked the question as a typical 5 point Likert Question.
To the second group he asked the exact same question but instead let them respond in open text.
As you can see the answers are remarkably similar – within 2% of each other. According to his experiment an open question would be as suitable to gauge the level of agreement as a traditional quantitive scale question.
Tom’s findings are remarkably similar to what we see when using open questions as a replacement to traditional quantitive questions. We tend to use a combination of structured and unstructured questions in employee feedback requests but by far the most important are the open text questions. Open questions can achieve most of the aims of a scale question but provide some notable advantages.
In his post, Tom later highlights the difference between the Likert and the open text where the latter provided much richer data on the long-tail of responses (he describes them as low-incidence insights). As he notes:
“While there is nothing more to be done with the Likert scale data, the unstructured question data analysis has just begun…”
Recently a client asked us why we didn’t have a sorting question available in Workometry. Our answer was that for that type of question we’d prefer to use open text.
Their proposed sorting question had 8 different choices for the employee to sort. I could show an almost identical question asked as open text by another client where we had identified just under 30 different options. Whilst we hadn’t done a controlled test like Tom did given our experience we’d expect pretty much identical results. A sorting would be right only if you want to limit the potential options to a subsection of the person’s true opinions
In a recent FT Tech Tonic Podcast with Astro Teller, the head of Alphabet’s ‘X’ lab Astro notes: (at about 14:45)
“If you are serious about learning you set up your test in a way you don’t know the answer to the question you’re asking and you’re indifferent to the answer. You have to do that or you’re not really learning you’re biasing either the test or how you read the data.”
However good your survey design is, if you’re using a structured, choice-based question you’re asking a question where you’re by design limiting what the answer could be.
Open questions on the other hand give you the option of not knowing what you’re going to learn, before you ask. In our experience, if we’re doing a question like the ‘most important’ / sorting question above it would be common to find 2 or 3 of the top 10 answers that you wouldn’t have included in a similar structured question.
The other aspect that text provides is context. A structured question might identify the strength of feeling (though the example above shows that text can do this equally well) and who holds which feeling but it can’t show why they feel it. It’s why when companies do large engagement surveys often the immediate action is to do a series of focus groups on particular topics to understand the context.
When would we recommend structured question?
Even though we believe that in most instances the right option is to ask an open question there are a limited number of occasions when a structured question might be better. We find ourselves using them:
- when we want to report a simple metric over time, e.g. engagement or eNPS
- when our objective is to build a model and you need a score on a target variable for each individual
In both of these instances it’s because you want to have a score on something simple and purposely constrain the answer. We might be using such a score to determine what topics those who are engaged (or disengaged) are more likely to be discussing. It’s important to note, however, that for any feedback request we might ask 2 or 3 scale questions.
Why are scale questions so popular?
If open questions hold so many advantages why are most surveys mostly scale questions?
A very big factor is that it’s easy to report and analyse scale questions. Before tools like OdinText for customer feedback or Workometry for employee feedback analysing large volumes of unstructured data was hard.
Now, however that is not the case. Given the rapid progress of text analytics I suspect we’ll start to see the gradual death of the traditional survey. If you’re serious about learning it can’t come too soon.
A version of this post originally appeared in EFMD’s ‘Global Focus’ Magazine.
“A new market has emerged: Employee feedback apps for the corporate marketplace. These tools are powerful and disruptive, and they have the potential to redefine how we manage our organizations.” Josh Bersin, Forbes August 26 2015
The measurement of employee engagement is changing. Businesses have been measuring engagement for about 15 years, the market is currently worth USD1bn per annum yet most reports suggest engagement is trending flat if not actually decreasing. Something is obviously not working.
There are many reasons businesses are growing frustrated with current methods. Slow, expensive and resource-intensive are some of the more common ones we hear. In many businesses the only things that are now measured on an annual cycle are engagement & performance management – both run by HR. Business leaders are demanding more real-time insight.
During the same time HR has been emphasising understanding employee engagement the measurement of customer engagement and feedback has been changing remarkably. Today many firms are capturing an always-on stream of customer data from a wide variety of channels from short surveys to social media.
Often called Voice of the Customer, the emphasis has switched to continual listening, rapid resolution and bringing deep insights of customers needs into everything from product development to service provision.
Business leaders are asking why there is a disconnect. Why have customer teams adapted whilst HR has stood still? Many of the trends that we’re seeing in engagement measurement could be viewed as an application of a Voice of the Customer philosophy to employees.
At the same time that there is this shift in the demand side we’re also seeing a shift in the supply caused by technology.
The technology changes can be classified into four categories.
Technology-led automation is something that is happening across society and it should be no surprise that it’s surfacing in our area.
Firms, especially new entrants, are automating parts of the engagement measurement & analysis process that typically was done by analysts. Whilst it started with relatively simple report automation – the production of thousands of template-based pdf reports moments after a survey closed – we’re seeing the level of sophistication increasing.
Whilst this might, and should, have been utilised by the traditional firms for some time their incentive was to increase margin. Most of the new entrants are using it to radically reduce prices and complexity. The new business models are disruptive.
Real time reporting via dashboards is becoming the norm. Production of large numbers of PDFs is possible. We’re seeing a shift from the multi-page result presentation to one page infographic style reports. Ultimately there is a shift from seeing the provision of a large numbers of individual reports as complexity to seeing it as a commodity solution.
The ability to automate however can blind the user to really question whether they’re addressing the real issue, or merely creating a faster, cheaper broken process.
For Workometry we took the full end-to-end feedback process back to first principles. At the beginning and end of this process there are likely to be two time consuming and expensive periods of qualitative research – designing a great survey at the beginning and running workshops to understand context regarding the issues at the end. Only by addressing these long, expensive activities can you make feedback truly agile whilst preserving the richness.
Mobile & User Interface changes
The second technology-led shift is to do with the way that employees are able to take surveys.
For several of our clients mobile has become the dominant channel for employees to take surveys. We see respondents taking surveys just before the working day, during lunchtime and even in the evening. They’re interacting on the edges of their working days and grabbing a mobile device to do it.
Consumer web technologies have changed the way we expect to interact with our devices & engagement surveys can’t escape this trend. Many of the question types we used were the same as we used on paper but digitalised. We used these methods often because they were easy to score. Digital-only surveys aren’t bound by these constraints.
Research in user interfaces is reinforcing these methods. In a world where people expect to touch, slide and scroll through long-form sites surveys have needed to adapt.
Big data technology
The majority of the new entrants are focussing on the previous two technologies. Whilst this is right for medium sized businesses, enterprise organizations typically have a set of needs that extend these simple use cases.
One shift that has occured during the last few years in a number of firms has been the building of sophisticated People Analytics capability. Firms in this position are increasingly wanting to combine and analyse employees’ demographic, behaviour and perception data to answer key, strategic business questions.
Whereas employee survey data has historically been treated as an island – analysed with the context of the perception data or a predefined limited set of demographic information – survey data is now used to give critical insight into the reasons why.
To do this type of analysis requires that the survey data can be linked on an individual basis to both an extended set of demographic data, and to behaviour data, either from HR or business systems.
Furthermore it’s often useful not just to analyse the result of one survey with the extended data set, but to also include all other survey data belonging to an individual. Such requirements quickly dictate the sizes of data processing systems.
As well as the ability to handle large data sets increasingly analysts are using non-table data structures to better answer questions. One alternative that offers great potential are so called ‘graph databases’ where data is stored in a network. Such data structures allow us to ask very different questions.
With network data we can more easily answer questions not only about the individual employees but also the relationships between the employees. We see early promise in a network perspective at looking at contagion of engagement – ie how changes in employee engagement can spread across an organization.
Network survey technology such as that produced by Australian start-up Polinode allow businesses to capture not only traditional survey questions but also ask questions about an individual’s working relationships. Alternatively it is possible to understand communication patterns through the data trail left by emails, telephone calls or participation in internal social channels.
The final technology trend which is starting to disrupt the survey world is the application of machine learning – the use of algorithms to search for and learn patterns in large quantities of data. Machine learning is also the basis of much so called ‘predictive analytics’.
With employee survey data we’re seeing great success with three applications of machine learning: using text analytics to make sense of vast amounts of open text answers, using pattern-spotting techniques to make probabilistic assessments of which populations are most likely to raise certain topics and finally to use survey data to answer business questions.
Historically it’s often been acknowledged that open text is the most valuable part of a survey, however it’s been very difficult and resource-consuming to deal with it at scale. Text analytics can solve this problem and therefore provide new opportunities to capture this richer form of information.
Our experience is that with these techniques we’re able to analyse open text responses in almost any language, categorise a comment against a continually evolving set of categories & score against things such as sentiment and to do so in near real time. With this capability it’s possible to radically rethink how and what data is captured.
The second use of machine learning is to identify groups most likely to be discussing certain topics. Whereas traditional surveys might show differences between one function’s scores against their peers, with machine learning it’s possible to segment the population a much more granular manner. For example you might discover that those complaining about a shortage of career development opportunities are much more likely to be women, gen ’Y’ers who are in the upper performance grades.
Finally survey data is increasingly important to answer strategically important business questions that involve the workforce. For example you might link the survey data to sales data from a CRM system to try and optimise sales performance. In some cases it’s possible to use existing survey data. In others surveys need to be used to collect new data.
So with all these opportunities where to get started? We typically advise our clients to do three things:
1) Make sure that the legal and other agreements are in place to use data in this new manner. Be open with your employees about how their data is being used and how the new approaches don’t need to mean lower levels of confidentiality
2) Pilot some approaches with new use cases or in discrete populations.
3) Consider those pilots as supplementing existing work. From our experience you might replace old approaches but there is often significant political capital invested in the established approaches.
The biggest change over the last 5 years in the field of employee engagement measurement is the frequency that employers are looking to poll their employees. Whilst it used to be common to run a survey every 12 or even 24 months now firms are wanting to run a survey multiple times per year.
One question that we’re frequently asked is what is the ideal frequency to ask employees engagement-related questions. I don’t think there is a perfect and generally-applicable recommendation however the following describes our thinking about this.
Why measure frequently?
If we think about personal health, even if we have the ability to take measurements at any time it doesn’t necessarily make sense to do so. I may measure my weight on a regular basis but I can’t remember the last time I measured my height.
The reason that I don’t measure my height on a regular basis is that it doesn’t change (much). I’ve been about 187.5cm for the last 20 years. However during this time my weight has had a range of over 20kg. I’m certainly heavier now than I was one month BC, otherwise known as ‘Before Christmas’.
So, logically the only reason we’d want to capture engagement-type data often is if we believe that it changes frequently. Does it?
Thinking about organization-level engagement
All measurement has uncertainty. Depending on your variable of interest and your measurement approach you probably have different amounts of confidence in any measurement.
Most people report Employee Engagement at an organization level as the sum of the individual scores of employees in that organization. This makes some sense in that it’s easy to do but it adds extra uncertainty – we can’t really disentangle how much of the uncertainty comes from the measurement error at an individual level and how much depends on who has been included at any period.
Lots of uncertainty in how we measure engagement
The next thing to recognise is that in almost every instance we’re not measuring engagement (if we can actually define a commonly agreed definition, but that’s another issue). What we are actually recording are the answers to a number of questions.
Given a whole range of factors from culture, language, user interface, number and type of scale etc. we get different responses.
It’s worth considering that as individuals we have a feeling to any particular question on some continuum between two extremes. When presented with a scale question with categories – e.g. a Likert scale – they have a burden to try and convert where they see themselves on their continuum to what they think is the nearest value on the scale. Two people with identical feelings can interpret this in a different way. The same person facing the same challenge might change the interpretation. There is uncertainty in almost everything.
Measurement uncertainty and regular engagement measures
In the old world where people did engagement surveys infrequently – say once every year or two – there would often be endless conversations in organisations on the right timing. There was an implicit feeling that organization events, such as restructure announcements, would make a big difference and therefore the measurement team would try and pick a favourable time. It was the team feeling that there was uncertainty.
We probably can think of the measured value as an underlying engagement value + short-term noise. As managers we want to understand the underlying value. If the noise was totally independent of any short term issues then with a large enough population we could probably assume that our distribution of noise was approximately normally distributed and it would average itself out.
However, the concerns about picking the right time raises two issues:
- We probably can’t assume that the noise is independent
- Often we report down to small groups and therefore we won’t see avergeing out.
When infrequent surveys make sense
If we think about what this logically implies there are two conditions for proposing the historic, infrequent measurement cycle:
- That an employees’ perception doesn’t change frequently
- That the noise or measurement error is small and therefore one measure is good enough.
Do these apply?
We’ve got Workometry clients who are doing regular, even monthly, pulse surveys. If we look at individual-level longitudinal data we do see it mostly stable most of the time. However from time to time we see an individual who was stable suddenly starting to change – usually becoming less positive.
This stability also implies that measurement error is probably relatively small. It’s easier to assume that a static value probably shows infrequent change rather than real change that is ‘hidden’ behind an opposite measurement error.
The downside of frequent
There is another issue though facing us as we think of how often to ask employees for their feedback – being asked the same questions introduces fatigue. Fatigue means less people might respond therefore reducing data quality.
When we asked irregularly most companies had a top-down action planning process. This process took many months to complete a cycle. However given the infrequent nature of asking it was still possible to make the changes and let the organization adapt before the next cycle.
With frequent feedback, even with a more agile approach of closing the loop in terms of feedback, it might still be difficult for the organization to adapt if the cycle is too short. We’ve seen this in terms of feedback – employees asking why clients were asking the same questions when there were other topics that they could poll.
The ideal frequency
Our current view is that unless there is significant amounts of organization change occurring (eg during a merger) then quarterly is probably as often as you’d want to ask all the same questions to employees. Any closer that this and we start to see employees telling us it’s too frequent.
But does this mean you shouldn’t ask for feedback more often?
We believe, from looking at our data, that whilst most of the time an individual’s perception is stable, there are times – probably after key events – when an employee’s views can change. (See here and here for more information). Given there is likely to be a trigger that causes this we feel that it’s important to be as close to the event as possible. From this perspective monthly is probably the ideal time.
Meet the hybrid feedback approach
So how do we keep close to the events yet not be repetitive? By building a feedback calendar.
An engagment survey usually has 2 components: a question or questions to understand or infer engagement and a selection of questions where the objective is to find variables associated with higher or lower engagement.
Our view is that we need to add a third section – asking if their perception has changed since the last time period and if so what triggered the change.
With most clients we’re using a version of the eNPS as a proxy to measure engagement. We always ask the question about change so we can be close to the event.
However, we don’t have to ask the supplementary ‘factor’ questions every time. These tend to be relatively stable. Instead we take the opportunity to ask employees about other important questions
Building a feedback calendar
How do we work with clients who want to take this approach?
- To start we set the months where we’re going to ask for engagement-focused feedback. This will proberly be quarterly so we’ve filled 4 months
- We then identify other events where it might be useful to get employee feedback. Examples of this could be around performance reviews, company results, promotions, business cycles etc. These are events that can be identified a long time in advance. Doing so might fill 3 or 4 more months
- Finally we work with executives to identify key business challenges that they might want to get the views of the employees. Sometimes this can be planned ahead but otherwise we might want to instigate a process to identify this topic closer to the time but with several dates in the diary.
The final thing to note is that we might identify topics based on earlier results. If you need to go back to the full employee base you might do a follow-up month. Often however you can add a question or two to the monthly questions of a defined segment.
Asking open questions
The key to all of this is that if you ask for feedback it has to be quick to complete and seen as relevant. Our view is that we get the best quality data by asking open questions and scaling this qualitative approach.
Doing so means that typically it takes less than 5 minutes to complete a feedback request. As we can reveal the company-wide feedback within a very short time of a feedback period closing we can work with our clients’ teams to communicate back to employees and to identify which groups and individuals need to act. The context provided by open questions means that managers can easily identify what needs to change.
Employee experience is rapidly becoming one of the key topics on the CHRO agenda. Yet many of the conversations that I hear miss a critical factor: that creating valuable employee experiences is a systematic and data-driven process.
When I left a senior HR role in 2009 to build a business ‘to help make HR an empirically-driven function’ one of the key areas of information that we started with was experience data. In the diagram above, which I’ve taken from one of our earliest presentation decks, the components at the bottom right are all ways of measuring experience.
Our earliest proposition said OrganizationView focused on 3 things:
- measurement & meaning – collecting data and making sense of it through analytics
- employee-centric design – as we said ‘use a scientific approach to ensure technologies and services are closely aligned to users’ needs and behaviours
- develop and deliver – moving analysis into production
Why such a focus on experience in 2009? Well, my background in the early noughties was centred around understanding in a deep way how to systematically understand user experience. Lots of this was in the area of candidate experience. You can see some of this in a 2004 article by David Bowen in the Financial Times – subscription needed – that came after a long conversation we had about candidate needs from my research at the time. It’s about building career sites and recruitment systems that are based around optimising the candidate experience.
As an aside when I joined UBS in 2005 to launch their first global careers site on the first meeting of the project team, when we were discussing governance I added one rule: “if we can’t decide what to do we’ll test it with users in an experience lab.” We tested lots (UBS had two user-research labs and we also ran tests in London) and the bank came (joint) top in the FT web ratings in the career section that year. We cut our marketing budget that year by over-investing in research.
Some of this philosophy came from working in a couple of firms where my close peers were working on projects with IDEO. We took this view and many of the techniques into recruitment, making it candidate centric and based on experience and relationships. The key though was that the process was heavily research-centric. Experience design is highly aligned with empirical decision making. It is systematic and based on data. A central theme is to actively and constantly listen and understand the experiences your stakeholders.
IDEO, in their 51 method cards, separate their ‘measurement’ approaches to 4 categories – Learn, Look, Ask and Try. What they all are is ways of understanding how the user experiences a product or service or the part of their life where the offering will fit. Some are very qualitative, some more quantitive. I believe all qualitative data can be quantitive if you capture enough examples. Also, the first thing you do with qualitative data is to add meta-data which makes it quantitive. In the end data is just information.
From Candidate Experience to Employee Experience
The roots of Employee Experience came from Candidate Experience. From 2002 I smashed my head against the proverbial wall for a long time trying to evangelising why it was critical. The Talent Board folks did a much more effective job.
One of the slides we used to show in the early days was the following graphic. In it we compared the importance of experience as a driver of satisfaction in banking and in work. We used internal bank research (not UBS) with some re-cut data from the CEB. It turns out that in each case components of the offer which could be classified as ‘experience’ account for about 70% of what drives satisfaction, and therefore engagement.
The way an employee thinks about their organization is the sum of their experiences. At different stages in their journey from consideration, through selection to employee and alumni their perception will change. How that perception develops is the sum of their experiences. I discussed how this is linked with the EVP in early 2011.
Employee Experience and People Analytics
What we can establish is that experience design is both systematic and data-driven. Yes, it incorporates systems and user experience but critically it includes experiences that have nothing to do with systems. Even with systems you need to understand what people were doing before they go to the system and what they do after using it.
Our vision of People Analytics is that it should drive evidence-based decision making about the workforce in organisations. We have always felt that that evidence is a mixture of quantitive and qualitative data. We believe that experience measurement is a core element of the role of a People Analytics team.
In the graph above we show that 70% of the drivers of satisfaction is experience based. If we think of the current state of People Analytics too many firms only use existing data from their HR systems to develop their models. None of this data is likely to be describing experiences. They’re building models trying to squeeze meaning without signal about the important part.
The analysts’ job is not to build accurate models, it’s to answer critical questions with data. Given how important a driver experience is it needs to include experience and therefore many analyses need to include experience data these models. The analyst needs a robust and automated way of capturing this data.
At the heart, this was the basis from which we decided to build Workometry. Capturing open, reliable experience data at critical touchpoints – what some call ‘moments that matter’ – and doing so in a way so that it can be integrated into sophisticated models is critical to understanding and managing the employee experience.
In the last post I gave some background on how we got to the current state on measuring engagement and employee perceptions. In this one I’m going to give an overview of where we see the market going. It’s a personal view, but one from someone at the coal-face.
There’s an expression in english that ‘if all you have is a hammer, everything looks like a nail’. I think a large part of what has driven us to the current position is that businesses felt that all they had was the long, annual engagement survey ‘hammer’.
In the last article I cited Andrew Graham’s 9 issues with traditional employee research:
- Not frequent enough
- Single scoring leads to issue distortion
- Aggregation reduces meaning
- Does not capture the specifics (context seldom captured)
- Lengthy or poor response planning
- Managers are busy & have no incentive to implement any actions
- Lot of resources & monitoring
- Surveys get old
- Causality not clear
There are some other drivers which seem to be contributing to change, or at least a desire to change:
- Most other business metrics are produced more frequently than they were previously, and business executives question why engagement should be different
- Businesses have become more customer-centric and as such have embarked on customer listening programmes, many of which are tightly integrated across the business (always-on listening rather than annual surveys). Their experience of capturing customer perception data has changed
- The rise in sustainable investment and accounting is encouraging firms to take a multi-stakeholder approach rather than just focussing on one group (investors?). Employees are seen as key stakeholders.
- Technology is changing the economics of employee measurement. I believe that it is possible to be better, quicker and cheaper now. This is not a move along a frontier curve, it’s shifting the curve and shifting it dramatically
- Enterprises are rapidly building People Analytics functions. These functions always want better data which can be integrated with other enterprise data
- Digitalisation is unbundling the consulting offering which the major providers have used. New start ups are using technology to capture value in a much more scalable manner and cherry-picking the profitable parts of the offer.
The first response: Pulse surveys.
Let me start by saying that I think the gold-standard is doing both an annual survey with other, more immediate ways of capturing feedback either on a schedule or/and aligned to key employee events. However, in the real world businesses need to allocate short resources to where they believe they get the most impact. I think the annual survey will become rare over time.
The biggest trend which we’ve seen over the last 5 years is the emergence of the pulse survey. These are regular, shorter surveys which typically use technology and automation to address many of the issues above.
All pulse surveys try to address several of these issues, most notably the issue of frequency (and the linked issue that survey data gets old). Reporting is usually automated and several tools automate more advanced analysis showing linkages between variables and engagement.
However there are tradeoffs. There is a burden on the survey taker each time they take the survey and the only way of achieving decent response rates whilst asking more frequently is to shorten the survey, potentially reducing richness. As I noted in the last article, this might not be such an issue if you randomise some questions and use missing-data inference techniques but few providers are doing this (and many HR departments remain unconvinced).
There is also an issue in terms of how frequently you can ask employees to take a pulse survey. There is no right and wrong answer but there are some things that are worth considering.
We’re seeing that engagement at the individual level is remarkably stable. Most people answer the same way month in month out, potentially only changing by by a small amount if they do change (and we have to assume this could be a measurement error). You only need to measure a stable thing frequently if you’ve got a large measurement error. In these instances data should be smoothed before reporting as your issue is how to deal with noise.
Furthermore employees get frustrated if they’re asked the same question on a regular basis & they don’t see action, which is hard to demonstrate if the survey frequency is too short. I think you can get away with this for one or two ‘trend’ questions but there needs to be some change to make it feel worthwhile.
In our experience, a happy compromise is gathering engagement-type feedback on a quarterly basis. The best firms are using the non-engagement months to plan a series of topics where employee feedback is valuable and by capturing experience-type feedback around key events
Better UI isn’t a sustainable competitive advantage
Al pulse survey providers have embraced modern web technology to provide a more consumer-like experience. The is seen in both the survey-taking interface, where a good, mobile-friendly interface is probably now a hygiene-factor and to a lesser degree the reporting side.
One of the early incentives that encouraged us to build Workometry was the experience of how much data cleaning & preparation was needed with the existing solutions. Capturing the data in a way that you need it for analysis and automating the boring data preparation process brings speed, quality and cost improvements.
From a reporting side it’s now relatively simple to build a manager- or even employee-level interface that reports the data in a clean and understandable manner.
Many of the current entrants into the market aim to do just this. They bring a reduced overall cost to what has been done manually. However my view is that the way to compete in this part of the market will end up being on price. This is great if you’re a small to mid sized client as historically the consultancy-led approach was too expensive without scale. It probably means that we’ll see consolidation in this part of the market.
Thinking about the data
Here is where I think it gets more interesting. As I noted before the advent of more sophisticated analysis activities with employee data is creating a demand for better, more integrated data. At the same time there continues to be a need to ensure confidentiality, especially preventing reporting down to small groups.
This is a challenge which can be addressed by technology. APIs can provide data consumption whilst maintaining confidentiality. Newer forms of data stores, like the schema-less one that underpins our Workometry product can provide great flexibility. We’ve got to a stage where realistically we don’t have a constraint from the size of the data. If we want to compare people who answered a particular question this month with how they answered another question last quarter that’s possible. If we just want to be in a position where we can capture and analyse many millions of data points a year technology or cost of technology isn’t a constraint.
The second factor about thinking about data is always to question where the best data is to be found. We shouldn’t be asking employees about things in surveys which we could understand from another source. In particular I think of the questions about process compliance. Surely we should be able to gather this data from a system? Only ask what you need to ask.
A key factor with employee listening is that it’s iterative and conversational. That means you can’t know with certainty what you’ll need to ask in the future, or what data you’ll need to integrate. Enterprises need to select these new technologies with this in mind. How can the data be integrated into models? How can it be queried in real-time for models in production?
Think about the analysis, but start at the action.
It’s easy to report data but this typically doesn’t bring much insight. We believe that with all analysis we need to work back from the outcome – what you’ll do when you have the insight. Understanding what you can and want to possibly change should be an input into what data you capture.
Dealing with survey data, and especially mixed survey-behaviour or survey-demographic data is difficult. Even if we understand that in many questions we’re dealing with ordinal data that guide us to a certain subset of possible analyses (and not the typical ones which are easy in Excel). As I’ve written in the past, we’re big fans of using graphical models and much of our survey work relies on relationships – relationships between variables and relationships between variables.
As the data size increases along with the complexity Machine Learning is increasingly being used to identify insight from the data. Where we used to cut the data by a demographic function we’re increasingly looking for non-linear relationships, identifying small and micro segments and attempting to link demographics, behaviours and perceptions.
I have written about this before. In our view capturing and understanding employees own words is the key breakthrough that will happen in employee feedback in the next few years. This mirrors what we’re seeing with customer research.
Traditionally there was almost a binary split between quantitive and qualitative research work. In reality there was always a continuum. It was always technically possible to do qual work at scale, however it was prohibitively expensive and time consuming.
What we’re seeing with modern text analytics is that that supply curve has shifted. It’s now possible to ask a survey with mostly open text questions and to ask it to hundreds of thousands of individuals. This opens up great opportunities in terms of increasing the richness of data, the attractiveness of providing data from the user’s perspective and a flexibility and agility in design.
From our experience (and we think we’re probably in a leading position in this space) text analytics is highly domain-specific. It is unwise to think you can parse text through a general tool, or one designed for another domain.
I mentioned this in the last article and received some questions. I think we’re heading for a position where technology-enabled employee research will be conversational. The employee will be asked an open question about a particular topic and depending on the response they give a relevant follow-on open question will be asked.
In the short term these technologies will be text-based but it’s likely it will move to voice. I see a future where a system identifies who it needs to speak to, telephones them and asks a series of questions to try and drill-down to the key drivers. It will be able to do this at scale.
The issue to some degree isn’t the analysis, it’s the ability to do this in a way which isn’t intrusive for the user and where there are strong incentives for them to participate. This is what is exciting about research. Analysis is closely related and enabled by design-driven changes. Better interfaces = better data = better analysis = better insight.
Where is all of this taking us?
I see the engagement-app market diverging. On one side are employers who want to understand engagement but do so in a cost-effective manner. These firms will gravitate to the mass of engagement apps offering real-time capture. Buyers should be aware of the value these tools are providing – it’s mainly focussed on delivering metrics in a speedy and cost effective manner. These providers will make most headway in the part of the market where traditional providers couldn’t be cost-effective, ie SME businesses. Those who can add sophistication and scale up to enterprise clients will be disruptive.
The other side are firms and products like our own which see machine learning technology as a way of automating the higher-skilled parts of the historic research bundle. In the first instance firms like ours are using ML to automate much of the skilled work of the experienced engagement consultants.
I think the ultimate role will be disrupting traditional management consultancy. Tools which can ask and understand an almost open set of questions can disrupt much of the information-gathering parts of the traditional consulting offering.
None of this completely removes the need for great consultants, but it means that deep expertise, or the creative parts of the process are where the opportunity arises. Consultants will increasingly focus on developing strategic plans and advising on change management. There might be fewer employed by employee research firms, but I think this shift will increase the demand for those at the top of their games.
In these articles I use the term ‘survey’ to mean both a survey, with sampling, and a census where everyone is asked.
There is a shift at the moment from long infrequent engagement surveys to shorter, ‘pulse’ surveys which are being used either as a replacement or supplement to the longer survey. With this and the following post I wanted to discuss some of what we see as reasons and advantages and in doing this. As always I hope to give a data-led perspective.
Organizations, recently through their HR departments but before that via operation research groups, have been conducting employee surveys for around 100 years. In the 1970s the focus was on organization commitment and job satisfaction and the focus went from OR to HR. There had been some earlier work by Katz (1964) on Organizational Citizenship Behaviour which talks about Organization Commitment.
Engagement was first described by William Kahn in 1990 but was made popular by Gallup’s ‘First break all the rules’ book of 1999. Since then most organizations have been conducting some form of engagment survey.
During the same sort of timeframe the technology for completing surveys has changed. In the 1990s and before most surveys were still done on paper. When web technology started to enter the employee survey space we saw surveys which were fundamentally a replication of a paper survey done electronically. This made sense at first as many organizations spent some time doing both in parallel. We still see paper in some environments such as delivery drivers.
About the surveys
Engagement surveys tend to follow a common design – they ask a set of questions to create an engagement index. Mostly this is in the region of 5 questions. They then ask a large number of questions to identify items which are linked to that engagement. Most annual surveys are in the range of 60 – 150 questions long. I would estimate it takes about 20 – 30 seconds for an employee to answer each question.
Data is also used to consider the demographics of each participant. We see both self-reported demographics and demographics that are imported from other HR systems. The latter is a more effective way of getting good data but in some firms there are concerns about privacy.
There is a potentially enormous number of factors that could be associated with engagement. As Kieron Shaw noted in “Employee engagement, how to build a high-performance workforce.”:
“It’s arguably unfeasible to directly measure in the survey all the actions behind engagement,” due to the fact that, “there are potentially thousands of different individual actions, attitudes, and processes that affect engagement.”
Hence, however long the survey is the designer has to have selected a subset of potential factors.
Criticisms of traditional surveys.
In a fascinating paper “Measuring Employee Engagement: Off the Pedestal and into the Toolbox” Andrew Graham of Queen’s University notes 9 issues with the traditional survey:
- Not frequent enough
- Single scoring leads to issue distortion
- Aggregation reduces meaning
- Does not capture the specifics (context seldom captured)
- Lengthy or poor response planning
- Managers are busy & have no incentive to implement any actions
- Lot of resources & monitoring
- Surveys get old
- Causality not clear
A tenth issue that we find as analysts is that there is typically an illusion of richness. Many firms think that by asking 80 questions they are capturing 80 independent data points. This is clearly not the case.
Issues with survey data
One of the analyses that we like to do with survey data is to build a correlation graph. This uses each question as a node and the correlation between each question as an edge. When you visualise survey data in this manner you typically get something like the following:
What we see is a hairball. Each question tends to be highly correlated with another. (In the graph above Questions 31 – 33 are questions that the HR team wanted to add relating to a process which obviously has little link to engagement).
We’ve done experiments with survey data where we ‘destroyed’ 80% of all answers randomly and then used recommendation algorithms to back-fill what had been removed. In most instances we’re able to accurately replace what has been removed. People answer in patterns (hence that hairball), and if you know some answers you can pretty accurately infer what all the others will be (this means that you could probably randomly ask each employee different questions and dramatically shorten the survey without much loss in accuracy).
Issues with User Interfaces
This is a bit more contentious. It relates to how questions are asked.
Most employee surveys use Likert-scale questions, mostly 5 points between strongly agree and strongly disagree. One of the reasons for doing this has been that on a paper survey it’s easy to get someone to code the data into a reporting system (it’s easy to see a check in a box). What has been done is to take this process that was designed for paper and put it onto the web with little thought in terms of adapting the question to take advantage of the opportunities presented by the new medium.
Employees actually have a true feeling on a continuum between the two end points. When you ask them to answer on a 5 or 7 point scale what you’re actually doing is asking them to ‘bin’ their true feeling to the nearest compromise point. Doing so is adding burden on the survey taker and potentially adding inaccuracy in the data. The data can’t be seen as linear, instead one should use statistical methods appropriate for ordinal data.
In a 2012 paper in the journal Field Methods “Why Semantic Differentials in Web-Based Research Should be Made From Visual Analogue Scales and Not From 5-Point Scales”, Funke & Reips show experimental evidence that show that marking a line between two points – a visual analogue scale – has numerous advantages over traditional 5 point scales. Two of these are better (more accurate) data and less burden on the survey taker.
Whether the answer is a visual analogue scale or something with a large but distinct number of points (the 0-10 scale used by NPS practitioners?) is harder to determine. However I see little evidence that 5 points is the right approach.
Should we even be asking scale-based questions?
Finally, too often what drives executive action from survey data is the responses to a few open text questions. As Graham notes on his fourth issue survey data rarely provides context. The qualitative nature of open text does provide this opportunity.
Often the initial action from a survey is to do more qualitative research focussing on several key topics. Such research is both time consuming and expensive. (Arguably acting without understanding the context can be more expensive).
There are instances where asking a scale question makes sense, most notably if you’re wanting to report a trend. However asking sufficiently broad, open questions will likely capture richer data. The challenge for many firms is how to do this at scale.
If we think about how we’d try to understand an issue in a conversation we’d ask open questions and then follow up with relevant and probing follow-up questions. I firmly believe this will be the future of employee feedback, though it will be a bot-based conversational approach which can be done in multiple languages and at scale.
As an industry we’re currently not there but the future is coming quickly. In the next article I’ll describe the current state, our findings of working with clients at the cutting-edge and highlight some approaches taken by other innovators in this market.
According to Josh Bersin, businesses spend USD 1 bn on employee engagement. We’ve been measuring engagement for the last 15 years. During this time engagement figures have been flat or declining. Why so little return on our investment?
I’m not going to question the concept of employee engagement. Yes, there are issues including a lack of commonly agreed definition but I don’t believe that this is the issue. Regardless of how you define engagement it doesn’t seem to have improved.
My view on why engagement has been so flat is because of the way the research-to-action cycle has been conducted. Basically, because the measurement & analysis approach has been flawed the actions have been misguided.
Systems dynamics to the rescue.
There may be some readers aware of systems thinking. Peter Senge who wrote about the ‘learning organization’ is part of the systems group at MIT. He’s also an aerospace engineer. This is important.
When I teach People Analytics the section about applying Systems Dynamics to workforce issues is always the most popular part. People are amazed at how the complex patterns we see in organizations can often be predicted using quite simple rules and models.
Engineering, and especially aerospace engineering is heavily systems based. It gives engineers a different way of viewing the world.
The two basic components of a dynamic system are stocks and flows. A simple explanation is a bathtub. If water is coming into the bath quicker than it is leaving the bath continues to rise, if the rate of outflow is greater than the rate of filling it empties. We can have stocks and flows of pretty much anything.
So what has Systems dynamics got to do with engagement?
With Workometry we’re in the fortunate position of capturing a very large amount of perception data. We have large, enterprise clients running monthly pulse surveys. We store the data in a way that enables us to study at the individual level up how perceptions change over time.
As analysts we built a systems dynamics model for engagement and plugged our data in. It worked, our data pretty much validated the model.
The great thing about systems dynamics models is that you can use them as simulators: make a change to the assumptions and the model will create predictions. We can therefore use it as a management simulator to understand what we need to change to improve what we care about – engagement.
How have we measured engagement?
Analysts tend to use a very simple approach to measuring engagement. Through a set of questions – an engagement index – we measure the number of engaged employees and divide this by the total employees and we get a rate.
If we step back and think about this model there are only two states – we treat people as either engaged or disengaged. With a systems dynamics model we can think of this as two stocks – engaged and disengaged employees, with flows defining people being recruited, leaving and moving from the engaged to the disengaged stocks and vice versa.
What does our data show?
- Almost everyone joins the organization engaged
- More people leave who are disengaged than engaged. Not only is this what the engagement literature predicts but it turns out to be necessary for the system to be stable
- There is a rate by which engaged people become disengaged
- There is a rate by which disengaged people become engaged. (it turns out that this must be lower than the disengagement rate for the model to be stable and the data certainly shows this)
The model also predicts that in periods of recession, when fewer people voluntary quit an organization, we see engagement levels fall. This is exactly what we saw in the last recession.
How have we been doing analysis?
At a high level we can think of an engagement survey as having two parts. First we ask a series of questions to create an engagement index. We then ask a large number of questions about various factors we think will be linked to engagment.
Traditional analysis uses a bunch of statistical techniques to work out what factors seem to drive engagement. What we’re effectively doing is seeing what factors most closely link with engaged / disengaged employees. We then build action plans based on this analysis
The problem with this analysis, however good your statistician, is that it is static. We take a cross-section of the data at one period. We’re looking at the differences between people who are engaged and those who are disengaged. We haven’t looked at what has changed an engaged employee to be disengaged. Basically we’re looking at the wrong thing and our action planning is therefore misguided.
What should we be doing?
It turns out the model, validated by our data, makes a very simple recommendation – the most important aspect to focus on is changing the rate by which you’re disengaging previously engaged employees. The only other reliable way of changing engagement is to encourage more disengaged people to leave.
Our model shows a very interesting and non-intuitive finding. If you do something to change the numbers of engaged and disengaged employees (our current post survey actions) the results will inevitably be short-term. You always return to the same engagement rate after a few periods!
The only reliable way of shifting the engagement rate is by changing the rate by which you disengage employees. Our traditional, cross sectional analysis is useless at determining this. We spend vast amounts of resources making big changes which are doomed to failure.
How people change their levels of engagement
What’s fascinating is seeing how people change from engaged to disengaged. We have found two generic patterns
- The Faders: These employees move from engagement to disengagement gradually over a period of around 3 or 4 months.
- The Fallers These employees move from a state of engagement to disengagement in one month. Often they’ll move straight from a high engagement score to the lowest in one month and the next month ‘bounce’ to a low, but a bit higher, state.
As predicted by the model we see few people who were disengaged becoming engaged.
Recommendations for fixing engagement
Fortunately the changes we need to make in engagement research are reasonably small:
- Conduct frequent pulse surveys to be close to the change
- Do longitudinal (sometimes called panel) analysis and look for switching patterns
- Ask people about their experiences / why they changed their view
- Identify factors that are related to these switching patterns
- Act on these factors to change the switching rate.
There have been a number of opinions that suggest that survey data isn’t useful to the People Analyst, especially when developing predictive models. These opinions go against our experience.
In a recent attrition project with a major client we were able to increase the accuracy of our models by around 10% at a relatively early stage by including survey data. Anyone who is doing Machine Learning knows that a 10% improvement isn’t easy to achieve.
A NY Times article written by two experienced data scientists from Google & Facebook, shows how these firms combine survey and system data to reveal what is really going on. As the authors note…
“Big data in the form of behaviors and small data in the form of surveys complement each other and produce insights rather than simple metrics. ”
This is certainly our experience, however how you use it can be non-obvious at first.
Most modelling in HR is Exploratory, at least initially
I would argue that the majority of modelling in HR has exploratory data analysis as at least one of its objectives. Most analysts default to tools such as tree models for understanding rules or something like naive Bayes for understanding variable importance.
HR models will almost certainly be used to inform, rather than automate a decision. Our loss or utility functions tend to be complex enough that for many scenarios data becomes another voice at the table.
For individuals to make decisions based on data they need to have some form of understanding how the recommendation was made. Actions tend to be aimed at altering one or more of the important attributes that is associated with the model and therefore interpretation is important. If we’re maintaining the confidentiality promises made when employees took a survey, for example by setting the minimum size of any leaf in a tree, do we need to use the survey data for ongoing forecasting and / or prediction?
Survey data is brilliant at informing feature selection
The longest part of any machine-learning centric project is feature selection or building. We take a few fields of data and create the variables which we expect to influence the model.
Using survey data in early stages of a machine-learning project can significantly contribute to your choices on which variables to create. For example in one recent attrition project we found a strong relationship between employees who self-reported that they didn’t have the necessary time to do their jobs to the best of their abilities & attrition. We then took this ‘lesson’ and created a set of variables that could represent this in a more accurate manner than relying on infrequent, self-reported data. We use the survey data to guide our feature engineering.
As the analysis continues what we find is that the survey’s influence on the predictive accuracy decreases as we replace survey findings with features that can certainly be used on an ongoing basis for forecasting or prediction without any confidentiality concerns regardless of the population size.
How to use survey data in model building
Survey data can be used a multiple levels within a modelling project, and the level to some degree depends on the question that is being asked. The fundamental need is to have a employee-level unique identifier with each row of data. Obviously if all you have is team data you can answer some questions, but the potential value is lower.
If we think of the organization as a tree where individual a reports into individual b who reports into individual c then via graph traversal it’s simple to aggregate responses to various levels – i.e. what was this individual’s response, what was the response of their team (all at a’s level who report to b), what was the response of the broader team (everyone who reports to c or someone who reports to a manager reporting the c). If you have a project or matrix organization it’s possible this way to have some more complex rules.
Within the feature engineering part of the project we can thus combine survey answers at any level to the data from your HR system. Some answers will add to the prediction at an individual level, others, for example about manager effectiveness might be more sensible to use as an aggregate for all people who report into that manager.
As noted, the models you create might inform what other data you add or collect. This is similar in many ways to including the knowledge of domain experts in your modelling team who can guide the analysts to where to look from their knowledge of the earlier research. The difference with survey data is that it is company-specific.
In conclusion, we believe survey data to be vitally important to most people analytics projects. As analysis is an iterative process we find it a rare project where during later stages, where we try and capture new data to improve an existing model, we don’t include some form of survey or questionnaire.
Insight depends not only on what has happened by illuminating why it is happening as well. Surveys can help provide the second part.