People Analytics

Employee feedback & engagement measurement – Part 2, the future

Employee feedback & engagement measurement – Part 2, the future V03-01.png

In the last post I gave some background on how we got to the current state on measuring engagement and employee perceptions. In this one I’m going to give an overview of where we see the market going. It’s a personal view, but one from someone at the coal-face.

There’s an expression in english that ‘if all you have is a hammer, everything looks like a nail’. I think a large part of what has driven us to the current position is that businesses felt that all they had was the long, annual engagement survey ‘hammer’.

In the last article I cited Andrew Graham’s 9 issues with traditional employee research:

  • Not frequent enough
  • Single scoring leads to issue distortion
  • Aggregation reduces meaning
  • Does not capture the specifics (context seldom captured)
  • Lengthy or poor response planning
  • Managers are busy & have no incentive to implement any actions
  • Lot of resources & monitoring
  • Surveys get old
  • Causality not clear

There are some other drivers which seem to be contributing to change, or at least a desire to change:

  • Most other business metrics are produced more frequently than they were previously, and business executives question why engagement should be different
  • Businesses have become more customer-centric and as such have embarked on customer listening programmes, many of which are tightly integrated across the business (always-on listening rather than annual surveys). Their experience of capturing customer perception data has changed
  • The rise in sustainable investment and accounting is encouraging firms to take a multi-stakeholder approach rather than just focussing on one group (investors?). Employees are seen as key stakeholders.
  • Technology is changing the economics of employee measurement. I believe that it is possible to be better, quicker and cheaper now. This is not a move along a frontier curve, it’s shifting the curve and shifting it dramatically
  • Enterprises are rapidly building People Analytics functions. These functions always want better data which can be integrated with other enterprise data
  • Digitalisation is unbundling the consulting offering which the major providers have used. New start ups are using technology to capture value in a much more scalable manner and cherry-picking the profitable parts of the offer.

The first response: Pulse surveys.

Let me start by saying that I think the gold-standard is doing both an annual survey with other, more immediate ways of capturing feedback either on a schedule or/and aligned to key employee events. However, in the real world businesses need to allocate short resources to where they believe they get the most impact. I think the annual survey will become rare over time.

The biggest trend which we’ve seen over the last 5 years is the emergence of the pulse survey. These are regular, shorter surveys which typically use technology and automation to address many of the issues above.

All pulse surveys try to address several of these issues, most notably the issue of frequency (and the linked issue that survey data gets old). Reporting is usually automated and several tools automate more advanced analysis showing linkages between variables and engagement.

However there are tradeoffs. There is a burden on the survey taker each time they take the survey and the only way of achieving decent response rates whilst asking more frequently is to shorten the survey, potentially reducing richness. As I noted in the last article, this might not be such an issue if you randomise some questions and use missing-data inference techniques but few providers are doing this (and many HR departments remain unconvinced).

There is also an issue in terms of how frequently you can ask employees to take a pulse survey. There is no right and wrong answer but there are some things that are worth considering.

We’re seeing that engagement at the individual level is remarkably stable. Most people answer the same way month in month out, potentially only changing by by a small amount if they do change (and we have to assume this could be a measurement error). You only need to measure a stable thing frequently if you’ve got a large measurement error. In these instances data should be smoothed before reporting as your issue is how to deal with noise.

Furthermore employees get frustrated if they’re asked the same question on a regular basis & they don’t see action, which is hard to demonstrate if the survey frequency is too short. I think you can get away with this for one or two ‘trend’ questions but there needs to be some change to make it feel worthwhile.

In our experience, a happy compromise is gathering engagement-type feedback on a quarterly basis. The best firms are using the non-engagement months to plan a series of topics where employee feedback is valuable and by capturing experience-type feedback around key events

Better UI isn’t a sustainable competitive advantage

Al pulse survey providers have embraced modern web technology to provide a more consumer-like experience. The is seen in both the survey-taking interface, where a good, mobile-friendly interface is probably now a hygiene-factor and to a lesser degree the reporting side.

One of the early incentives that encouraged us to build Workometry was the experience of how much data cleaning & preparation was needed with the existing solutions. Capturing the data in a way that you need it for analysis and automating the boring data preparation process brings speed, quality and cost improvements.

From a reporting side it’s now relatively simple to build a manager- or even employee-level interface that reports the data in a clean and understandable manner.

Many of the current entrants into the market aim to do just this. They bring a reduced overall cost to what has been done manually. However my view is that the way to compete in this part of the market will end up being on price. This is great if you’re a small to mid sized client as historically the consultancy-led approach was too expensive without scale. It probably means that we’ll see consolidation in this part of the market.

Thinking about the data

Here is where I think it gets more interesting. As I noted before the advent of more sophisticated analysis activities with employee data is creating a demand for better, more integrated data. At the same time there continues to be a need to ensure confidentiality, especially preventing reporting down to small groups.

This is a challenge which can be addressed by technology. APIs can provide data consumption whilst maintaining confidentiality. Newer forms of data stores, like the schema-less one that underpins our Workometry product can provide great flexibility. We’ve got to a stage where realistically we don’t have a constraint from the size of the data. If we want to compare people who answered a particular question this month with how they answered another question last quarter that’s possible. If we just want to be in a position where we can capture and analyse many millions of data points a year technology or cost of technology isn’t a constraint.

The second factor about thinking about data is always to question where the best data is to be found. We shouldn’t be asking employees about things in surveys which we could understand from another source. In particular I think of the questions about process compliance. Surely we should be able to gather this data from a system? Only ask what you need to ask.

A key factor with employee listening is that it’s iterative and conversational. That means you can’t know with certainty what you’ll need to ask in the future, or what data you’ll need to integrate. Enterprises need to select these new technologies with this in mind. How can the data be integrated into models? How can it be queried in real-time for models in production?

Think about the analysis, but start at the action.

It’s easy to report data but this typically doesn’t bring much insight. We believe that with all analysis we need to work back from the outcome – what you’ll do when you have the insight. Understanding what you can and want to possibly change should be an input into what data you capture.

Dealing with survey data, and especially mixed survey-behaviour or survey-demographic data is difficult. Even if we understand that in many questions we’re dealing with ordinal data that guide us to a certain subset of possible analyses (and not the typical ones which are easy in Excel). As I’ve written in the past, we’re big fans of using graphical models and much of our survey work relies on relationships – relationships between variables and relationships between variables.

As the data size increases along with the complexity Machine Learning is increasingly being used to identify insight from the data. Where we used to cut the data by a demographic function we’re increasingly looking for non-linear relationships, identifying small and micro segments and attempting to link demographics, behaviours and perceptions.


I have written about this before. In our view capturing and understanding employees own words is the key breakthrough that will happen in employee feedback in the next few years. This mirrors what we’re seeing with customer research.

Traditionally there was almost a binary split between quantitive and qualitative research work. In reality there was always a continuum. It was always technically possible to do qual work at scale, however it was prohibitively expensive and time consuming.

What we’re seeing with modern text analytics is that that supply curve has shifted. It’s now possible to ask a survey with mostly open text questions and to ask it to hundreds of thousands of individuals. This opens up great opportunities in terms of increasing the richness of data, the attractiveness of providing data from the user’s perspective and a flexibility and agility in design.

From our experience (and we think we’re probably in a leading position in this space) text analytics is highly domain-specific. It is unwise to think you can parse text through a general tool, or one designed for another domain.


I mentioned this in the last article and received some questions. I think we’re heading for a position where technology-enabled employee research will be conversational. The employee will be asked an open question about a particular topic and depending on the response they give a relevant follow-on open question will be asked.

In the short term these technologies will be text-based but it’s likely it will move to voice. I see a future where a system identifies who it needs to speak to, telephones them and asks a series of questions to try and drill-down to the key drivers. It will be able to do this at scale.

The issue to some degree isn’t the analysis, it’s the ability to do this in a way which isn’t intrusive for the user and where there are strong incentives for them to participate. This is what is exciting about research. Analysis is closely related and enabled by design-driven changes. Better interfaces = better data = better analysis = better insight.

The future

Where is all of this taking us?

I see the engagement-app market diverging. On one side are employers who want to understand engagement but do so in a cost-effective manner. These firms will gravitate to the mass of engagement apps offering real-time capture. Buyers should be aware of the value these tools are providing – it’s mainly focussed on delivering metrics in a speedy and cost effective manner. These providers will make most headway in the part of the market where traditional providers couldn’t be cost-effective, ie SME businesses. Those who can add sophistication and scale up to enterprise clients will be disruptive.

The other side are firms and products like our own which see machine learning technology as a way of automating the higher-skilled parts of the historic research bundle. In the first instance firms like ours are using ML to automate much of the skilled work of the experienced engagement consultants.

I think the ultimate role will be disrupting traditional management consultancy. Tools which can ask and understand an almost open set of questions can disrupt much of the information-gathering parts of the traditional consulting offering.

None of this completely removes the need for great consultants, but it means that deep expertise, or the creative parts of the process are where the opportunity arises. Consultants will increasingly focus on developing strategic plans and advising on change management. There might be fewer employed by employee research firms, but I think this shift will increase the demand for those at the top of their games.

Engagement surveys – Part 1, issues with the traditional approach

Engagement surveys – Part 1, issues with the traditional approach V02-01.png

In these articles I use the term ‘survey’ to mean both a survey, with sampling, and a census where everyone is asked.

There is a shift at the moment from long infrequent engagement surveys to shorter, ‘pulse’ surveys which are being used either as a replacement or supplement to the longer survey. With this and the following post I wanted to discuss some of what we see as reasons and advantages and in doing this. As always I hope to give a data-led perspective.

Some background

Organizations, recently through their HR departments but before that via operation research groups, have been conducting employee surveys for around 100 years. In the 1970s the focus was on organization commitment and job satisfaction and the focus went from OR to HR. There had been some earlier work by Katz (1964) on Organizational Citizenship Behaviour which talks about Organization Commitment.

Engagement was first described by William Kahn in 1990 but was made popular by Gallup’s ‘First break all the rules’ book of 1999. Since then most organizations have been conducting some form of engagment survey.

During the same sort of timeframe the technology for completing surveys has changed. In the 1990s and before most surveys were still done on paper. When web technology started to enter the employee survey space we saw surveys which were fundamentally a replication of a paper survey done electronically. This made sense at first as many organizations spent some time doing both in parallel. We still see paper in some environments such as delivery drivers.

About the surveys

Engagement surveys tend to follow a common design – they ask a set of questions to create an engagement index. Mostly this is in the region of 5 questions. They then ask a large number of questions to identify items which are linked to that engagement. Most annual surveys are in the range of 60 – 150 questions long. I would estimate it takes about 20 – 30 seconds for an employee to answer each question.

Data is also used to consider the demographics of each participant. We see both self-reported demographics and demographics that are imported from other HR systems. The latter is a more effective way of getting good data but in some firms there are concerns about privacy.

There is a potentially enormous number of factors that could be associated with engagement. As Kieron Shaw noted in “Employee engagement, how to build a high-performance workforce.”:

“It’s arguably unfeasible to directly measure in the survey all the actions behind engagement,” due to the fact that, “there are potentially thousands of different individual actions, attitudes, and processes that affect engagement.”

Hence, however long the survey is the designer has to have selected a subset of potential factors.

Criticisms of traditional surveys.

In a fascinating paper “Measuring Employee Engagement: Off the Pedestal and into the Toolbox” Andrew Graham of Queen’s University notes 9 issues with the traditional survey:

  1. Not frequent enough
  2. Single scoring leads to issue distortion
  3. Aggregation reduces meaning
  4. Does not capture the specifics (context seldom captured)
  5. Lengthy or poor response planning
  6. Managers are busy & have no incentive to implement any actions
  7. Lot of resources & monitoring
  8. Surveys get old
  9. Causality not clear

A tenth issue that we find as analysts is that there is typically an illusion of richness. Many firms think that by asking 80 questions they are capturing 80 independent data points. This is clearly not the case.

Issues with survey data

One of the analyses that we like to do with survey data is to build a correlation graph. This uses each question as a node and the correlation between each question as an edge. When you visualise survey data in this manner you typically get something like the following:


What we see is a hairball. Each question tends to be highly correlated with another. (In the graph above Questions 31 – 33 are questions that the HR team wanted to add relating to a process which obviously has little link to engagement).

We’ve done experiments with survey data where we ‘destroyed’ 80% of all answers randomly and then used recommendation algorithms to back-fill what had been removed. In most instances we’re able to accurately replace what has been removed. People answer in patterns (hence that hairball), and if you know some answers you can pretty accurately infer what all the others will be (this means that you could probably randomly ask each employee different questions and dramatically shorten the survey without much loss in accuracy).

Issues with User Interfaces

This is a bit more contentious. It relates to how questions are asked.

Most employee surveys use Likert-scale questions, mostly 5 points between strongly agree and strongly disagree. One of the reasons for doing this has been that on a paper survey it’s easy to get someone to code the data into a reporting system (it’s easy to see a check in a box). What has been done is to take this process that was designed for paper and put it onto the web with little thought in terms of adapting the question to take advantage of the opportunities presented by the new medium.

Employees actually have a true feeling on a continuum between the two end points. When you ask them to answer on a 5 or 7 point scale what you’re actually doing is asking them to ‘bin’ their true feeling to the nearest compromise point. Doing so is adding burden on the survey taker and potentially adding inaccuracy in the data. The data can’t be seen as linear, instead one should use statistical methods appropriate for ordinal data.

In a 2012 paper in the journal Field Methods “Why Semantic Differentials in Web-Based Research Should be Made From Visual Analogue Scales and Not From 5-Point Scales”, Funke & Reips show experimental evidence that show that marking a line between two points – a visual analogue scale – has numerous advantages over traditional 5 point scales. Two of these are better (more accurate) data and less burden on the survey taker.

Whether the answer is a visual analogue scale or something with a large but distinct number of points (the 0-10 scale used by NPS practitioners?) is harder to determine. However I see little evidence that 5 points is the right approach.

Should we even be asking scale-based questions?

Finally, too often what drives executive action from survey data is the responses to a few open text questions. As Graham notes on his fourth issue survey data rarely provides context. The qualitative nature of open text does provide this opportunity.

Often the initial action from a survey is to do more qualitative research focussing on several key topics. Such research is both time consuming and expensive. (Arguably acting without understanding the context can be more expensive).

There are instances where asking a scale question makes sense, most notably if you’re wanting to report a trend. However asking sufficiently broad, open questions will likely capture richer data. The challenge for many firms is how to do this at scale.

If we think about how we’d try to understand an issue in a conversation we’d ask open questions and then follow up with relevant and probing follow-up questions. I firmly believe this will be the future of employee feedback, though it will be a bot-based conversational approach which can be done in multiple languages and at scale.

As an industry we’re currently not there but the future is coming quickly. In the next article I’ll describe the current state, our findings of working with clients  at the cutting-edge and highlight some approaches taken by other innovators in this market.

People analytics is a means to an end, not the end

I’ve had a few interesting meetings this week with senior HR managers who are actively building their People Analytics functions. A common concern is understanding how such a group delivers value and therefore how to identify how much to invest and which topics to tackle.

The goal is better decision making and policy changes

The biggest issue that I see is that too often the perceived outcome of an analytics project is seen as the insight provided by doing a good analysis. Therefore teams are set up to deliver insight and results. Companies are going out and hiring data scientists, often without HR experience. The real way of creating value is by making better decisions and making the relevant changes.

I think a contributing factor to this issue is because so many people who are working in these functions come with a background in academia. In academic work the outcome is the research paper.

In the commercial world value isn’t realised until a change is made. I’ve seen far too many projects that end with the final powerpoint deck. The analyst often blames their client for not managing the implementation, their client often responds that the analyst didn’t consider how the recommendations were to be implemented.

One way to increase the amount of times analysis moves to action is for the team to be engaged with the business who are going to be implementing any recommendations at an early stage, and ongoing through the project. It’s important to be looking for policy changes where there is a ROI. You need to be considering the costs of these changes and the ease of realising the benefits.

The objective is not the most accurate model. It’s realising as much value as possible. Factors that influence this are the predicted benefits, the cost of making the change and the probability that any change will deliver the predicted outcomes.

Focus on things that you can change

Most models will highlight a decent number of potential changes. Some of these will be easier to change than others. At least at the early stages it’s better to focus on those things that are easy to change and have reasonable value.

With workforce models many of the most important variables are difficult to change. Things such as tenure or location are usually up there as some of the more important factors. Both are difficult to change in the short term. Others such as time in a role might be easier.

When you’ve got other factors such as communication quality this could be more difficult to change, with uncertainty of managers changing old habits being quite high. Process changes or resource availability could be easier. The only way of understanding this is to have domain expertise.

Ideally, what you will want to do is to identify the costs & uncertainty of any changes. These will be fed into your model to optimise on expected value, not on the workforce variable such as attrition.

Look at subgroups for unintended consequences

Your workforce is not homogenous and whilst a model might produce a good overall effect it will usually do this by increasing some ‘value’ in one area and reducing it in another.

To compound the issue, the value of these subgroups is never homogenous, neither will the cost of implementing changes. In a commercial setting you’ll never be able to explore every question. You need to direct your analysis to chase value.

The HR team have to be aware of not only what outcome will be achieved but also how it will be realised. Ultimately whilst it’s possible to capture as much detail as possible in the loss function, it still requires human judgement to make decisions.

If you’re deploying ongoing predictions, understand how humans will react

Businesses can afford to make rational decisions, your managers often can’t. In most organizations the upside to the individual of making a decision is much less important to the individual than the downside of making a wrong decision. Humans protect themselves from loss.

How the predictions are being presented, the training the individual who is presented the outcome, and the likely decisions are critically important. If the model suggests Jim has a 60% chance of leaving does their manager see that risk as a prompt for decreasing Jim’s risk or as a prompt to reduce the cost of losing Jim, for example by stopping investing in Jim’s development?

We believe, in most instances, it’s better to communicate factors that increase risk, and how to address them, rather than communicating individual-level predictions. With a good model achieving 80% accuracy it’s worth remembering that 1 in 5 predictions will be wrong. Focus on subgroups rather than individuals.

Decide how you’re going to measure the changes

We believe that changes need to be iterative and tested. The good analytics team will work with those responsible for making the changes to design ways of capturing fresh data to monitor and validate the changes. In turn they should use this data to refine their models.

In an ideal world you’d run pilots (a more organization-friendly way of describing experiments). Here it’s important to be looking for the unintended consequences. It’s best to make several small changes rather than bundling changes into a series of large, broad programmes. Small and targeted is the right approach.

Measurement should be designed not only to capture reliable data but also to reduce the friction in capturing that information. This way it’s likely to be measured.

Look for 4x+ projects

So which projects to attempt, and how much is it worth?

Given all this uncertainty in implementation we probably need to be tackling projects where the expected net return (return – cost of implementation) is several multiples of a conservative estimate of value. Our crude guide is to look for areas where the potential value is at least 4x the cost of the analysis.

A good way of doing this is to use your financial model early on to identify the outcome if you could improve the situation by 5%. For a topic which hasn’t previously been improved using empirical data this is usually a conservative estimate yet reasonable to achieve.

The next phase is to build your model in a series of stages. Use milestones at regular places to identify if it’s worth refining the model. Don’t be afraid to stop early. Modelling suffers from decreasing marginal value quite quickly.

Final thoughts

I suspect that 50% of a People Analytics function needs to be people who understand how to make the changes and not analysts. You’ll want this ‘consultant’ role to have as good an understanding of how to make organization changes as understanding the models.

However, it’s important that they have a good knowledge of the way data can support decision making. It is important that they understand how models work, what they can say and what they can’t. They need to understand where to apply analytics and where it’s probably best to rely on managers intuition (because they are making good decisions already in relatively low-value areas). They’re the organizations’ coach on how to apply data to decision making.

The ultimate goal of any People Analytics function is not to build models – it’s to encourage empirically-driven decisions about business issues that depend on the workforce.

Unconscious bias? Using text analytics to understand gender differences in performance reviews

Earlier this year we conducted an analysis on the comments in performance reviews for a major European industrial firm. The purpose of the work was to identify if we could identify gender bias in the language used by managers, and differences in the way that men and women talked about their own performance.

There have been a number of articles recently exploring the same problem. In an article in Fortune, Kieran Snyder reported that high performing women were described as more abrasive than men. In a more recent article in HBR Shelley Correll and Caroline Simard noted that women were more likely to be given vague feedback. They also reported that the majority of references to being ‘too aggressive’ were seen in womens’ reviews.

The fortune article was based on a small data set – 180 people of which 105 were men and 75 women. The HBR article doesn’t mention what size data set they had to deal with though it does mention they studied 200 reviews from a technology company in more detail.

In our analysis we were able to study 36,700 manager comments and 37,300 employee self-assessments. We used our text analytics ‘engine’ that powers our Workometry product to identify the semantic themes behind the statements and then used a variety of predictive techniques to identify those themes which were more likely to be used by and about women.

As well as the text we were able to link both demographic and structured performance data to the comments and therefore themes. We linked data about the manager to the employee data. The following is a snapshot of some of the important gender-related outcomes that we identified.

Replicating the results

When we did this study the Fortune article had been published but the HBR hadn’t. Though we tried hard to find instances of aggressiveness or abrasiveness we weren’t able to reproduce the results that Kieran found. I have no way of saying whether this was due to our much more extensive data size or whether the results that we saw are firm specific (it was a different sector).

The results in the HBR article do have some overlap with what we found, though we initially didn’t interpret the results in the same manner.

Needing to control for lots of variables

One of the issues with much firm-based gender diversity work is that when comparing men and women in a firm we have to control for numerous variables. Women choose different careers, different working arrangements, are present in different proportions at different ages, there are fewer women at the top levels and more at the bottom. With our models we had to ensure that we controlled for these types of factors to isolate the themes which were there because of gender differences and those present because of the jobs women do.

A good example is references to HR systems. Our client, like most firms, had much more women in HR than in other functions (as a proportion of the people in a function). More administrative roles in the business were more likely to be using the HR systems. This theme was one of the stronger predictors of whether an employee was a man / women but this was because of job-related aspects, not gender-related aspects.

Men describe what they’ve achieved, women describe how they achieved it.

This is where our work overlaps with the HBR article. However we found that women used similar terminology in their self-assessments as in the manager assessments. It should also be noted that women were somewhat more likely to be managed by a woman than a man was (because of the gender differences in functions).

Women were much more likely to be described as a team player, to be seen as helpful and supportive and to be sharing knowledge with others. They would embrace changes more than men and be interested in continuing to learn.

Their managers were more likely to describe them as having a positive attitude, demonstrating willingness and determination and as someone who could be relied on. They were more likely to take ownership, go the extra mile and were willing to take on new challenges.

Women were much more likely to describe themselves as responsive. They would highlight offering assistance to others, making themselves and others feel comfortable and working hard.

Both women and their managers highlighted some common themes, several of which had the strongest likelihood to be made by / about women. Women were seen as strong communicators and presenters, being task-orientated, working with high accuracy and following processes and procedures carefully. Interestingly being described as demonstrating analytical thinking was a more female-specific trait

It’s hard to separate the language of business and the language of men

This is where our analysis and that published in HBR has the most overlap. The terminology used by men and their managers is much more likely to be about business results. Many of the male-specific language could equally be seen as the way that businesses describe themselves.

Both men and their managers use phrases such as “Caring about profitability”, “Caring about the competition” and “Having measurable achievements”. What is striking though is that unlike women, where a large proportion of gender predicting themes we found were used by the women and their managers, most male-specific themes were used by either their managers or the men they managed – there wasn’t much on an overlap.

Managers were much more likely to describe men as “making good decisions”,“having good perspective” “meeting targets and goals”, “delivering good performance” and “creating value for the company”.

Men were much more likely to be “seeking constructive critisism” or “asking for honest feedback”. This is of course the heart of the HBR article – that women are given less specific feedback. Our work didn’t find that but did, quite strongly find that men were more likely to ask for this type of feedback.

Unlike the women who were seen, and saw themselves, as being good team players men were much more likely to say they were “interacting with colleagues”. It seemed a much more transactional approach to relationships.

The other aspect that was striking was the way that men used numbers. They talked about how the percentage they had overachieved their targets. Men were more likely to say that “they had a good year” or they met or exceeded their objectives. If you saw a comment which included numbers it was more likely to be made by or about a man. This in some ways is unsurprising. Research in the computer science literature suggest that if you want to design a game to appeal to men you should provide them opportunities to get a high score, earn recognition etc. To build a game for women let them create something.

Both managers and their employees write more about women

One of the things that we found striking was that both women and their managers wrote longer amounts of text. Managers of both genders wrote about 10% more about their women employees than they wrote about their men.

However, a man being managed by a women wrote almost 20% more in their self assessment than a man being managed by another man. As women overall wrote more is this men adapting to the way their managers communicate?

Is feedback language self-supporting?

Whilst we can identify with many of the themes in the HBR article we’re not so keen to jump to conclusions about whether managers are biased against women. In fact in our experience the ‘vague’ feedback is as likely to be used by women to describe themselves as by their managers. We also see that men are more likely to ask for specific feedback.

I wonder whether managers adapt to the terminology used by their employees. If an employee talks to themselves with numbers and ‘hard’ facts then respond with numbers and ‘hard’ facts. If an employee talks about supporting others or being a good communicator then does the manager also talk in these terms?

There is obviously huge opportunity to apply advanced analysis, including the analysis of unstructured ‘text’ data to inform a nuanced conversation about gender within organizations and to help develop effective, targeted solutions. My recommendation for firms wanting to move to a more data-centric approach here would be:

  • If you haven’t done so, do this type of analysis with your own performance data. Given the differences with our study and especially Kieran Snyder’s work I suggest some findings would be industry and even firm-specific
  • Develop data-based development for women to help them understand how they and their male colleagues differ, especially how men are more likely to use the language of business when describing them.
  • Demonstrate using company-specific examples how managers can use different language when describing men and women. This should be addressed to women managers as much as men as many of the terms we found were as likely to be used by women managers as male mangers.

Being predictive in HR analytics is probably less important than you think

A few years ago the talk in HR was about Big Data. Now, the talk is about Predictive Analytics. In both instances people had grabbed a popular and exciting topic from outside HR and claimed that this was the new big thing. In both instances the labelling probably gets in the way of understanding the real value that the core techniques provide. In both instances what’s hidden behind the labels is where the value is.

In a recent podcast interview with Matt Alder I mentioned that size was never really the issue with Big Data in HR – the issue was always that the variety and quality of the data meant that HR analysts were dealing with data that wasn’t easy to analyse. When you hear quotes on the size of data increasing exponentially then it’s worth realising that a large amount of this is unstructured data – text, images, video, sound. The Big Data challenge for HR has always been how to deal with this unstructured  and messy information. Size per se is less important.

With prediction, the prediction per se is usually less important in HR. Unlike marketing where a simple rule can be created – ‘send offer or don’t send offer’ – and to some degree it’s less important how that recommendation was made (and far more important the outcome – i.e. optimisation of revenue or profit) with HR the primary objective of most analyses that we do and see is not the prediction but being able to interpret the model. Understanding what is going on is much more important than just a prediction.

To understand what is going on let’s take a step back and understand what analytics practitioners are talking about. With a predictive model we’re using a series of techniques to do one of two fundamental things: we’re either providing a predicted outcome – usually binary (will leave, will stay) or we’re providing a likelihood of an event happening (a probability, also described as a risk).

To make a decision we need to optimise not on this prediction but on an estimate of the resulting outcome by combining it with a so-called ‘loss function’. A loss function describes the costs and benefits of the decision being right or wrong. I’ll probably cover this in another post as far too many HR applications of analytics seem to ignore the loss function entirely – they stop with the analysis and prediction rather than applying this to help make decisions.

With HR modelling we’ve got a big, difficult issue. For us not only are our models pretty inaccurate and therefore there is reasonable uncertainty in the recommendations even of the best models, but the loss functions are often highly asymmetric and themselves contain uncertainty (i.e. we can’t even be certain of the real value / loss if we make the right or wrong decisions and getting it wrong often can provide a loss multiples the size of the potential benefit of getting it right.)

How good are our models? Let’s take a simple but common application which is modelling employee attrition. Talk to an analyst today and they’ll tell you that the best models at predicting attrition are achieving about 85% accuracy. Is this good enough? The odds of you ‘winning’ at Russian Roulette is 83.3%, so about the same. If the outcome of getting it wrong is small then that accuracy might be good enough. In other cases (like Russian Roulette) it’s probably not.

The natural outcome of a complex loss function and uncertainty in the model is that data should be another voice at the table. We think the number of use-cases in HR where a decision, at least a decision of reasonable importance, can be automated are small.

For humans to make decisions they need to understand not only what the recommendation is, but how that recommendation was made. They need to know what could be driving the prediction or recommendation.

If we start our analysis at the most logical step – the set of actions / responses that you can make – for HR most of those actions are policy-type changes. Those policies may be highly targeted and the parameters might differ by individual but we’re typically constrained to a set of responses.

To make these policy changes what becomes the most important is arguably not the model accuracy but the interpretability of the model.

With interpretability our models become far more useful at driving effective policy changes because they describe what needs to change. The most important way of realising value is by taking the right, and highly targeted, action.

Secondly, we typically don’t have a single loss function because our populations aren’t homogenous. The value / cost of various segments of the population differ. Typically, to optimise the decisions we need to optimise build segment-specific models. This also enables us to make segment-specific actions.

These segment-specific models might not be the most accurate at predicting, but they are much more useful because they describe what we care about and facilitate better policy changes. We have a bunch of constraints that marketers don’t typically have. We need our policy changes to seem reasonable and often need to show fairness by treating various segments in the same way. We need to explain to our employees why we’ve made certain changes and how we’ve identified who should be targeted.

So if the key thing for a model is interpretability, not prediction, why do we place such an emphasis on prediction?

One of the techniques that we’ve found is really powerful is building multiple models, typically using various techniques. Some of these techniques are easier to interpret than others. Others may historically yield better accuracy but can be ‘black boxes’.

Say the policy recommendation of your interpretable model is to raise the salary of a specific segment of your population by x%. If you simulate (predict) what would happen by plugging this back into your model you’re likely to see the best-case result. Your change and model are self-supporting.

What we’ve found really useful is to simulate / predict that policy change on a model built using a different technique. If the resulting outcome is good it will suggest that you’ve found something reliable. If the two outcomes are widely different it is likely to mean you’ve got a policy which is less likely to perform in the real world. Interpretability was important for identifying the change but less important for validating it.

If you really want to add value to your organisations using advanced modelling techniques then it’s not prediction you should focus on. It’s interpretability and incorporating sensible loss functions.

Fixing employee engagement (and why we’ve been doing it wrong)

According to Josh Bersin, businesses spend USD 1 bn on employee engagement. We’ve been measuring engagement for the last 15 years. During this time engagement figures have been flat or declining. Why so little return on our investment?

I’m not going to question the concept of employee engagement. Yes, there are issues including a lack of commonly agreed definition but I don’t believe that this is the issue. Regardless of how you define engagement it doesn’t seem to have improved.

My view on why engagement has been so flat is because of the way the research-to-action cycle has been conducted. Basically, because the measurement & analysis approach has been flawed the actions have been misguided.

Systems dynamics to the rescue.

There may be some readers aware of systems thinking. Peter Senge who wrote about the ‘learning organization’ is part of the systems group at MIT. He’s also an aerospace engineer. This is important.

Last week I had a long conversation with the wonderful Alistair Shepherd or Saberr about viewing organizations as systems. Alistair is an aerospace engineer. Notice a pattern?

When I teach People Analytics the section about applying Systems Dynamics to workforce issues is always the most popular part. People are amazed at how the complex patterns we see in organizations can often be predicted using quite simple rules and models.

Engineering, and especially aerospace engineering is heavily systems based. It gives engineers a different way of viewing the world.

The two basic components of a dynamic system are stocks and flows. A simple explanation is a bathtub. If water is coming into the bath quicker than it is leaving the bath continues to rise, if the rate of outflow is greater than the rate of filling it empties. We can have stocks and flows of pretty much anything.

So what has Systems dynamics got to do with engagement?

With Workometry we’re in the fortunate position of capturing a very large amount of perception data. We have large, enterprise clients running monthly pulse surveys. We store the data in a way that enables us to study at the individual level up how perceptions change over time.

As analysts we built a systems dynamics model for engagement and plugged our data in. It worked, our data pretty much validated the model.

The great thing about systems dynamics models is that you can use them as simulators: make a change to the assumptions and the model will create predictions. We can therefore use it as a management simulator to understand what we need to change to improve what we care about – engagement.

How have we measured engagement?

Analysts tend to use a very simple approach to measuring engagement. Through a set of questions – an engagement index – we measure the number of engaged employees and divide this by the total employees and we get a rate.

If we step back and think about this model there are only two states – we treat people as either engaged or disengaged. With a systems dynamics model we can think of this as two stocks – engaged and disengaged employees, with flows defining people being recruited, leaving and moving from the engaged to the disengaged stocks and vice versa.

What does our data show?

  • Almost everyone joins the organization engaged
  • More people leave who are disengaged than engaged. Not only is this what the engagement literature predicts but it turns out to be necessary for the system to be stable
  • There is a rate by which engaged people become disengaged
  • There is a rate by which disengaged people become engaged. (it turns out that this must be lower than the disengagement rate for the model to be stable and the data certainly shows this)

The model also predicts that in periods of recession, when fewer people voluntary quit an organization, we see engagement levels fall. This is exactly what we saw in the last recession.

How have we been doing analysis?

At a high level we can think of an engagement survey as having two parts. First we ask a series of questions to create an engagement index. We then ask a large number of questions about various factors we think will be linked to engagment.

Traditional analysis uses a bunch of statistical techniques to work out what factors seem to drive engagement. What we’re effectively doing is seeing what factors most closely link with engaged / disengaged employees. We then build action plans based on this analysis

The problem with this analysis, however good your statistician, is that it is static. We take a cross-section of the data at one period. We’re looking at the differences between people who are engaged and those who are disengaged. We haven’t looked at what has changed an engaged employee to be disengaged. Basically we’re looking at the wrong thing and our action planning is therefore misguided.

What should we be doing?

It turns out the model, validated by our data, makes a very simple recommendation – the most important aspect to focus on is changing the rate by which you’re disengaging previously engaged employees. The only other reliable way of changing engagement is to encourage more disengaged people to leave.

Our model shows a very interesting and non-intuitive finding. If you do something to change the numbers of engaged and disengaged employees (our current post survey actions) the results will inevitably be short-term. You always return to the same engagement rate after a few periods!

The only reliable way of shifting the engagement rate is by changing the rate by which you disengage employees. Our traditional, cross sectional analysis is useless at determining this. We spend vast amounts of resources making big changes which are doomed to failure.

How people change their levels of engagement

What’s fascinating is seeing how people change from engaged to disengaged. We have found two generic patterns

  • The Faders: These employees move from engagement to disengagement gradually over a period of around 3 or 4 months.
  • The Fallers These employees move from a state of engagement to disengagement in one month. Often they’ll move straight from a high engagement score to the lowest in one month and the next month ‘bounce’ to a low, but a bit higher, state.

As predicted by the model we see few people who were disengaged becoming engaged.

Recommendations for fixing engagement

Fortunately the changes we need to make in engagement research are reasonably small:

  • Conduct frequent pulse surveys to be close to the change
  • Do longitudinal (sometimes called panel) analysis and look for switching patterns
  • Ask people about their experiences / why they changed their view
  • Identify factors that are related to these switching patterns
  • Act on these factors to change the switching rate.

Surveys in People Analytics models

There have been a number of opinions that suggest that survey data isn’t useful to the People Analyst, especially when developing predictive models. These opinions go against our experience.

In a recent attrition project with a major client we were able to increase the accuracy of our models by around 10% at a relatively early stage by including survey data. Anyone who is doing Machine Learning knows that a 10% improvement isn’t easy to achieve.

A NY Times article written by two experienced data scientists from Google & Facebook, shows how these firms combine survey and system data to reveal what is really going on. As the authors note…

“Big data in the form of behaviors and small data in the form of surveys complement each other and produce insights rather than simple metrics. ”

This is certainly our experience, however how you use it can be non-obvious at first.

Most modelling in HR is Exploratory, at least initially

I would argue that the majority of modelling in HR has exploratory data analysis as at least one of its objectives. Most analysts default to tools such as tree models for understanding rules or something like naive Bayes for understanding variable importance.

HR models will almost certainly be used to inform, rather than automate a decision. Our loss or utility functions tend to be complex enough that for many scenarios data becomes another voice at the table.

For individuals to make decisions based on data they need to have some form of understanding how the recommendation was made. Actions tend to be aimed at altering one or more of the important attributes that is associated with the model and therefore interpretation is important. If we’re maintaining the confidentiality promises made when employees took a survey, for example by setting the minimum size of any leaf in a tree, do we need to use the survey data for ongoing forecasting and / or prediction?

Survey data is brilliant at informing feature selection

The longest part of any machine-learning centric project is feature selection or building. We take a few fields of data and create the variables which we expect to influence the model.

Using survey data in early stages of a machine-learning project can significantly contribute to your choices on which variables to create. For example in one recent attrition project we found a strong relationship between employees who self-reported that they didn’t have the necessary time to do their jobs to the best of their abilities & attrition. We then took this ‘lesson’ and created a set of variables that could represent this in a more accurate manner than relying on infrequent, self-reported data. We use the survey data to guide our feature engineering.

As the analysis continues what we find is that the survey’s influence on the predictive accuracy decreases as we replace survey findings with features that can certainly be used on an ongoing basis for forecasting or prediction without any confidentiality concerns regardless of the population size.

How to use survey data in model building

Survey data can be used a multiple levels within a modelling project, and the level to some degree depends on the question that is being asked. The fundamental need is to have a employee-level unique identifier with each row of data. Obviously if all you have is team data you can answer some questions, but the potential value is lower.

If we think of the organization as a tree where individual a reports into individual b who reports into individual c then via graph traversal it’s simple to aggregate responses to various levels – i.e. what was this individual’s response, what was the response of their team (all at a’s level who report to b), what was the response of the broader team (everyone who reports to c or someone who reports to a manager reporting the c). If you have a project or matrix organization it’s possible this way to have some more complex rules.

Within the feature engineering part of the project we can thus combine survey answers at any level to the data from your HR system. Some answers will add to the prediction at an individual level, others, for example about manager effectiveness might be more sensible to use as an aggregate for all people who report into that manager.

As noted, the models you create might inform what other data you add or collect. This is similar in many ways to including the knowledge of domain experts in your modelling team who can guide the analysts to where to look from their knowledge of the earlier research. The difference with survey data is that it is company-specific.

In conclusion, we believe survey data to be vitally important to most people analytics projects. As analysis is an iterative process we find it a rare project where during later stages, where we try and capture new data to improve an existing model, we don’t include some form of survey or questionnaire.

Insight depends not only on what has happened by illuminating why it is happening as well. Surveys can help provide the second part.

Employee feedback should be more than asking people about their job – it should change the business

One of the most exciting things about developing a new product is putting it in clients hands and watching how they use it.

When we developed Workometry the original use case was for it to capture and analyse frequent employee feedback in a ‘Pulse’ survey. A second use case was for it to be used to understand experiences throughout the employee lifecycle.

To recap, Workometry is our employee feedback tool, launched in 2015 whose differentiator is a focus on capturing and analysing open text feedback. It is a responsive survey front end on top of some very sophisticated computational linguistics and machine learning algorithms.

Our clients usually ask 3 to 5 scale based questions and 2 or 3 open text questions in each questionnaire. Employees typically take no more than 5 minutes to answer the questions.

We first code the responses, using an approach which is unsupervised meaning we detect topics automatically from the data that we capture. Typically we capture between 30 to 50 topics (things such as “shortage of staff”, “better work-life balance”, “the car park situation”) from each open text question. We can do this across multiple languages. We tune our algorithms at the question level.

We then link the perception data to other employee data on an individual-level and use a probabilistic modelling approach to identify groups of employees who are especially likely or unlikely to discuss each topic. These segments can be quite refined, e.g. “women under the age of 30 with university education are especially likely to mention a shortage of career development opportunities”.

Most of our clients started by asking traditional employee-engagement or satisfaction type questions. However increasingly they’ve (and we’ve) realised that the tool can be used for a much more business-specific set of questioning.

One client recently used it to capture and understand large volumes of employee feedback on a newly announced strategic initiative. The two open questions were asking for examples of good-practice where the new approach had been applied and the barriers to execution of the initiative. We collected over 8000 responses during a 1 week period.

Almost immediately after the feedback period closed we were able to start interpreting the results. As often is the case things that are perceived in one part of the organization as a strength were seen in another as a weakness. We can see how manager involvement and buy-in to the initiative reflected on the perception and even engagement of the initiative.

Responses typically were categorised with more than one category. By linking the data together in a co-occurence graph we were able to identify groups of topics most frequently mentioned together. We were able to link the two questions together and look at the linkages between both. We can also filter the scale questions by the topics or vice versa.

Getting such rich data on management challenges usually takes a consulting team considerable time. Workshops are expensive, and by their very nature only include a smallish number of people. Traditional surveys often don’t ask some of the really important questions (who would have thought engaged folks are most frustrated by the car park situation & therefore this was asked in the survey?). What the open text approach does is enable you to apply a qualitative approach in a wide, quantitative manner.

Of course executives love being able to quickly gauge the opinions of their employees, especially those at the coal-face whose thoughts are often filtered by layers of middle management.

What’s clear to us is that a conversational approach, asking open questions to the whole organization, summarising and communicating the results quickly encourages participation.

With several hundreds of thousands of rows of data captured and analysed in the last months I think we’re only just starting to understand how to get the most out of this approach. However, what we are seeing is the passion of employees to have their say on key aspects of how the firm treats not only its’ employees but the customers and broader community they operate in.

Being able to ask people in their own words is revealing the collective wisdom of the whole organization.

postscript: why should HR be involved?

One of the most powerful parts of this approach is using employee data ranging from demographics, performance data, job information and even business data to identify segments of employees who are more likely to answer in particular ways. In most organizations HR is the controller of such information.

What is important though is for HR to feel confident to move the conversation with employees from exclusively their perceptions of their jobs to a broader, hybrid approach which includes business topics. We know from comments that employees really value being listened to about their views on how the organization can better respond.

What is employee engagement good for?

Greta Roberts, who I have a lot of time for, yesterday published an article called Employee Engagement? It’s Just A Meaningless, “Feel Good” Business Metric. Looking at our data and analysis we disagree.

Greta describes Engagement as a middle measure. We would probably use the term ‘leading indicator’. I would argue both are emotional terms, but both are potentially accurate.

The article mentions 6 reasons, each of which I think are worth addressing.

Employee engagement isn’t the goal. Business performance is the goal.

I suspect a good parallel is that customer satisfaction isn’t a goal, sales is. Greta mentions that most businesses don’t and can’t link performance to engagement results. My view is that (a) I suspect more businesses could do this if they wanted – just ask their survey provider and (b) this argument is more a criticism of howengagement is measured and analysed than it is of the usefulness of measuring engagement.

Justifying any kind of program based on someone else’s research is a less than rigorous business practice.

OK, there isn’t much to disagree about this. Much of the published research has (a) been mostly correlations, written as to provide an impression of causation (b) written by consultants with a tool to sell. We’ve done a big literature review on this as part of a presentation to an industry group.

OK, so one could level this claim on us, however we’d be more than willing to quantify the results with clients. I think it’s necessary to justify the ongoing use.

Are these studies useless? For the analyst they’re not. I would argue one of the ways of using them is as hypotheses to test. Testing others’ findings in your own context is a valuable activity for analysts. In some ways, it’s one of the advantages of using a firm with deep domain expertise, like Greta’s firm or our own – we often know where to concentrate analysis resources.

I propose that you should be starting an engagement project with the objective to link engagement to performance. Design the intervention on that basis. Run an experiment but certainly capture the data in a way to enable you to do the analysis.

Rigorous analytics often show little or no correlation between high engagement and an increase in business performance or a decrease in turnover.

This conflicts with what we’re seeing from our data and analysis.

Now, let me be clear, there is muddied water between the relationship between engagement and business performance. The key issue is that whilst we see engagement linked to performance we also see employee populations of higher performing companies more engaged.

There are a few ways of doing this analysis to disentangle the results. First, you need to capture engagement at an individual basis. In truth most survey firms do this. Most surveys are confidential rather than anonymous. Whilst we don’t report or analyse groups below a certain number we do have the ability to do analysis using linked data. It’s pretty easy to tune ML algorithms to do this and also reduces the chance of overfitting.

A simple test to see if your survey is really anonymous – if you’re survey isn’t asking employees where they work they’re almost certainly linking the perception data to demographic data later because you’ll need to link to function etc for reporting purposes. There is no reason you couldn’t be linking individual performance data, for example from a CRM system this way.

Second, as engagement data is captured more regularly it’s becoming more valuable to analyse as a time series. The additional frequency is helpful identifying the order of events – i.e. which comes first, the engagement or the performance? We could use such patterns to infer causality. In fact when a presenter from IBM was questioned on how they had identified causality to engagement during a presentation I chaired at People Analytics 2015 it was via time-series data.

One of the reports we provide for clients with Workometry is the rates of which employees are shifting between engaged and disengaged states. We do this by looking at patterns of engagement at an individual level over time.

An earlier one of my posts used systems dynamics to explain why this is so important. I argued that it isn’t the amount of engagement that is important, but the rate that the business was disengaging people. Either way, you need to be measuring engagement to get this.

Engagement is a middle measure.

Lots and lots of KPIs are so called middle measures. The number of sales prospects doesn’t pay the bills; as I mentioned, customer satisfaction isn’t the same as sales. Few managers would question if you recommended measuring these.

We measure and target these measures because we see a link between the measures and what we really care about. Of course if there is no link then there is no point but (a) make sure that you’ve done the analysis with individually linked data (b) look at the rate of switching or other change over time metrics.

Engagement scores are not actionable.

There are two sides to this. Greta mentions that data isn’t linked which as I’ve argued is unlikely to be true. With this data it’s quite possible to identify events that precede a change in engagement at an individual level. Second we can ask employees about such events, and if we do this close to the event we increase the chance of getting accurate data. Workometry provides the option of triggered feedback requests by key events & integrates the data from this with regular pulse surveys.

Customer research suggests that such events are a big driver of customer engagement. Our research shows the same. The data clearly shows when a ‘shock’ event has occurred and where engagement at an individual level decreases over time. We can also see how and when people regain engagement (at a much lower frequency than those who become disengaged).

It’s a vanity metric for the company.

Can it be? Yes, of course. Nobody should measure engagement and accept the links as gospel. In fact, I would argue that there is little point measuring it if you accept it without being critical about it’s application in your firm – in these instances it would arguably be as useful to implement engagement-raising measures without measuring the outcomes. It might surprise you that some HR departments do initiatives without implementing measurement systems at the same time. It’s not even just limited to engagement!

Final words

Just because sometimes measurement is poor doesn’t necessarily imply that the measure should be abandoned or rubbished. If there is strong anecdotal evidence that a relationship often exists I think the smart approach is to try and replicate it with your own data, measured in the most effective manner possible.

From an analysis perspective what we’ve found useful as regards engagement are three things:

1) To link engagement with other measures at the individual level, measure it frequently and treat it as time-series data 2) To create derivatives of variables when doing feature-engineering pre-analysis 3) To use systems dynamics to understand how engagement interacts with other ‘harder’ variables. This simulation-based approach will often provide better forecasts and explanations than simple machine-learning approaches.

2015 in People Analytics

At university, my Econometrics Professor used to always say ‘don’t predict the future as you’ll most likely be wrong’. (He also was paid every year by a big investment bank who wanted him to try and do just that.) Hence in this post I decided to end the year taking a look back at 2015 and what has happened in People Analytics, seen through the lens of one of Europe’s prominent People Analytics practices.

People Analytics has taken off

I think 2015 was the year when we started to see real momentum in firms doing People Analytic projects. Let’s be clear, most firms still aren’t doing anything substantial but those who are are now starting to bring it into most of their initiatives creating ongoing and substantial demand. Those who haven’t started are certainly being left behind.

There is certainly more hiring in this space. Our clients have increasingly got numerous people in their HR analytics functions. The skill levels are increasing. We know several firms with strong, analytical Phds in their teams and an increasing number of analytical MBAs. An increasing number of our projects have ‘knowledge transfer’ as an aim.

HR leaders are eager to become more data-fluent

In 2015 we’ve run analytics training in Europe and Asia and the diary already includes sessions throughout 2016. The audiences have always been similar – senior level HR managers who want to understand enough about analytics to be able to identify opportunities, select vendors and generally feel confident using data to inform decision making.

We’re seeing this move down through the organization, and to more junior levels of HR. We focus on providing skills to help people think analytically rather than teaching how to do analytics. Knowing how to manage analysts will become a core skill for HR and managers need to know enough so that these relations can be fruitful.

Predictive analytics is rapidly becoming standard

OK, so I have a big problem with people using a binary classification between ‘doing’ and ‘not doing’ predictive analytics. We’ve always, since founding in 2010, been using machine learning techniques to help make better decision making for HR. It’s pretty straightforward to build a simple model but to increase the accuracy or usefulness requires greater and greater effort.

It also shouldn’t be seen as an activity which can be completely automated. There is a big difference between taking photos of your friends with your smartphone and creating work like Nan Goldin. As any analyst knows, there’s a lot of craft in building models. It takes time to learn, domain knowledge is vital – not just of HR but also of the specific business situation and how the data has been collected.

In 2015 we saw a continuing push by most big HR technology vendors to provide ‘predictive analytics’ into their products. Clients need to understand whether these generic models are good enough for their needs. My suggestion is that if you’re wanting to gain competitive advantage they’re probably not. There’s also a question of how the results are applied but that probably demands a post to itself.

Graph analysis is becoming more regularly used

The next two trends I predicted during a presentation that I made when I chaired HR Tech Europe’s Big Data conference in early 2013. They’ve now really come to fruition.

Last year I wrote a post on graph or network data structures and how they provided a different way at looking at your data, enabling new questions to be answered. For us 2015 was the year when using graph analysis became mainstream.

I can’t think of one project this year which didn’t involve some form of graph analysis. Some of the time it was about the relationships people had within their businesses but often it was simply to look at data, or metadata from a different perspective.

Our dominant tool for graph analysis this year has been igraph used via the R package. I got a very nice, slightly early Christmas present when Gephi 0.9 was launched on 20 December. During my training I use the wonderfully user-friendly yet powerful Polinode to teach networks and network thinking to teams of senior HR managers.

Text analytics is providing a rich goldmine of information

I wrote in the last post about how text analytics is transforming the employee survey, a central part of our Workometry platform. However in 2015 we saw a strong demand in text-analytics-centric work across HR.

Some of my favourite uses of text analytics have been from looking at 360 review data and even with operational data from a large shared service centre. Text projects almost always involve graph analysis and prediction.

In 2015 we’ve learnt that few HR tech vendors have ever expected the data in text fields of their data to be analysed. It is no longer, unfortunately, a surprise to find numerous systems-generated quasi-xml littering a text-field data extraction. 2015 was the year we got very good at regular expressions.

HR Analytics platforms have fallen flat (with some exceptions)

What we haven’t seen in 2015 is a growth in the usage of dedicated HR Analytics platforms. I covered these last year in a post on The 5 types of HR Analytics Vendors (they’re the technologists).

I think that HR Analytics platforms fall into a difficult space. On one side we see the rise of more sophisticated analytics in HR systems as standard functionality. On the other there’s a realisation that to do the job well requires business-specific models / good analysts. The platforms find themselves in the middle – arguably in no-mans land.

There are some vendors which we like. In 2012 I wrote about OrgVue and it’s been good to see how they’ve risen. I like what the Talent Lab team have built. In both these instances the platforms arguably have taken a specific analytic need and built a tool which meets that need. They’re not trying to provide an all-encompassing solution.

It’s this specific-need route we’re taking with our two products. Workometry can be used by analytics teams in their models – the datastore can be queried directly by the clients’ team and the data comes in a very modelling-friendly manner though of course it can be used as a service in it’s own right. We’re currently working with a few clients on another, early Beta product which takes ‘standard’ data from HR systems and automatically generates the type of features / variables that are typically needed in predictive HR models. I suspect this will end up being an API service.

2015 conferences

I suspect that we’re getting close to saturation with People Analytics conferences and that over time People Analytics will just become part of most HR conferences.

In 2015 we did over one conference presentation / chair a month and it was great to see them moving from ‘why do analytics’ to really sharing some successes. It was also flattering to find clients talking about some of our work at conferences, sometimes even without our prior knowledge.

If I had to pick one presentation that I enjoyed doing in 2015 it was the one that I gave at HR Tech Europe in London on ‘Open Source People Analytics’ where I tried to show what People Analysts actually do using tools such as R and RapidMiner.

During the hour demonstration I showed how to visualise data using ggplot2 with labour market data via calls to quandl. Using R I did live analysis of the conference twitter stream and geospatial analysis using UK supermarket locations, including calling APIs to get driving times between locations. I demonstrated building simple prediction models using RapidMiner. Of course being live things didn’t completely go to plan but it was huge fun and I got great feedback from the audience.

2015 thanks

There are far too many people and teams to include here so apologies if I miss you out.

Of course the first and biggest thanks goes to the OrganizationView team for doing such great work, being always open to try new things and for putting up with my demands to discuss algorithm selection. We achieved great things & I couldn’t have achieved it without them.

A very close second goes to our clients. We’ve had some great problems to solve this year and of course they pay the bills. We can all say we’ve learnt a huge amount helping solve some thought-provoking and challenging issues.

We couldn’t do what we do without the great work that the R community has done this year. R really goes from strength to strength and should probably be the default choice for HR analytics teams. I suppose a related thanks must go to all those who pose and answer questions on stackoverflow.

Finally, a big thanks to my peers in other analytics firms and those who are providing great services & tools to this ecosystem. Even though we often compete I think in general the community has made a big contribution to raising and refining the awareness of People Analytics.

How text analytics is changing the employee survey

How text analytics is changing the employee survey-01.png

In my last post I discussed the process by which employee engagement increases and decreases and therefore what are the most effective interventions leaders can use to make a long-term difference. In this post I explore the related question – what do we need to change?

As People Analysts OrganizationView have conducted a lot of surveys over the last 6 years. One thing we’ve come to appreciate is how important employees’ open text comments are to driving action from management. The issue has usually been how to deal with tens of thousands of comments, provided in multiple languages. This was one of the driving forces behind our decision to develop Workometry, our employee feedback platform.

Before the advent of reliable text analytics, analysing these comments at scale was time and cost prohibitive. Much of the way employee surveys have developed is because of this constraint. However, just in the same way that statistics has changed by the availability of large, easily available computing power and cheaper acquisition costs for data we predict that understanding employees will change by the availability of text analytics.

With text there are two main tasks that we want to do: we want to categorise the text into one or more topics and we might want to score the text on some sort of scale, for example a level of sentiment.

When categorising we want to go further than just words. We look at two dimensions – the topic (eg ‘career development’) and the context (eg ‘shortage’). This has to be more than just keyword as we’ll want to link together the multiple synonyms – it’s the meaning not the words that they’ve chosen which is important.

Doing this is adding metadata to our text. We can then apply various statistical techniques to the metadata. Typically we’re identifying in the region of 40 – 60 different topics for any text question. We can think of this as equivalent to adding another 40–60 scale questions to a survey. Therefore we can ask the short surveys that are needed to maintain response rates when you’re doing them frequently whilst capturing very rich data. We use an unsupervised learning approach meaning that the topics are suggested by the data, not precoded based on generic results.

One of the reasons that we do analysis is to draw attention to the parts of the information that managers need to focus on. We’re currently doing that by combining two techniques.

One of the joys of working with Employee data is that we often have extensive records about the employee – who they are, what role they’re doing, who they work with, how they’ve performed and the key events (job changes, pay rises etc). Linking this data to the perception data lets us provide much greater insight than if we just analyse the much smaller survey data on its own.

With Workometry we use probabilitic models to identify which groups are most likely to discuss a certain topic. We can incorporate information from HR systems, business systems and even other survey answers provided at a different time – for example looking at their perceptions to their onboarding process.

The results from these models can then be used within interactive visualisations to guide the user to the most interesting groups / results. The interactivity lets the user drill down into the data in new ways, guided by the models and ultimately lets them see the underlying text comments.

One very simple, but from our experience very powerful, way of looking at text data is to identify which topics are most likely to be discussed by engaged / disengaged employees. We see clear patterns that demands for work become far more transactional as a result of people moving from engaged to disengaged. This fits with information we get from exit interviews. We can think of a two stage process – first they become disengaged & then some leave. This supports the systems dynamics model I discussed in the last post.

Finally, what we’ve learnt from looking at vast quantities of text in this manner is that in a survey information in text comments seem to change much more quickly than scale-type questions. For one client we saw a jump in comments about technology 3 months before we saw changes in a related scale question. This ability to provide information to enable managers to fix issues whilst they’re still emerging should be seen as a key benefit of collecting and analysing text.

Demonstrating People Analytics

Deloitte recently published their annual Human Capital Trends publication. In it, they gave their article on people analytics the subtitle ‘stuck in neutral’.

It’s now over 5 years since I gave up big corporate HR life to build a startup in the people analytics. I had seen how data and analysis had revolutionised marketing and felt it was inevitable it would do the same to HR. In fact one of my earliest projects with OrganizationView was in a marketing department, using visualisation of their activities to drive a behavioural change in the marketing team towards collaboration.

Back then HR was interested in talking about analytics but had little real interest in starting. Now I think we’ve changed to knowing that it’s coming, that it’s of benefit but being confused about how to start.

There is a thought, propagated by some of the technology vendors, that technology solutions can solve the problem. I don’t fully agree with this view.

Certainly certain common requests can be automated but the key part of good analysis is linking it to the business strategy; not just enabling but providing opportunities which might help refine how you’re creating competitive advantage. This takes experience and knowledge – it takes good analysts.

Next week at HR Tech Europe I am running a session on People Analytics with a difference. Instead of presenting case studies or trying to demonstrate the potential ROI I’m going to show the sort of things people analysts actually do.

The session is called ‘People analytics using open source technology’. It’ll be a demonstration not a set of slides. I’m going to show a few different analyst activities that can be achieved quite easily without having to make a big technology investment. I’ll walk through how to do it and provide support materials so the audience can try it themselves.

Some of the things I’ll show how to do include:

  • Exploring data, spotting outliers, dirty data, reshaping it etc.
  • Creating great quality graphs that can be used to communicate
  • Using external data sources through APIs
  • Using location-based data, for example finding commute time for employees
  • How to get started with building predictive models

I hope to finish by doing something fun – analysing the twitter stream for the conference in real time.

When you look at the job descriptions for people analysts at the firms who are arguably leading this field they’re asking for experience of the sort of technology I’ll be using.

If you’re an HR Director I hope that I’ll be able to show you that moving from neutral into first gear doesn’t need to be a big step. If I can raise the confidence of some of the audience to go off and do some interesting things with their data I’ll have met my objective.

Separating reporting and analytics is (usually) a bad idea

One of the common things that HR does when building an organization to deliver analytics is to separate reporting and analytics. In many instances this is a bad idea.

I worked in one such team way back. The HR analytics team would do the statistical work, build measurement and models then (or so was the idea) reporting would run these as operational activities. Problem was that it doesn’t work. Far too many great solutions would not be moved into production effectively, arguably because they were never designed with production in mind.

What are some of the problems?

Analysts and reporting teams speak different languages

A generalisation I know but if you look at the skill sets and backgrounds of the analytics teams and the reporting teams they’re often different.

HR analytics teams in big firms often comprise a large percentage of I/O psychologists. Different disciplines historically have used different analytics tools. I’m an economist by education and economists often use Stata. Psychologists often feel most comfortable with SPSS because that’s what they used at university.

Reporting teams often have a decent number of people with a more traditional BI / IT background. They’re comfortable with enterprise BI tools, relational databases and SQL.

A problem I’ve seen time and time again is that the analysts don’t give enough attention to the process of how their analysis is going to be moved to a production environment. Many are using their tools in a point-and-click manner which makes reproducibility difficult.

Often this difference in background and ways of working breeds mistrust.

The outcome is too often predictable. The analysts end up doing projects which are one-off pieces of analysis. The benefits fail to be realised because the analysis outcome is a slide deck rather than tools to support ongoing decision-making.

Great reporting needs predictions

The other issue is more fundamental, and that is what information is needed to create good reporting.

There are two good reasons to do reporting. Either you want information to enable decision-making or you want to change behaviours. In both instances reporting which includes predictions will likely be more effective than that that only presents historic data.

We all make decisions based on our assessment of what is likely to happen in the future. With traditional reporting we do this be presenting data on past events and leaving it to the audience to predict what is most likely to happen in the future. Lots of studies show know how poor we all tend to be at predicting the future!

Good reporting takes a different approach. It starts with the decision that needs to be made and then uses past data and a predictive model to explain what we expect to happen.

A good example of such a graphic is the Bank of England’s fan chart, an example of which is shown above. The fan shows a range of probable outcomes based on the model. You will see similar charts in finance or in scientific publications.

Using models like this in reporting can be shown to increase the effectiveness of decision making. In our experience it can also increase the demand for reporting.

Multi-disciplinary teams are the way forward

Over the last 5 years of OrganizationView we’ve learnt how to deliver great reporting for HR clients. Our finding is to do this really well you need a multidisciplinary team:

  • statisticians are needed to correctly interpret the data and build predictive or forecasting models
  • technologists are needed to automate report generation and distribution. They’ll probably be needed to move the statisticians models to a production environment.
  • graphic designers / data visualisation specialists are needed to design visualisations that are effective and highly professional. We target the quality of a good company report or the visualisations created by the likes of the Financial Times, Economist or New York Times.

Experience has shown that all these people need to be engaged at the beginning as recommendations and work of one will impact the work of the others. It’s hard to do that if you separate reporting and analytics.

How employee measurement is changing, and what this implies

Companies have vast amounts of data about employees, but probably won’t have all information for the decisions they need to make. It’s therefore rare to find that a people analytics project, or even good reporting analysis doesn’t include the need to collect additional data.

Over the last 18 months we’ve seen a big shift in how companies, and especially HR departments, are thinking about measurement. The headline would be ‘measure less, more often’.

Technology is obviously facilitating, but not necessarily driving the change. The biggest driver we see is a changing expectation in business about the recency of information. The executive team are asking the CHRO why some data is only changing on a yearly basis. HR leaders worry that theare being left behind.

The datasets that historically have been captured yearly include:

  • Performance management data
  • Career discussion data
  • Employee engagement

Of these, the one that seems to be moving most quickly towards regular measurement is employee engagement. I can’t remember a client asking for an annual survey for at least 18 months. The most typical frequency we’re seeing is quarterly though several firms and moving to monthly. We’re also talking to one large firm about what will be, in effect, a weekly survey of a sample population.

Moving to more regular surveys make several important demands on the process and technology. Much of this is driven by the greater need of immediacy – if you’re not quick the next survey is upon you.

  • Reporting / analysis needs to be very quickly produced and distributed. Automation is vital
  • Action planning will probably need to be bottom-up
  • Discussions tend to happen at a more localised level, usually team level
  • Our experience is that managers still want information in a format that can be printed, however we’re seeing more demand for one-side ‘infographic’ style reports – quick to comprehend and presented in an engaging manner to promote a good discussion
  • Interactive reporting is needed for central teams to identify / edit stories that can be presented to executives. At a local level interaction rarely adds anything & mostly serves to increases the clicks needed to get what the user wants
  • Communication has to be adapted – big ‘survey-event’ comms is out, replaced by more community-based or social. Think ‘always on’
  • To get usable data from a shorter survey which is often designed to be discussed at a team-level increases the importance of open-text questions
  • On a company-wide level this is driving far more sophisticated text analytics needed to make sense of a continual stream of unstructured information

Many of these points would apply to other types of employee measurement, most of which are based on subjective data, if the frequency increases. Of course the gold-standard is to link this data with other, probably observational data. This could include other non-structured data such as company social systems. As a rule, only ask questions that are difficult to infer from other sources.

Finally, companies need to pay attention to how the employee will benefit from providing data more frequently. This needs to be something that they want to do or the initiative will fail.

The implications of the skill crunch for HR analysts

The conversation we seem to be having at the moment most of all with CHROs is about acquiring and developing the talent that they need to do people analytics. For many I don’t think they’ve considered this in detail to date. It’s going to be a sizeable challenge.

In Q1 2014 I gave a conference presentation to a senior HR audience here in Switzerland where I addressed this issue. I showed job after job where HR departments were looking for folks with advanced degrees in quantitive subjects and where instead of asking for experience with Excel they were asking for SAS, R or Python.

These weren’t just the high-tech firms like Google, Facebook or to some degree Amazon who have been looking for these people for some time. These were jobs at retailers, at oil companies, in FMCG. From what we’ve tracked this year whatever sector you’re in either you or one of your competitors is starting to do advanced people analytics is a serious manner.

We hear about the stories of what people are starting to do. The Google case-studies have been widely reported. However, let me put it to you in another way. I don’t think we’re hearing about the really interesting stuff. The really interesting things are the projects where firms are creating real, sustainable competitive advantage and they’re certainly not going to want to broadcast that to their competitors. What’s more, it’s relatively easy to quantify the value the projects are realising.

The key story is this:

  • the interest in HR / people analytics is growing tremendously
  • the demand from firms for these people is rising at a rapid rate
  • the skill requirements are shifting quickly
  • the stock of folks with people analytics skills is small as few firms have historically done this
  • lots of other functions and businesses are looking for exactly the same skills.

How to persuade analysts to come and work in HR? That’s a good question. Certainly the compensation levels need to be adjusted to reflect the real competition. Can you define a compelling career path?

Earlier in the year I had lunch with an old friend, a statistician in charge of the marketing analytics team for a big bank. I described the sort of people we were looking for & asked if he knew anyone. “Sure, I know lots. We employ lots of these people as quants on the trading floor.”

He was only half joking. Take this bank (recruiting via a third party). They are hiring the 3rd person in a team doing people analytics, so we’re not talking about the team head. Advertised salary – Circa $150,000 – $250,000 + exceptional bonus and full benefits. As the ad states:

“The chosen Candidates will be rewarded and compensated at the same level as front office Quant strategists and will grow exponentially in headcount and importance through 2015.”

As HR leaders take some time off and think about their challenges for 2015 I would put having a strategy to how to deal with this skill vacuum high up on the list. Are you ready to do what is needed to compete for these people with your competitors? Given they’re seeking to deliver competitive advantage can you afford not to?

The 5 types of HR Analytics Vendors

In my last post I mentioned ADP as part of a group we call the Data Aggregators. Since that article several people have asked what the other vendor categories that we see in the HR Analytics market.

Running a firm in this space I feel it important to be able to define our broader market to help stakeholders understand where we fit in. I think it’s a useful framework for understanding how the market is developing.

Firms will often fall into multiple categories but have a dominant approach which drives their business models. Business models are important, even if you’re a client because it can make explicit what firms are selling and ultimately what is the product.

The question also is ‘what is HR analytics?’ We have to set the boundaries somewhere. For this classification I’m counting firms in if their creating tools that repurpose multiple data sources, especially a firms’ own data, in a way to enable better decision making. Doing so excludes two groups:

  • Technology firms who make products which are useful to do HR analysis. SAS would be an example of this type of firm. Analysts will typically want such a tool, but the tool by itself doesn’t provide the service
  • Measurement firms, like survey houses or selection tool providers. This was a harder one to decide on. Again, I took the view that measurement was a building block for analysis. Many could fall into Data Aggregators but I think many are wedded to their old ‘process’ mindset.

So, without further ado, here is the OrganizationView HR analytics market classification:

The Technologists

The Technologists take an IT-centric approach to analytics, with a revenue model that is heavily based on software licences and implementation services. In today’s market software is likely to be cloud-based.

A Technologists’ approach is likely to be successful if you are convinced that your requirements for analytics are predictable and stable. Alternatively they may be useful to cover the basics in a hybrid or combinational model.

The Data Aggregators

There are several firms who’s strength, and to some degree value, is from their large stores of cross-firm workforce data, usually at the level of an individual employee. These firms are now creating new business lines through the aggregation and analysis of this data.

During the last few years several large acquisitions have been linked to this approach. These would include CEB’s $660m acquisition of SHL and IBM’s $1.3bn acquisition of Kenexa. Previously conservative firms like ADP are worth watching in this space.

Analytics as a Service providers

Analytics as a Service firms can provide an analytics CoE capability on an outsourced basis, either to manage fluctuating capacity or specific skills. Most firms in this category will bring their own technology which enables them to automate common activities, delivering economies of scale. Whilst in other functions or sectors the market is established it is new within HR. The relatively small size of deals has made HR unattractive to date for the larger firms such as Accenture, Mu Sigma and Fractal, though all have shown interest in this area and will probably become more active as the market develops.

OrganizationView is an Analytics as a Service provider

The Consultancies

Most, if not all of the major business consultancies with HR teams are offering People Analytics service lines. Whilst many have built, and in many instances retired, People Analytics technology, their project-based services are likely to meet the needs of many HR buyers.

EY is a good example of a firm in this space. Assignments are likely to be in the areas of workforce planning, workforce productivity & performance improvement modelling and setting up and advising analytics CoEs.

The Start-Ups

Some of the most exciting and advanced analytics currently being done in HR falls within this sector. Most firms in this sector make one or two products targeting niche areas. Many of these are in the area of hiring, especially for certain in-demand groups such as IT skills. The availability of large volumes of public data – including social media data – make machine learning to identify patterns possible. These services can be great add-ons to your analytics approach


I don’t think that any of these firms will dominate the overall market. Large organizations are likely to want to develop and manage an analytics ecosystem. It is possible that the analytics as a service providers could act as the cultivators of such an ecosystem on behalf of clients, in much the same way that an architects practice manages a host of specialists.

At the moment the market is immature and growing quickly. It’s a good time to be in HR analytics.

#HRTechEurope – The power of collaboration | Yves Moreiux, BCG

HR Tech Europe was kicked off by Yves Morieux of BCG who gave an interesting presentation on how what we’re seeing in the workplace is, at least partly, due to the increasing complexity of organization structures.

Complex organization structures divert effort of the members of the organizations – the managers and employees – from doing the work to navigating what Yves called the labyrinth.

This barrier to work is one of the key contributors (according to BCG) of the declining engagement that we see in the workforce across most developed nations (and emerging markets if our experience applies). Employees are fighting the organization structure that increasing organizational complexity has created.

Where to go? According to BCG the answer is relieving collaboration. Here is where we diverge in opinion. According to Yves collaboration can’t be measured. Our view is anything that can be observed can be measured. I believe that it only can’t be measured within our existing frame of reference.

Conversations with others after the presenation hinged on the difficulties in measuring collaboration. My own view is that if we store information in a traditional table structure, be that flat tables or a relational database, we encourage our analysis to be an the level of aggregation of the data – ie the row, typically the individual.

If we want to measure collaboration we need to be looking at data which focuses on the relationships between people, not exclusively about the people. This is hard to do in a table, but is natural in a graph-based data structure. We’re letting our tools force our thinking.

BCG had hit on a critical aspect or deficiency in what we as a function are doing. What drives performance is not the individual but the collaborations, the relationships between the poeple.

Predictive analytics in employee engagement reporting


In the last post I explored how the opportunities presented by data-mining could enable us to create and use surveys which reduced the number of questions we ask employees whilst potentially expanding the survey’s breadth. This one suggests another use of data-mining – to build predictive models to help understand survey results.

When presenting data we should always identify and use relevant comparitors to put the current data in context. With surveys there are three comparisons we typically make:

1) We compare with other internal groups – eg comparing one function’s results with another

2) We compare with the same group in a different time period (to see change)

3) We might compare with an external group, such as another firm in the same industry.

The validity of any comparison depends on how similar the groups in question are. What we’re really trying to do is hold all other potential variables constant and therefore show that what we’re displaying (the results of a survey question or dimension) is due, at least in part, to the differences between comparitors.

How good are these comparitors at doing that?

If we take the worst first then we can see that comparing your employees to those in another firm has too many differences to produce strong validity. There are too many different potential factors of differences in the variable in question for us to draw any real conclusions. These comparisons are good to massage senior managers’ egos but they’re not terribly useful to drive good decisions.

The other two comparisons are much better and we’d recommend using them. However, we add another comparitor which can further aid comprehension of what is going on – the predicted value.

Predicted Engagement as a comparitor

Let’s consider Engagement, typically a significant question for any employee survey. What we’re really trying to understand when doing a survey is twofold:

  • Where do we stand in relation to our chosen comparitors?
  • What can we do to improve levels of engagement, and hopefully the business outcomes we really care about?

If we take the former question, as mentioned above what we want to show is the difference in engagement due to the variable we’re using to compare, eg the function. We therefore need to account for the other potential reasons for a difference.

We know engagement is related to both the experience of the employee and certain underlying factors such as tenure. Engagement over tenure can often be shown to form a bathtub curve, where it first falls as the honeymoon period wears off and then increases as disengaged employees select themselves out by leaving the firm. There are other similar relationships either industry-wide or specific to the firm.

Let’s make a simple example. Let’s compare two departments, A and B. Department A has mainly established employees, and therefore a natural level of engagement matching this group. Department B is new and hired from outside. We’d expect Department B to have a higher level of engagement regardless of how the experiences that employees in either department had received.

How do we make realistic comparisons between these groups? We use a predictive model. This model assigns a probability to each employee of them being engaged based on who they are and their employment history. We then roll these engagement probabilities up to the level of the group being compared – in this instance Departments A and B. 

With this figure we can visualise the data in two ways:

  • We can show the levels of engagement between the two departments with two comparitors – the other department and the predicted level for each department. We use variations of bullet graphs to do this.
  • We can show the differences of both departments recorded engagement levels from their expected levels (ie observed – expected). In so doing we can rank departments based on the differences from their expected level.

As with any model it’s not perfect, but we believe that by using a predictive model to present survey data in this way is reducing the issues of making comparisons between groups and thus increase the likelihood we can make effective decisions.

Workforce planning – the dynamic approach


Better decision-making depends on us reducing uncertainty about what is going to happen, not what has happened. Of course, the past is often a good predictor of the future but a way of understanding how the future is likely to develop is key.

One approach that I’ve found particularly appealing over the last 10 or so years is that of systems dynamics. There are two defining elements of this approach; first that the business can be thought of as stocks and flows. HR falls nicely into this approach – on one level the workforce can be seen as a stock of employees who join, leave and are promoted or move. But we can also think of stocks of skills which the business can develop.

The second component is to think of the system not exclusively in a linear cause / effect manner but to consider feedback loops and how these help or hinder the development of stocks. Put together these two elements enable complex, and often counterintuitive behaviours to be understood and predicted.

One of my earliest introductions to system dynamics was through a pre-publication copy of Kim Warren’s 2002 book ‘Competitive Strategy Dynamics’. I recently took some time to talk to Kim about his work and how dynamic models work so well for HR.

Hi, how did you get started with Systems Dynamics?

I have a career background in strategy – I was head of strategy for Whitbread retail arm and wanting a change I left to join London Business School to teach strategy. This was a time when Michael Porter was everything you did about strategy and Gary Hamel’s work on Core Competencies was starting to be known. I taught this stuff for a few years but I felt that it didn’t really answer any of the questions that I had to deal with in the real world.

During my PhD I was studying change in the brewing industry and it was suggested that I looked at systems dynamics modeling as a way of understanding the industry change. I took one look at it and realised that it was just the tool that you need for strategy as it connects all the decisions you’re making with all of the outputs and how those things are dependent, change over time and deliver performance into the future.

I spent my following 5 years trying to translate a rather technical, OR view to building models to terms and language that ordinary folks can understand. The book was a way of trying to translate system dynamics to a way of dealing with everyday problems and challenges that managers face.

Because it is about how things change over time you can’t answer these problems with static tools. You’ve got to have dynamic models and you can’t understand dynamics without experiencing it. This is why we concentrated on developing simulation based learning materials. That got me to where I have been working over the last 10 years – extending and making the thing and making it more and more accessible.

The other tools are big, expensive, complicated and not the sort of thing the average person sitting at their desk can use. It seems to be working. I recently sat down with someone at one of the major banks and by the end of the afternoon she was developing business models of the companies that they we lending to.

Given the long time when systems dynamics have been taught in schools such as MIT or LBS why hasn’t this approach developed more than it has?

It’s a tough sell. Part of the problem is that there is a significant proportion of management that isn’t comfortable with numbers but that doesn’t satisfactorily answer the problem as some sectors are full of numerical managers and it hasn’t taken off there.

I suspect that it’s because it is a bit difficult. It certainly isn’t the sort of check-list thing that people like. You do actually have to do a bit of work and beyond the simple cases it gets a bit complicated. People are quite wedded to the spreadsheets that they use and they don’t see the need to do anything different even though spreadsheets have let them down.

Another reason for the slow uptake has been the process for building models that most in the field recommend you should build the model. In Senge’s book ‘The fifth discipline’ you draw a chart of how a key indicator has and will change over time. You then get lots of people to explain what they think is going on and you end up with a big, complicated feedback diagram known as a causal loop diagram which you then change into a systems dynamic model. 

What you’ve built is a purely qualitative model. As soon as you move away from that one chart you are left with a feedback diagram with no numbers on it as all that are full of opinion. As soon as someone has to build a quantitive model they’re working with something which is probably wrong and certainly where large parts have no evidence.

My approach has been to build out from the known structure. We know people move from level to level given the company’s promotion system. We can get those numbers and rates.

I put it to Kim that with historical simulation tools as quite expensive and specialized it was hard for managers to start apply tools in simple scenarios.

I suppose so, but we need to separate the thinking from the tools. What I’ve tried to do was to get people to start thinking in this manner. It’s easy to put a stock and flow on a piece of paper and work out month by month what the numbers are. As soon as you go beyond there you do get into the software. The first tool – iThink – looks very nice but as soon as you start building something you really do need to know what you’re doing. The new software [Strategy Dynamics’s Sysdea] brings that learning curve right down.

For the HR person wanting to take this approach where do they start? There are some key challenges in the area of workforce planning such as effects of demographics that the dynamics approach is probably the right one. 

Yes, I’ve seen this over and over again. A friend who works in the utilities industry did one of these models for looking at the challenges of the demographics in one major European energy firms and they were so shocked by the HR implications that they sponsored the creation of 3 university degree programmes with the promise to take 50% of the graduates.

The problem with HR is the long timescales involved. They’re struggling now because of decisions that they made 10 to 15 years ago.

Of course you can predict these things. You can put conditional probabilities on the chances people leaving or getting promoted.

Yes, one of the great things about HR is that the demographics are nice and reliable. You know every year people are getting 1 year older.

How would you see an HR person start using SD?

There are probably two parts to the answer. The first part is just within the function. Let’s assume that the business is telling you good information about the number of people they need at which levels then it’s a relatively straight forward for the HR team to build a model and show what has to happen to hiring, promotion and training rates.

You’d almost certainly want to go further and starting to think about skill levels or experience which you can do.

There are lots and lots of HR challenges which these models can be turned to. With one company I worked with the head of strategy recently they’re changing the business model. This has huge staffing challenges. They currently have x thousand people with skills a,b and c and they need to move to y thousand people with skills d,e and f. Over what rate can we transfer people from x to y, what resourcing challenges are needed and what numbers do we need to hire and how do we handle the transition of x type people that we don’t need any more.

The second issue is the relationship between the HR function and the rest of the business. So in the previous example we assumed the numbers coming from the business were good but what if they’re not quite right? You end up not making the progress you expected. To answer these questions you’d need to have an integrated model between the business and the HR team who are supporting them.

The final approach is a full, dynamic workforce planning system.

As Kim said, the best way of understanding this dynamics approach is by example. In a future article I’ll introduce a simple dynamic workforce model created using sysdea.

Quantifying the intangible

Quantifying the intangible-01.png

When pressed to nominate one book which the HR analyst should have on their shelves I tend to suggest Douglas Hubbard’s ‘How to measure anything’. In it Hubbard give a clear and systematic approach to quantifying the so-called intangibles which most HR managers wrestle with, and many avoid or dismiss as unmeasurable. 

This week I spoke to Douglas about his work, measuring HR activities and how the internet is opening new forms or real-time information that can aid decision-making. We started by talking about the career journey he took:

I entered the workforce after my MBA with Coopers & Lybrand in 1988 in Management Consulting Services and tended to get involved with projects a quantitive angle – Operational research, Management Sciences, Decision Sciences – and I recall running into a variety of things which were described as immeasurable and I knew I had just measured it on a previous project. So I wondered whether in circumstances when I didn’t have a counter-example it also could be measured.

I realized that I could identify only three reasons when somebody would say something was immeasurable and they were all illusions. People misunderstood what was meant by the word measurement – that it was being used in a different manner than in the sciences. Second, they misunderstood the object of measurement or what they were trying to measure was ambiguous. 

Finally people were unfamiliar with how measurement was done, especially the use of samples. Scientific method was never about having data, it was about getting data. So when people say ‘we don’t have the data to measure’ that they making multiple assumptions. 

They’re presuming they can’t get any more. You can always get data. Secondly, it’s not as if they sat down and did the math where they said ‘this is the much uncertainty reduction’ they will get from additional data. You usually don’t need as much data as our intuition tells us. When the math disagrees with out intuition it’s the intuition that’s wrong.

I call these three illusions ‘concept, object and method.’

People also underestimate what you can get from messy data. If you needed perfect data all the time most of science wouldn’t be possible. 

Douglas talks about the cost of getting the information, and the benefit, through uncertainty reduction that the information would bring. In this way the point is reframed from ‘we can’t measure’ to ‘is it worth measuring this aspect.’ If we say something is hard to measure we’re presuming the cost is more than the benefit.

Over time I realized that the high-information-value were things they wouldn’t be measuring, and what they were measuring had low value. I call this the ‘measurement inversion.’ People focus on what they already understand how to measure. People would give up on things that mattered most, but they didn’t know how to measure.

This is seen when viewing business cases. Managers tend to concentrate a lot on cost estimation but often fail to quantify other benefits, which are probably the most important factors. For example they may say ‘This software product will improve the quality of information’ but not quantify this in their case. As Douglas noted:

When you do this you end up putting in the one value that you’re pretty certain isn’t true – zero. What you should be doing is putting a highly uncertain range on it, compute whether it’s worth refining and see if you should measure it further.

One point that he made that I felt mapped very well to HR is that much of the time labeling something as intangible and immeasurable is used as a defense mechanism when someone isn’t comfortable with quantitative analysis. Given that this group is significant in HR is this the reason the profession is so keen on describing benefits as intangible?

This brought us onto performance management approaches, how sometimes bad measurement leads to unforeseen consequences and people trying to game the system.

We need to make a distinction between the measurement of performance and an incentive structure. There’s a lot of things you can measure that you don’t have to put in an incentive structure.

Much of this comes back to not measuring the important factors. If you don’t you’re creating inefficiencies. Furthermore, if you understand that people will be gaming the system, design the incentives so people do this in a way to encourage the behaviours you want.

If you’re incentivizing project managers to come in under budget or before time and the project managers are going to be responsible for the initial estimates then guess what will happen – they’re going to make those estimates high to make themselves look good.

Finally we discussed his latest book ‘Pulse. The New Science of Harnessing Internet Buzz to Track Threats and Opportunities

I would call it one of the most important scientific instruments of a generation. It’s right up there with CERN and Hubble.

Large numbers of people are leaving breadcrumbs and footprints on the internet. We can see what they’re tweeting, we can see what a 1999 Dodge sedan goes for on eBay, we can see what the adverts are for on Craigslist and the ranks of books on Amazon or what people search for on Google. This information is broadly useful.

A lot of these things correlate with a lot of macro trends as ‘nowcasting’ in other words tracking what is happening now rather than back-casting. 

As an example, you can correlate not only unemployment but when people start worrying about being made unemployed – when people start searching for insurances for example. This correlates fairly well with the following unemployment rate.

Never before have people had this level of information. I think this is going to revolutionize the social sciences. They have information on par with the particle physicists and astronomers.

In Pulse I talk about how early examples of getting the big picture work – massive surveys which took multiple years. 

Now we can see things in real time. A big driver has been the penetration of mobile phones currently running at 70% of the world population. There has been no other phenomena in history – I mean diseases, government systems, fads, technologies – that have penetrated such a proportion of the world’s population in such a time. 

There is sceptisism in some quarters about how relevant this information is, how relevant the sample is or the noise that you get in the system. As Hubbard mentions the key element is that correlations do occur. As we discussed earlier measurement doesn’t have to be perfect, rather it needs to be useful. Used correctly this sort of information can be valuable.

In my conversations with Jacqui Taylor, who helps some of the largest FMCG use this data in their decisions, we discussed that the data wasn’t substantial enough to do analysis for HR purposes. Speaking to Douglas I realized that it was the way you framed how to use the information that was important. Using it like an FMCG does to monitor brand reputation might not be there but using it, for example, as a proxy for consumer confidence which we know is important to employee turnover forecasting was likely to yield results.

For me, this is an area that needs monitoring. As Douglas notes:

Every time a new scientific instrument has come out a flurry of new science has followed.