A few years ago the talk in HR was about Big Data. Now, the talk is about Predictive Analytics. In both instances people had grabbed a popular and exciting topic from outside HR and claimed that this was the new big thing. In both instances the labelling probably gets in the way of understanding the real value that the core techniques provide. In both instances what’s hidden behind the labels is where the value is.
In a recent podcast interview with Matt Alder I mentioned that size was never really the issue with Big Data in HR – the issue was always that the variety and quality of the data meant that HR analysts were dealing with data that wasn’t easy to analyse. When you hear quotes on the size of data increasing exponentially then it’s worth realising that a large amount of this is unstructured data – text, images, video, sound. The Big Data challenge for HR has always been how to deal with this unstructured and messy information. Size per se is less important.
With prediction, the prediction per se is usually less important in HR. Unlike marketing where a simple rule can be created – ‘send offer or don’t send offer’ – and to some degree it’s less important how that recommendation was made (and far more important the outcome – i.e. optimisation of revenue or profit) with HR the primary objective of most analyses that we do and see is not the prediction but being able to interpret the model. Understanding what is going on is much more important than just a prediction.
To understand what is going on let’s take a step back and understand what analytics practitioners are talking about. With a predictive model we’re using a series of techniques to do one of two fundamental things: we’re either providing a predicted outcome – usually binary (will leave, will stay) or we’re providing a likelihood of an event happening (a probability, also described as a risk).
To make a decision we need to optimise not on this prediction but on an estimate of the resulting outcome by combining it with a so-called ‘loss function’. A loss function describes the costs and benefits of the decision being right or wrong. I’ll probably cover this in another post as far too many HR applications of analytics seem to ignore the loss function entirely – they stop with the analysis and prediction rather than applying this to help make decisions.
With HR modelling we’ve got a big, difficult issue. For us not only are our models pretty inaccurate and therefore there is reasonable uncertainty in the recommendations even of the best models, but the loss functions are often highly asymmetric and themselves contain uncertainty (i.e. we can’t even be certain of the real value / loss if we make the right or wrong decisions and getting it wrong often can provide a loss multiples the size of the potential benefit of getting it right.)
How good are our models? Let’s take a simple but common application which is modelling employee attrition. Talk to an analyst today and they’ll tell you that the best models at predicting attrition are achieving about 85% accuracy. Is this good enough? The odds of you ‘winning’ at Russian Roulette is 83.3%, so about the same. If the outcome of getting it wrong is small then that accuracy might be good enough. In other cases (like Russian Roulette) it’s probably not.
The natural outcome of a complex loss function and uncertainty in the model is that data should be another voice at the table. We think the number of use-cases in HR where a decision, at least a decision of reasonable importance, can be automated are small.
For humans to make decisions they need to understand not only what the recommendation is, but how that recommendation was made. They need to know what could be driving the prediction or recommendation.
If we start our analysis at the most logical step – the set of actions / responses that you can make – for HR most of those actions are policy-type changes. Those policies may be highly targeted and the parameters might differ by individual but we’re typically constrained to a set of responses.
To make these policy changes what becomes the most important is arguably not the model accuracy but the interpretability of the model.
With interpretability our models become far more useful at driving effective policy changes because they describe what needs to change. The most important way of realising value is by taking the right, and highly targeted, action.
Secondly, we typically don’t have a single loss function because our populations aren’t homogenous. The value / cost of various segments of the population differ. Typically, to optimise the decisions we need to optimise build segment-specific models. This also enables us to make segment-specific actions.
These segment-specific models might not be the most accurate at predicting, but they are much more useful because they describe what we care about and facilitate better policy changes. We have a bunch of constraints that marketers don’t typically have. We need our policy changes to seem reasonable and often need to show fairness by treating various segments in the same way. We need to explain to our employees why we’ve made certain changes and how we’ve identified who should be targeted.
So if the key thing for a model is interpretability, not prediction, why do we place such an emphasis on prediction?
One of the techniques that we’ve found is really powerful is building multiple models, typically using various techniques. Some of these techniques are easier to interpret than others. Others may historically yield better accuracy but can be ‘black boxes’.
Say the policy recommendation of your interpretable model is to raise the salary of a specific segment of your population by x%. If you simulate (predict) what would happen by plugging this back into your model you’re likely to see the best-case result. Your change and model are self-supporting.
What we’ve found really useful is to simulate / predict that policy change on a model built using a different technique. If the resulting outcome is good it will suggest that you’ve found something reliable. If the two outcomes are widely different it is likely to mean you’ve got a policy which is less likely to perform in the real world. Interpretability was important for identifying the change but less important for validating it.
If you really want to add value to your organisations using advanced modelling techniques then it’s not prediction you should focus on. It’s interpretability and incorporating sensible loss functions.