Big data and HR: Pt I. What is Big data and why should you care.

Big Data is undoubtedly one of the buzzwords of 2012 but what does it mean for HR and how should HR be responding?  This is going to be the topic of my next 2 posts, this one discussing Big Data from an HR perspective and the next talking about the future and how HR should respond.


To do this I sat down with Jacqui Taylor, CEO of OrganizationView’s technology and analysis partners to get the perspective of one of the pioneers in this area.  For the last 3 years FlyingBinary have been working with some of the pioneers in this area across verticals doing the sort of project others are now talking about.


We started by talking about what is Big Data and took the Gartner Volume, Velocity and Variety model.  I put it to Jacqui that in HR the key part is about the Variety part.


“Well no, it’s also about the value, where can the value be realised?  It is about the variety, especially when extending the view of data sets can give better insight.


The corporate view is ‘I have the data I need’. Actually no because outside your walls is data that will add value and insight.

And so that variety piece is about the different data sets you have but equally the value of what you can do is so closely linked to the bottom line of any company. The approach you can use from a variety and value perspective are different.  You will tend to approach this from a variety point of view, but as you mature you start to look from a value perspective. Value is derived from the insight you bring and the action you take.

But I want to take it back a bit to what is Big Data?  Big Data is just something that is bigger than you can handle.  It doesn’t have to be big.

Velocity is deeply interlinked.  The speed it’s coming at you is often too quick and this is also a common issue organisations face.

The part that everyone people forgets is viscosity: are your pipes big enough to do this? Is your plumbing able to handle this?

Legacy systems and old architectures were all about what you could control and what you can consume.  In the cloud world, and in our new web world that doesn’t work.  You need a new approach.  You can’t manage the peaks and troughs we now have deal with in corporate systems. Many high-profile outages have been viscosity issues.

If you develop the philosophy of a data ecosystem you’re dealing with big data. I don’t mean you need the worlds biggest Enterprise Data Warehouse – that’s not what it is about now.  Understanding the different data sources you have, and they’re likely to be fragmented.   The question is “How can you mine the value with the data in place?”

So is getting the data the issue?

Getting access to the data isn’t a problem.  The social platforms, the news platforms are all easy to access and harvest data.

When drilling for insight you should have more.  Now we have an alternative to sampling. So for example in our data centers with have 30m logs a day, we analyze all of them and the 80/20 rule applies.  The approach is key, dealing with all your data rather than sampling allows real insight to be driven from your data. The exploration of your data is limited by the time you can reasonably spend on it.  What you do with it, the publication part, is the easy bit.  The exploration part is what’s driving the value.

The exploration piece needs the full data set to get the initial insight but when you know what you’re looking for you don’t necessarily need the full data.

So one of the key messages with Big Data is that we don’t need to sample. So if we’re a supermarket we don’t need to sample, we can look at all items in the basket for each customer and link it to customer data I’ve got from their store card.

Yes, that’s what they’re doing, and they’re taking direct feeds from the weather centers and other data sources to build their understanding in real time.

Employee Data not HR Data

I can understand this in customer data, but the number of employees are limited and their transactions are irregular. How do we use Big Data lessons in HR?

You need to have a look beyond the traditional HR questions.  It requires you  to look at the broader data ecosystem and identify what additional information can add value.

So we’re saying stop limiting yourself to HR data and start thinking about all corporate systems which tell you which have an employee link.  Of course most enterprise systems log files do this for audit purposes.

Yes, Look at operational data, might it tell you the performance management piece that you might not know?  At the moment the two key issues are managing talent and being more efficient.  So you need to look outside your HR system and ask ‘what other data do I have that can help me understand and drive answers to these issues?’

Also don’t ignore what is going on outside the company, because your customers are telling you how your employees are doing.

So I know in supermarkets where you physically sit cashiers makes a big difference on productivity.  So we’re saying your till data might hold the key to driving employee productivity?

Yes, but in most companies it’s broader. If we mine your email we can tell you who is sending the most emails, who they’re connected with, when they’re doing this.  Why does that matter?

Because effective organizations don’t run on email. So we can use analysis to tell you who’s clogging up your organization.  Rather than getting up and walking across the room or having a conversation at the water cooler they’re sending an email.  We can tell you that.  That is ‘where in your culture can you change what you do to get a more effective organization’.

I want to make sure that this isn’t seen as Big Brother-ish.  Yes it is for the organizations’ benefit but it can also be for the employees’ benefit because you can collect, analyze and understand the culture and understand what’s getting in the way of people doing what they want.  Using this insight to make changes to make it better for employees, that’s the future.

My conversations with fellow HR professionals suggests that many are scared on this, that they want to stick with what they know.  If we return to our email analysis topic, we’ve spoken before that it might not be the content of the emails that is important but understanding the networks that emails show.  If we spot that one group always cc’s their boss on emails this probably highlights a trust issue which we might want to act on.  So what we’re saying is that it’s not analyzing the data that’s the issue, it’s what you do with it.

At an aggregate level the data allows you to continually transform your organization and preempt what you need to do.  What you use the data for is the key. If you’re working at an aggregate level you tend to be OK.  If you are identifying and targeting individuals you tend to be on very sticky ground. If you’re looking across a division and spotting patterns and you act at the division level you tend to be OK.

The other thing we need to talk about whilst we’re looking to the future is very important. Before very long there’ll be a movement for organizations to give up this data, because actually they don’t own this data. If companies want to use it, and especially if they want to monetize it then individuals want to be part of this.  Individuals will start demanding it back as a citizen – in the UK we’re already treading this argument. The movement is very much ‘I own me’.  In 4 years time Gen Y will be the biggest part of the workforce, this is very much how they’re thinking.  This is a key change.

So this links to driving email analysis?

Well no, with email it’s because we’re at the end of time when companies do have control of email. Email analysis is important now as we currently have access to how this is working. The culture of email is a Gen X culture. When they’re not in charge the culture of email goes.

With the bring your own device culture you lose the ability to understand and control what is going on.  If you want to act, you need to do so now.

In the future we’re looking at social tools and office collaboration tools. The office collaboration tools are easy to mine, but more difficult to mine than email.  This transition is throwing up a whole new set of problems.

Already HR is behind the game.  Some more exploratory analysis on this is really needed.  Companies that get this right will be the ones that get the talent and keep the talent.”