Using data science to shake-up employee surveys

Employee surveys typically run something like this. We ask all employees once a year a number of questions and then carefully report the results. There are always more potential questions than we can ask with ‘every man and his dog’ wanting to ask about his / her / their pet topic. After a long discussion and much political fighting we agree on about 40 questions that will be included in the survey.

Why not ask more or all the good questions? Well we know that survey respondents get fatigue resulting in poorer quality data and unhappy colleagues.

What if it didn’t need to be this way? What if we could ask 80, 120 even 200+ questions without respondent fatigue and still get good data? Well, we believe that we can.

The issue of sparse data sets is an issue for many areas of analysis. If you want to predict how someone would rate a film that they haven’t yet watched you’d calculate this from the information that you knew about them (other reviews of films they had seen) and the reviews of similar people. You might want to do this if you wanted to recommend films that you think they might like.

We can use similar techniques with survey data if we don’t have all the responses. As an example we’ve done an experiment with a small sample of employee survey data (1000 respondents).

The survey data had been coded from Strongly Agree : Strongly Disagree to a number 1:5

We took our survey results and at first removed 50% of the answers at random. There were still 1000 respondents but on average instead of each person having 40 answers then now had on average 20 with the rest set as missing values.

We then used a machine-learning algorithm to estimate what the missing values would be. Finally we compared these estimates with the real answers.

63% of our estimates were absolutely accurate. 97% of estimates were within one score of the right answer.

We then ran our experiment but removed 80% of the answers. Again we got about 63% of the estimates right but with 88% within 1 point of the real answer.

Of course this was at an individual question level. When we summarised the results across questions or creating scores for various categories of questions much of this noise was removed. 

So what are the implications for this? Let’s say you could pool all the questions that various groups wanted to ask employees and in the worst case you got 200 questions. Well even if you only show an employee 40 questions at random you can still use this data to pretty accurately calculate how they would have answered the remaining questions.

Of course we don’t have to sample the questions at random. If you really want to increase accuracy in some areas you could oversample those questions. We could develop a survey which presented questions based on earlier questions – ie it is conceivable to develop a survey which learns which questions to present in order to reduce the errors it makes when predicting the questions it doesn’t ask.

Employee surveys haven’t changed much in the last 20 years but using analytical techniques from other applications we can create help companies innovate and really start to learn about their employees.