Unconscious bias? Using text analytics to understand gender differences in performance reviews

Earlier this year we conducted an analysis on the comments in performance reviews for a major European industrial firm. The purpose of the work was to identify if we could identify gender bias in the language used by managers, and differences in the way that men and women talked about their own performance.

There have been a number of articles recently exploring the same problem. In an article in Fortune, Kieran Snyder reported that high performing women were described as more abrasive than men. In a more recent article in HBR Shelley Correll and Caroline Simard noted that women were more likely to be given vague feedback. They also reported that the majority of references to being ‘too aggressive’ were seen in womens’ reviews.

The fortune article was based on a small data set – 180 people of which 105 were men and 75 women. The HBR article doesn’t mention what size data set they had to deal with though it does mention they studied 200 reviews from a technology company in more detail.

In our analysis we were able to study 36,700 manager comments and 37,300 employee self-assessments. We used our text analytics ‘engine’ that powers our Workometry product to identify the semantic themes behind the statements and then used a variety of predictive techniques to identify those themes which were more likely to be used by and about women.

As well as the text we were able to link both demographic and structured performance data to the comments and therefore themes. We linked data about the manager to the employee data. The following is a snapshot of some of the important gender-related outcomes that we identified.

Replicating the results

When we did this study the Fortune article had been published but the HBR hadn’t. Though we tried hard to find instances of aggressiveness or abrasiveness we weren’t able to reproduce the results that Kieran found. I have no way of saying whether this was due to our much more extensive data size or whether the results that we saw are firm specific (it was a different sector).

The results in the HBR article do have some overlap with what we found, though we initially didn’t interpret the results in the same manner.

Needing to control for lots of variables

One of the issues with much firm-based gender diversity work is that when comparing men and women in a firm we have to control for numerous variables. Women choose different careers, different working arrangements, are present in different proportions at different ages, there are fewer women at the top levels and more at the bottom. With our models we had to ensure that we controlled for these types of factors to isolate the themes which were there because of gender differences and those present because of the jobs women do.

A good example is references to HR systems. Our client, like most firms, had much more women in HR than in other functions (as a proportion of the people in a function). More administrative roles in the business were more likely to be using the HR systems. This theme was one of the stronger predictors of whether an employee was a man / women but this was because of job-related aspects, not gender-related aspects.

Men describe what they’ve achieved, women describe how they achieved it.

This is where our work overlaps with the HBR article. However we found that women used similar terminology in their self-assessments as in the manager assessments. It should also be noted that women were somewhat more likely to be managed by a woman than a man was (because of the gender differences in functions).

Women were much more likely to be described as a team player, to be seen as helpful and supportive and to be sharing knowledge with others. They would embrace changes more than men and be interested in continuing to learn.

Their managers were more likely to describe them as having a positive attitude, demonstrating willingness and determination and as someone who could be relied on. They were more likely to take ownership, go the extra mile and were willing to take on new challenges.

Women were much more likely to describe themselves as responsive. They would highlight offering assistance to others, making themselves and others feel comfortable and working hard.

Both women and their managers highlighted some common themes, several of which had the strongest likelihood to be made by / about women. Women were seen as strong communicators and presenters, being task-orientated, working with high accuracy and following processes and procedures carefully. Interestingly being described as demonstrating analytical thinking was a more female-specific trait

It’s hard to separate the language of business and the language of men

This is where our analysis and that published in HBR has the most overlap. The terminology used by men and their managers is much more likely to be about business results. Many of the male-specific language could equally be seen as the way that businesses describe themselves.

Both men and their managers use phrases such as “Caring about profitability”, “Caring about the competition” and “Having measurable achievements”. What is striking though is that unlike women, where a large proportion of gender predicting themes we found were used by the women and their managers, most male-specific themes were used by either their managers or the men they managed – there wasn’t much on an overlap.

Managers were much more likely to describe men as “making good decisions”,“having good perspective” “meeting targets and goals”, “delivering good performance” and “creating value for the company”.

Men were much more likely to be “seeking constructive critisism” or “asking for honest feedback”. This is of course the heart of the HBR article – that women are given less specific feedback. Our work didn’t find that but did, quite strongly find that men were more likely to ask for this type of feedback.

Unlike the women who were seen, and saw themselves, as being good team players men were much more likely to say they were “interacting with colleagues”. It seemed a much more transactional approach to relationships.

The other aspect that was striking was the way that men used numbers. They talked about how the percentage they had overachieved their targets. Men were more likely to say that “they had a good year” or they met or exceeded their objectives. If you saw a comment which included numbers it was more likely to be made by or about a man. This in some ways is unsurprising. Research in the computer science literature suggest that if you want to design a game to appeal to men you should provide them opportunities to get a high score, earn recognition etc. To build a game for women let them create something.

Both managers and their employees write more about women

One of the things that we found striking was that both women and their managers wrote longer amounts of text. Managers of both genders wrote about 10% more about their women employees than they wrote about their men.

However, a man being managed by a women wrote almost 20% more in their self assessment than a man being managed by another man. As women overall wrote more is this men adapting to the way their managers communicate?

Is feedback language self-supporting?

Whilst we can identify with many of the themes in the HBR article we’re not so keen to jump to conclusions about whether managers are biased against women. In fact in our experience the ‘vague’ feedback is as likely to be used by women to describe themselves as by their managers. We also see that men are more likely to ask for specific feedback.

I wonder whether managers adapt to the terminology used by their employees. If an employee talks to themselves with numbers and ‘hard’ facts then respond with numbers and ‘hard’ facts. If an employee talks about supporting others or being a good communicator then does the manager also talk in these terms?

There is obviously huge opportunity to apply advanced analysis, including the analysis of unstructured ‘text’ data to inform a nuanced conversation about gender within organizations and to help develop effective, targeted solutions. My recommendation for firms wanting to move to a more data-centric approach here would be:

• If you haven’t done so, do this type of analysis with your own performance data. Given the differences with our study and especially Kieran Snyder’s work I suggest some findings would be industry and even firm-specific
• Develop data-based development for women to help them understand how they and their male colleagues differ, especially how men are more likely to use the language of business when describing them.
• Demonstrate using company-specific examples how managers can use different language when describing men and women. This should be addressed to women managers as much as men as many of the terms we found were as likely to be used by women managers as male mangers.