The pay gap that might not be there

Need to prove an argument? Chuck in some statistics. Any old ones will do.

Sometimes it's hard to tell if a journalist is ignorant of statistics or is being deliberately misleading. Like most of us, they seldom question statistics that seem to prove what they already think they know. 

The Guardian is no better or worse than most, and it's been doing a lot of it lately. Some of its statistics are misleading, but mostly they simply don't say what they seem to say. Whether the writer knows this and is trying to pull a fast one or just doesn't understand what the numbers mean is hard to say. 

Take this example, in a recent article about how the press should report suicide in the wake of Robin Williams' death:

"Many years ago an episode of Casualty showed someone taking an overdose of a particular drug. The following week there was a 17% increase in this kind of suicide attempt." Guardian, 13 August 2014

That's a pretty damning percentage, isn't it? But it doesn't tell you anything unless you know the hard numbers behind it (which are hard to pin down, but you can try if you like). After all, 17% of 10,000 is 1,700 people. But 17% of six is only one person. And if the numbers are small, 17% could be within the normal range of variation.

There's another little fact that should make the careful reader sit up and take notice: that episode was watched by 22% of the UK population. When was the last time a prime-time TV show was watched by so many people? Given the proliferation of TV channels and the rise of on-demand viewing, you'd probably guess that this show was broadcast over 15 years ago. And you'd be right. Why has the writer chosen such an old piece of research? Who knows, but it's reasonable to infer that there hasn't been a recent case that shows the sort of correlation she wants you to see. 

Now look at this little gem:

"Including men and women managers of all ages, the CMI [Chartered Management Institute] said that the pay gap stands at £9,069, with men getting an average salary of £39,461 where women get £30,392." Guardian, 19 August 2014

Your antennae should start twitching at the phrase "of all ages". Such a wide sample of ages makes the analysis, and the headline 'pay gap', meaningless. Taken on its own, it could equally be evidence that the pay gap is an old problem that has now been solved. 

How so? Well, we can be sure that the top bosses with the biggest salaries, and who are overwhelmingly males in their 50s, get the most. Junior managers get paid much less. If up-and-coming junior executives in their 30s are overwhelmingly female, then the average female salary will be much lower than the average male salary, which is exactly what we see here. The more young women get promoted into management, the wider the pay gap will be (as measured by this statistic). So top management has a male past and present, but a female future.

I'm not saying this is actually the case, but the statistic the article relies on 'proves' this argument just as well as it 'proves' the writer's.

But there are more numbers in this article that suggest my devil's-advocate analysis might actually be right:

"The CMI said, "…the gap is far worse for women aged 40-plus, where the problem is twofold. Not only does the salary gap increase with age and seniority, but there is also a persistent 'bonus pay gap'." 

If things are significantly worse for older female executives, there could be several reasons for this.

1) Discrimination gets worse the higher up you get (what the writer wants you to believe)
2) Discrimination was much worse 20-30 years ago, so women of talent were held back in their careers and haven't caught up yet
3) Discrimination is much less of a problem now, because younger women are rapidly closing the pay gap

All three arguments can equally plausibly be made from the data on offer, but the writer has chosen the one that suits his prejudices. The impression that the writer is carefully selecting his statistics is strengthened by the phrase "female managers aged over 40 … earn 35% less than men", followed by "The average pay gap between men and women aged between 46 and 60 stands at £16,680 a year." Why the sudden change from "over 40" to "between 46 and 60"? Forty-six seems a strange number to choose. Why not 45? Or 43, or 48, or 40 (since he's just been writing about the over-40s)?

I'm not arguing that sexism doesn't still exist in business; my point is that these numbers don't prove it, however much the writer wishes that they do. And I find myself questioning the existence of a problem that very probably does exist.

Moral: Sloppy statistics fool some people, but they undermine your argument.