Facts can also lie

Surveys are one of the staples of journalism. Those numbers with their reassuringly scientific percentage signs look so convincing, don't they? We just know we can rely on their cold, dispassionate objectivity. Or can we?

The short answer is yes, usually we can. It's very seldom that a survey arrives on a newsdesk with numbers that are actually false. But those numbers don't necessarily say what we think they say, and any journalist with a modicum of training and a minimum of scruple can make those numbers say whatever she, or her editor, wants them to say. As Hannen Swaffer reportedly put it, "Freedom of the press in Britain is freedom to print such of the proprietor's prejudices as the advertisers don't object to".

So, when a German research organisation tested the relative parking skills of men and women, it found that women take on average 20 seconds longer to park an unfamiliar car. The Daily Mail gleefully reported that the old 'prejudice' was actually true: women really are worse than men at parking. Someone else spun exactly the same bit of data to say that women are more careful than men at parking (a conclusion that was consistent with other findings in the same survey, which were unsurprisingly given less prominence by the Mail). Same numbers, different conclusions.

Scepticism is essential. That's easy when the survey says something you don't believe, but it's more important to be sceptical about studies – or anything in fact – that seem to confirm one's own beliefs.

Equal pay is a good example, because it's been widely studied, the statistical results are consistent and almost everyone has experience of either working with women or of being one. One of the most common responses to such studies is that the workplace remains far too sexist and that the problem will only be solved once men and women receive equal pay.

I will not argue that the workplace is or isn't sexist, because I'm not qualified to do that and because I'm trying to make a point about understanding surveys, not about equality in society.

The problem with these surveys is that they reduce a huge and complex thing – the working conditions of every employed person in the country – to a few headline numbers. Writers and analysts need to look beyond those numbers to work out what they actually say, rather than what they appear to say. That means asking hard questions and being prepared to find answers that you don't like.

Question 1: Are we comparing like with like?

Female directors definitely get paid less than men. But I've seen directors' posts advertised for under £40,000, while other directors are paid ten times that figure. When looking at a survey, ask how many directors of comparable status in comparable industries are included. Are there enough of them to make a sensible comparison?

To use a personal example, my job title is "Editorial Director". Today, I saw a managing director's post (two echelons above mine) advertised at half my salary. If they appoint a woman, it will emphasise the disparity between senior women's pay and men's. If they appoint a man, it will add to the disparity in numbers at senior level. It's easy to see that gender can't be an issue because the job hasn't even been filled, but whoever gets the job will skew the gender statistics one way or the other.

Feminists rightly argue that women should be paid the same as men for the same work. However, away from the production line nearly all jobs are different, making comparisons impossible, even within the same organisation. I was once editor of a magazine when I had just two journalists on staff. My successor had six. Same company, same job, same pay, very different work. Unfortunately, sexist managers sometimes use that argument to wriggle out of their legal responsibilities. 

Question 2: What is the real reason for the disparity?

Playing Devil's Advocate, I once posted a comment on a newspaper story about pay disparity. It went like this:

The Equal Pay Act gave women the right to have the same deal as men, but maybe they looked at the rat-race with its long hours, its devotion to the job above all else and its destruction of the human spirit and said "no thanks". Meanwhile us men slave away with our eyes glued to a computer screen and our noses glued to the boss's backside for 20 years till our wives see what empty, soulless shadows we have become and walk away; taking kids, house, money and all, leaving the man with his salary and his drink problem. You need a big salary to accept that deal. Women are smarter than men. Consciously or unconsciously, they say "no".

My assessment was deliberately exaggerated and might be completely wrong, but my point was that we should consider all possible explanations before assuming that the "obvious" one is correct. It is also just as likely that women really are being held back by powerful men who don't give them the same opportunities as similarly placed men, or that sexist culture subconsciously persuades women to settle for less. 

I'm not arguing for any of these interpretations, and possibly all three factors come into play depending on individual circumstances, along with a few others I haven't thought of. My point is that the headline numbers don't prove any of these theses, because they show effect, not cause.

Question 3: Who says so?

If a survey is commissioned by, say, Fathers For Justice, we can be pretty sure that they won't release it unless it supports their 'women have it all and men are the victims' manifesto. No pressure group will publicise findings that show their job is done or they're wasting their time and should pack up and go home. With that in mind, you could also ask them a question posed in Darrell Huff's How to Lie with Statistics : "How many juries did you poll before you found this one?"

Question 4: What questions did they ask to get those answers?

Pollsters are adept at asking questions that get the answers their paymasters want, and respondents have a bad habit of giving answers that they want the pollsters to believe. For instance, get a team of women to ask married men in face-to-face interviews whether they use the internet for pornography, and I don't think you'll get very reliable results. Similarly, it would be easy to get a big majority of people to say 'yes' to the question, "Do you oppose the EU meddling in Britain's affairs?", but that's because everyone opposes "meddling" of any sort. It's a very pejorative word. The resulting headline would be something like "Brits say 'hands off!' to Brussels", even though the respondents were only expressing an opinion on a subjective and hypothetical situation.

Once again, I'll stress that I have only chosen gender equality because it's a widely studied phenomenon of which we all have experience and where the headline numbers are consistent and therefore reliable, making it the simplest example I could think of. It should not be taken to mean that I don't believe in equality (I do) or that I think equality has been achieved. The hardest thing to do is to question evidence that seems to support one's own beliefs, but it is vital that we do so (whether or not we are journalists). From an activist's standpoint, it is crucial that the real nature of the problem be understood, because you can't fight the enemy if you don't know where he is. To use a medical analogy, a pain in the leg might stem from a problem in the lower back, so targeting the leg would be pointless and even dangerous.

Moral: Always look deeper and draw your own conclusions, not the conclusions someone else wants you to draw.