Journalists and statistics

In recent days there has been much play

over the fact that CNN, the Associated Press, NBC and other news organisations

swallowed a statistically worthless report which said that underage

drinkers drank 25% of all the alcohol consumed in the US. It turns out

that the true figure is closer to 11%, and that the 25% figure, distributed

by the National Center on Addiction and Substance Abuse at Columbia

University, was the result of not adjusting for an oversampling of teenagers

in the National Household Drug Survey. (The Center’s retraction

of its original report makes it seem just like a surly teenager being

forced to apologise for something he doesn’t think he ought to apologise

for: "In its report CASA estimates that underage drinkers consume

25 percent of the alcohol sold in the United States… If the over sample

is adjusted to reflect the population, the percentage is 11.4 percent…

Nevertheless, CASA’s estimate is that underage drinkers consume 25 percent

of the alcohol consumed in the U.S.")

I don’t blame the news outlets for buying the 25% figure. It came from

a highly reputable university, and it’s within the bounds of possibility:

teenagers account for 20% of the population, and there’s no doubt that

18-20 year-olds do a lot of drinking. If the story is well sourced,

busy journalists can’t be blamed for not checking the statistical basis

for every finding. Michael Kinsley says

that the whole story demonstrates America’s "national innumeracy";

I think it doesn’t, but that other stories do.

The real problem is not the innumeracy of the nation in general, but

rather of journalists specifically. Journalists are generally the sort

of people who never really liked mathematics, stopped studying it as

soon as their schools allowed them to, and like to rely on experts whenever

numbers find their way into their stories. It’s the old division between

the arts and the sciences: journalists are writers, they shouldn’t be

expected to understand numbers.

Fact is, the experts upon whom journalists typically rely often don’t

have a strong grasp on their own numbers. I used to write up the weekly

league tables for Bridge News: the obsessed-over rankings of which banks

had underwritten the most bond deals. Pretty much every week, after

receiving the fax from the people who drew up the numbers, I would find

some problem with them, phone them up, and eventually get a corrected

version a couple of hours later. No one else, as far as I knew, approached

the league tables with any sort of critical eye: they just ran whatever

numbers they were given.

But never mind statistical snafus: some stories in highly-respected

publications quite happily print numbers which can’t possibly be true.

Reporting that 25% of alcohol is consumed by teenagers is a mistake;

reporting that 98% of alcohol is consumed by teenagers would be an obvious

sign of complete innumeracy.

Let’s look at David Broder, dean of the Washington press corps, writing

last week:

Between 1980 and 2000, an analysis by the nonpartisan

Alliance for Better Campaigns showed, the amount spent on political

ads in major market TV outlets more than quadrupled, from less than

$200 billion to almost $800 billion, even after adjusting for inflation.

This factoid was picked up by The Week, where I read it as part of

its round-up of the best columns in the US. The news summary magazine

did appreciate how startling the figure was: "In 2000," it

says, "political candidates paid TV stations an astonishing $800

billion to reach voters."

Astonishing? Unbelievable, more like. $800 billion works out at $3,200

per US citizen, or getting on for $10,000 per voter. It’s roughly 16

times the market capitalisation of the Walt Disney Company, and is certainly

a large multiple of the value of all the TV networks in America combined.

Going back to the

source, you won’t be surprised to hear, we find that the real number

is not "almost $800 billion" but rather $771 million. The

Washington Post was out by more than three orders of magnitude.

Two mistakes were made here. Someone, somewhere, misread a "million"

as a "billion": this is understandable. But then no one stopped

to think about whether the new number made sense: not David Broder,

not the Washington Post copy editors, not the editors at The Week. These

are people who are incredibly anal when it comes to errors of language,

but they all seem to have a huge blind spot whenever a number enters

a paragraph.

I think this is one of the reasons behind the press’s failure to pull

back the curtain on Enron. Journalists don’t like numbers: they’re nearly

always happy to repeat whatever they’re told, no matter how implausible

it is. And as Broder shows us, they haven’t learned their lesson.