Error, Randomness, and Payrolls Statistics

Just how error-prone are the monthly payrolls statistics? I’ve been reading Len Mlodinow’s very good new book, The Drunkard’s Walk: How Randomness Rules Our Lives, and one of his statements really stood out for me: if you have a data series and the moves within that data series are within the margin of error, he says, those moves are "essentially meaningless".

I immediately thought of the payrolls data which come out on the first Friday of every month and which can cause huge market moves. Here’s the montly payrolls data from the beginning of 2007:

The blue line is total non-farm payrolls; the red bars are the monthly change in payrolls, which is the number the market most concentrates on. I’ve put the monthly change on the left-hand axis, and I’ve set the scale to 104,000, which is the level of statistical significance.

As you can see, 12 of the 17 datapoints, by Mlodinow’s standard, are within the margin of error and therefore essentially meaningless. But here’s the thing: the series simply doesn’t look like one where anything under 104,000 is meaningless. In fact, the series looks convincingly as though payrolls rose steadily throughout 2007, and then have been declining steadily throughout 2008. And the market, certainly, doesn’t behave as though anything less than 104,000 is statistically insignificant.

So I asked Mlodinow: Is it normal or likely for each

month’s change to be so close to the previous month’s change, if the

margin of error were really so large? Who are we to believe, the BLS’s statisticians, or the evidence of our own eyes?

Mlodinow replied:

When you look at trends rather than the difference between two

points, things become more complicated. For example, though a movement of

less than 100,000 may not be statistically significant, the way random

error behaves you would expect the fluctuations to be both up and down,

and so a more or less steady rise of, say, 50 – 75,000 upward, over many

periods, could well be an indication that something really is happening.

The case you quote is far less clear cut, but the string of upward numbers

could well indicate a trend.

Here’s what I take Mlodinow’s reply to mean: yes, there was an upwards trend in 2007, and there’s probably a downwards trend in 2008 as well. But that’s something which can be discerned only by looking at the series as a whole. Given the margin of error, it’s silly to read much if anything into any one month’s data.

But I’m still not convinced there isn’t something very weird going on here. The series just doesn’t seem to behave like one where the margin of error is 104,000. Is there some kind of massaging going on at the BLS before the data is released? Is the margin of error being overestimated? Or are we just falling into the natural human habit of seeing patterns when in fact there are none? Maybe this is just a case of the latter, and when the Dow swings wildly on a positive or negative payrolls report, that’s just another case of the market being predictably irrational.

In any case, I can recommend Mlodinow’s book. I’ve always had a healthy interest in, and solid grasp of, the principles of probability, possibly because my father taught me to play backgammon at a very young age. As a result, a lot of the paradoxes and counterintuitive results that fill up the first half of the book came as little surprise to me – and they probably won’t come as any surprise to anybody who’s read Taleb’s Fooled by Randomness, either. Still, Mlodinow has read widely, his examples are extremely well-chosen, and one in particular – his own "girl name Florida" problem – was counterintuitive even for me, and took me quite a while to really understand. If you’re interested, Carl Bialik sums it up here.

And in its second half, when Mlodinow’s book moves on from probability to statistics, there’s loads of interesting stuff – including the discussion of statistical significance, and just how easy or difficult it is to reverse-engineer margins of error from datasets.

One of the tragedies of contemporary macroeconomics is that the business of actually generating the statistics on which economists and the market rely is not well paid and comes with almost nothing in the way of prestige: "I’m a statistician" is not a great way to kick off any dinner-party conversation. Mlodinow’s book is not going to change that, but at least it’s a step in the right direction.