In 1995, the owner of the Great Smoky Mountain Railroad sued Warner Brothers. The movie studio had used the railroad as a location for the train crash scene in The Fugitive, and the railroad had asked in return for an end-credit. For some reason the end-credit never appeared in the final film; the owner was convinced that if it had done, visits to his attraction would have been much higher. And to try to prove his point, he commissioned an academic paper from a group of three expert witnesses.
The expert witnesses did what they were paid to do. They took a list of very successful films set in easily-identifiable locations, such as Devils Tower National Monument (Close Encounters) or the baseball field from Field of Dreams. They then looked at the visitor numbers at those locations, and concluded that
When the median increases of the aggregated data were graphed, the span of movie inducement lasted for at least four years and increased between 40 and 50%. Data not presented in this paper would suggest that the induced effect could be longer at some sites and not exist in other cases where locations were not identifiable or accessible…
Additional research is important to document this largely unexplored phenomenon… As yet, no studies have documented the economic impact of one movie on a location or community.
The paper’s not exactly iron-clad. It never gives us any actual datasets, instead just reproducing charts; and the data it does give imply that the margins of error are pretty enormous. To take one example: the 12 locations were split: six were public parks, and six were private attractions. Three years after a film came out, the median public park saw its attendance rise by 33%; the median rise for all 12 public parks and private attractions combined was 34%. And yet the median rise for the private attractions was 67%.
In other words, numbers are certainly possible, but they range around so much that I wouldn’t want to extrapolate them at all.
In any case, the fact that the research was commissioned by someone who knew very clearly what result he wanted should make anybody consider the paper essentially worthless. Ex post, it’s easy to find film locations which benefitted from a certain movie coming out. But that just means there was acres of room for the researchers to cherry-pick the best results. And they never give absolute visitor numbers, just percentage increases, which will skew things even further when even they admit that the field in Field of Dreams had "zero visitors before the release of the movie". (Yes, this really is a research paper where the authors divide by zero and come up with a supposedly useful result.)
But that’s not the only reason to treat any results from this paper with extreme prejudice. There’s also the fact that the paper’s lead author, Roger Riley, said in an email to me that if the results ever were true, they’re probably not any more:
This data was collected in the early to mid 1990’s. Things have change a lot today. Movie companies release to DVD and television much sooner than they once did, thereby reducing the amount of time that a movie is in the public eye. I would be surprised if that could be proven in today’s world.
You know where I’m going with this one, don’t you. This paper, which even its lead author doesn’t consider remotely useful, has become a standard reference point in movie-industry journalism. For instance:
According to the Annals of Tourism Research, locations featured in a successful film can expect to see visitors increase by an average of 54 percent over the four years following the film’s release.
I don’t quite know how the 54% figure became the standard one cited (I would have expected people to say that visitors increased 77% over five years instead), but somehow it did. Thus does Dan Glickman, CEO of the Motion Picture Association of America, say things like this:
Take The Dark Knight for example. It was filmed in Chicago last summer. Over the course of the 65-day shoot, Warner Bros. created more than 1,000 local jobs which generated $13.5 million in wages. They also spent more than $3 million on hotels, $1 million on catering, and $900,000 on lumber and other set materials. In total, The Dark Knight injected more than $35 million into the Chicago economy in two months. Another bonus? On-location filming is terrific marketing. It’s estimated to boost tourism by up to 54 percent.
Never mind the fact that it’s really hard to see how you can get to $35 million once all the big line-items add up only to $18.4 million. The key thing to note here is how a virtually value-free statistic becomes an Impressive Number with which to blind credulous journalists, readers, and interlocutors. No one ever bothers to look the number up and ask if it’s true (present company excepted).
There are a couple of lessons to learn here. The first is that when a statistic fails the smell test, there’s a good chance it isn’t true. But there’s a broader implication: that there’s a good chance that any given statistic isn’t true, whether it fails the smell test or not. And you certainly can’t trust the Fourth Estate to do your fact-checking for you.
