Most of you are interested in predictions of Tuesday’s outcome based on state polls. This is an area that started in 2000, acquired cult status in 2004 (WSJ link and alternate), and went fully mainstream in 2008. But before that, political scientists took their stab at the prediction problem. How do all the methods compare?
Here’s the upshot: In the spring and summer, political scientists generate the best predictions. Starting around Labor Day, state poll aggregation becomes the way to go. Online hobbyists like Andrew Tanenbaum (electoral-vote.com), Nate Silver (FiveThirtyEight), and I all use the same data, so our estimates tend to be quite similar. Small differences arise because we use different definitions of which polls are fresh, as well as some other assumptions. On Wednesday morning, after the votes are in, we’ll find out which of us is closest.
And now, a brief history of prediction.
Models by political scientists. For years, political scientists have been interested in using economic variables and incumbent-president approval ratings to predict Presidential election winners (for one review see this paper, especially page 424). They focus on the popular vote, which is a good but not perfect predictor of the winner, a lesson learned in 2000. That year they overestimated Gore’s popular win percentage. Some models incorporate national polling data into their models, which improves performance after Labor Day (no surprise there).
Such models do well during spring and summer months of the election year. This one, by Wlezien and Erikson, has a hit rate of about 75%:
That’s a time when state polls alone aren’t predictive, as my analysis from 2004 showed:
In 2004, both state polls and political scientists succeeded in their own way. The Meta-Analysis indicated an Electoral College outcome of Bush 286 EV, Kerry 252 EV, the correct result. On the political scientist side, see the last paragraph on page 750 of Wlezien and Erikson’s paper, which predicted a narrow Bush win. The two approaches converged nicely. This year, political scientists’ models predict a large Democratic victory (for a summary of many of them, read PollyVote).
My synthesis of this is that we can know what the natural “set point” of a Presidential campaign from political scientists’ models. The current projection of Obama 364 EV, McCain 174 EV, may not be far from that point.
In this context, what polls show us is movement toward and away from the set point. These movements can be driven by campaign events as well as other events. Readers of the Princeton Election Consortium know the big events: the “Celebrity” ad, McCain’s houses gaffe, the Palin VP selection (and subsequent flameout), and the first debate.
Poll meta-analysis. Now let’s turn to state poll meta-analysis. Many of you are familiar with my nitpicking over Nate Silver’s methods. But at core, we are doing the same thing: (1) estimating state by state win probabilities, then (2) combining all the probabilities to generate a distribution of possible outcomes. To my knowledge the main pioneers in this activity were me and Andrea Moro, as well as a few others.
One question comes up: can we improve on state polls alone? If only state polls are used to generate win probabilities, then my approach is more or less optimal. Beyond that, I would define improvements as additional modeling steps that (a) increase the probability of getting the winner right, or (b) increase the accuracy of the EV total or vote share.
Nate Silver is attempting to make such improvements. We don’t know yet whether his many assumptions (which keep changing – here’s today’s installment) constitute any improvement over “naked” polls. He claimed success in primaries, but he hasn’t been tested in a general election. This Tuesday’s election will serve as that test. For that matter, he could try his hand at the 2004 data – the acid test there would be whether he could get Wisconsin right.
Although I’ve made much of the differences between our methods, what’s interesting is that his added assumptions lead to few significant differences. For example, consider our state-by-state Obama win probabilities, which are quite similar.
For all intents and purposes, these probabilities are identical (r=0.989). Discrepancies may arise from the fact that he accepts polls over a much larger time window, or from a national poll-based correction he applies. The fancier stuff probably doesn’t matter much. Here are the biggest differences:
|State||Electoral votes||PEC win %||538 win %|
Of these differences, Florida has the largest effect on the expected outcome.
Anyway, I think all that extra math is like watching the chef at Benihana. The knives fly around to impress and scare the customers, but the product tastes the same as other restaurants, and it’s okay as long as no damage is done.
P.S. Since electronic markets derive their predictive value from polling data, I’ll cover them separately. But not tonight.
P.P.S. Is Benihana still around?