Princeton Election Consortium

A first draft of electoral history. Since 2004

How’d we do?

November 11th, 2008, 6:30pm by Sam Wang


For a scientist, a moment of truth for a hypothesis is the experiment. In this case the “experiment” is the election. Here’s a quick run-through comparing our predictions with outcomes so far. Performance is extremely good. As was the case in 2004 and 2006, the results are consistent with the idea that relatively pure polling data can predict outcomes accurately. That’s n=3 elections.

The Electoral College
Outcome: Obama 365 EV, McCain 173. The map (NE 2 not shown):

FiveThirtyEight: 348.5 EV. Error: 18.5 EV.
Electoral-vote.com: 353 EV. Error: 12 EV.
Our last-day Median EV Estimator for Obama: 352 EV. Error: 13 EV.
Our prediction: Obama 364 EV, McCain 174. Error:1 EV.
Variance-minimized polling snapshot (post hoc): Obama 364 EV, McCain 174. Error: 1 EV.
Closest: Princeton Election Consortium.

Individual state wins
FiveThirtyEight: 50 out of 51 correct, Indiana missed.
Electoral-vote.com averages: 49 correct, 1 incorrect (Missouri), 1 tie (Indiana).
Our prediction: 50 correct, Indiana missed.
Closest: Tie between the Princeton Election Consortium and FiveThirtyEight.

State margin estimates
Correlation of PEC state poll margins with actual margins, +0.975. FiveThirtyEight absolute error: 3.9 +/- 4.9 %, median 2.5%.
PEC absolute error: 4.1 +/- 3.8 % (mean+/-SD), median 3%.
PEC’s margin was closer in 21 out of 51 races (95% CI, 19 to 32 races).
Closest: Tie between the Princeton Election Consortium and FiveThirtyEight (FiveThirtyEight ahead by a statistically nonsignificant margin).

Net Bradley effect plus cell-phone bias
Outcome: Obama +0.7 +/- 0.8% from Meta-Analysis, 0% from popular vote.
Our prediction: maximum of Obama +1%.
FiveThirtyEight: Obama +2.8%.
The closer estimate: Princeton Election Consortium.

More after the jump.

Popular Vote Share
Outcome: Obama 52.7%, McCain 45.9%, third-party 1.3%.
Our prediction: Obama 53.0%, McCain 46.0%, third-party 1.0%.
FiveThirtyEight: Obama 52.3%.
Other comparisons are here, along with lively discussion.
The closer prediction: Princeton Election Consortium.

Total Turnout
Outcome: an estimated 132.6 million.
Prediction: 135 million.
(I wasn’t expecting to do well on this one – it was squeezed out of me for a contest.)

House of Representatives
Outcome: 257 Democratic seats.
Prediction: 257 Democratic seats, 68% confidence interval [254,260].

Senate
Outcome: 59 Democratic/independent seats, 41 Republicans.
Prediction: 58 Democratic/independent seats.

The Senate has unresolved issues:

In Alaska, convicted felon Ted Stevens (R-i) appears to be ahead of the last pre-election polls, which showed clear Begich leads but were also volatile (a 22% lead, then only an 8% lead). About 20 times as many early voting ballots as the current margin are yet to be counted. So Begich is likely to pull this one out. Even if Stevens wins, he still faces probable expulsion from the Senate. This would be followed by another election to replace him. Paging Senator Palin? Update: Begich won.

In Georgia, Chambliss (R-i) did not clear 50% and faces a Dec. 2 runoff against Martin (D). The odds favor Chambliss. He received 49.9% the first time and now does not have to run with Obama at the top of the ticket. Update: Chambliss won.

In Minnesota, the final count shows Coleman (R-i) ahead of Franken (D) by 0.02% 0.01% (fewer than 500 400 300 206 votes) . State law mandates a recount. Unsurprisingly, Coleman doesn’t want one. Update, 1/5/2009: looks like Franken by 225 votes, a margin of 0.01%.

And of course we don’t know who will take the place of Obama and Biden, though they will be Democrats.

I estimate that the Democrats have an 85% chance of getting to 58, and a 40% a 45% chance of getting to 59. But it will be some time before we find out. Update: 59 it is.

Finally…A note on methods. The Electoral College and state win predictions were done using state polls only. The cell-phone adjustment was done using Pew Center data. The Senate and House predictions were done using Congressional polls only. The popular vote share prediction was done using national polls only. Turnout was estimated using the work of Curtis Gans and data from InTrade and the Census Bureau. In no case was any demographic or pollster-specific information used. Overall, the results show that a high degree of accuracy is possible without complex model-building.

Tags: 2008 Election

34 Comments so far ↓

  • gprimos1

    Congratulations Dr Wang!

    As statistics student I really appreciate how your site provided such a practical use of of statistics during this exciting election. A few final comments from me:

    - While it is exciting that you were so spot on for the EV prediction, does that imply that your final CI [316,378] was too wide? Do you think you could have tightened further and still likely have been right most of the time? What would you guesstimate your percent chance of hitting that 364 EV number was?

    -For individual states: You had Indiana, North Carolina and Missouri as toss-ups, and the results we saw would be not be unusual if you were flipping coins. So as far as I am concerned, you had 51 out of 51 correct. The ones that were tossups really were tossups!

    -Correlation of state poll margins with actual margins:
    This is exactly what I was looking for earlier. Can you post a link with data you used? I would like to add some variables for each state and see if there is anything else interesting to find.

    Thanks again!

  • bks

    Overall, the results show that a high degree of accuracy is possible without complex model-building.

    I wonder if that an be extended to other biological phenomena, e.g. econometrics?

    –bks

  • Five Acres with a View » I Love This Map

    [...] Full Size 500×376Post-election evaluation from the Priceton Election Consortium [...]

  • Christian

    Dr Wang,

    Congratulations on your good work. I’ve been following your efforts on this field since 2004 when Dr Moro and you pioneered this type of approach.

    Remembering back then I remember your model predicted the election outcome but your personal twist regarding undecideds caused an error. Reasonably, you explain this as the reason for the deviation from the final outcome. This time, however, the model was slightly off (352 for Obama) but you claim success owing to your subjective adjustment at the end. You cannot have it both ways, surely.

    Other than that, brilliant work and look forward to see more of this on forthcoming election cycles.

  • Sam Wang

    gprimos – The CI overstates uncertainty because of the spikiness of the histogram. Look at the EV history. Of the last 32 reported Median EV Estimators, 20 are one of three values: 353 (8 times), 364 (5 times), and 367 EV (6 times). It helped that MO and IN have the same number of EV (11).

    I was aware of these facts when I made the prediction. Since 364 EV was the middle value, I probably had a 50-50 shot at getting the right answer. Finally, all this presupposes that polls are accurate, a prior that was a subject of lively discussion.

    If you are interested enough to work on this yourself, take a look at this scratch sheet, which contains a comparison of outcomes with my last polling margins. (For yet more numbers, see this directory as well as the poll medians file and the win probabilities used to draw the maps.)

    One fact that emerges is that discrepancies are about 1.5 times as large my error calculations indicated. It’s known that discrepancies tend to be in the direction of the winner and are larger in states with big polling margins. So for Electoral College modeling the discrepancies did not matter much.

    I’m going to attempt to give this a rest for a little while. I’ll be back eventually with more technical validations.

    bks – Well, I apply the same aesthetic to analyzing research data in neuroscience. It’s more a point of view than anything special to election analysis. So sure, why not.

    Christian – You have a point, especially since the bias estimate is not significantly different from zero. I am guardedly pleased to have such attentive readers.

    However, as I just wrote, the general structure of the data did suggest an outcome of 364 EV. I wanted to use polls over a longer time period, during which the estimator appeared to be floating around an unchanging mean. This would likely have arrived at the same result as the prediction I gave. But I ran out of time, and thought it would be worthwhile to take a leap. I guess I’m just a maverick.

    These and other issues deserve careful attention now that we are out of the heat of battle.

  • Scott

    As a physical scientist it appears to me that there just was not enough precision to differentiate between the three outcomes you mention.

  • Sam Wang

    Scott – As a fellow physical scientist, I encourage you to examine the statistical reasoning more carefully.

  • Evans

    As a physical scientist, I agree with Sam.

  • Mary

    Bravo and thank you, Sam. Your site provided the best information and most comfort to so many of us. So, what stocks do you like?

  • Paul

    To add to the chorus: well done!

    Something else worth trumpeting is your excellent graph of likely outcomes. You may not have called IN — but the beauty of the meta-analysis is that it doesn’t just bet on a single likely outcome!

    Instead, it gives a picture of the likelihood of ALL possible outcomes. And a glance at the distribution tells us that the actual outcome was indeed a likely one.

  • ndam

    Thanks so much Prof. Wang. You gave me the science to support my optimism.
    By the way I believe that electoral-vote.com missed Indiana.

  • Sam Wang

    Mary – At the moment, none!

    Paul – Thanks. You are correct about the beauty – and original point – of the meta-analysis. The MO/IN degeneracy reminded me of this.

    ndam – thanks, fixed.

  • Observer

    Mary (and Sam): For stocks, go with Sam’s Princeton colleague Burton Malkiel. (His famous study, A Random Walk Down Wall Street, is out in paperback, revised 9th ed. 2007) Malkiel’s work is also a kind of meta-analysis of massive data. Hint: It leads to reliance on index funds for best long term results. (The best advice for short-term trading is: don’t do it.) WSJ recently asked several Nobel economists for their investment advice. They all said they followed Malkiel and stuck with index funds.

    (I’m shocked — shocked! — that Sam didn’t promptly come up with this connection. But I grant that he’s pretty tired today.)

  • William Ockham

    That’s NE-CD2 (Douglas and part of Sarpy County). It’s pretty sure to go to Obama. McCain has a 569 vote lead, but there are 10,000 early votes from Douglas County left to count as well as 5200 provisional ballots and about 500 provisional ballots from Sarpy County. Obama led McCain in early voting from Douglas County by about 61%-39%. He’s almost certain to pick up enough votes to take the district. If you want to follow the story, go over to the Omaha World-Herald’s web site, http://www.omaha.com .

    I’ve been following it obsessively because I picked Obama to win that electoral vote. If he pulls it, then Missouri is the only one I missed, assuming that the 7000+ provisional ballots don’t change that result. McCain has an almost 6000 vote lead and usually only 40% of provisional ballots get counted in Missouri. So even though most of them are from Obama strongholds, it’s unlikely to flip the state.

  • Sam Wang

    W.O. – That’s interesting – I wasn’t expecting that, and left it out. I thought about it when the Obama campaign opened an office in Nebraska. Lack of data though, hard to model well.

  • William Ockham

    Given that neither Maine nor Nebraska has ever split their electoral votes before and that it’s hard to imagine a scenario where the real outcome would hinge on either state, nobody polls it. Nebraska has a pretty big divide between the two urban areas (Omaha and Lincoln) and the rest of the state. My brother lives in Nebraska, so I thought Obama’s chances were good there.

  • Mason

    Great work, Sam. One thing I can’t readily find on your site are your actual predictions for each state, including the predicted Obama percentages in each state. Can you show me the link where you published those?

    Thanks.

  • Nicholas J. Alcock

    Dear Prof. Sam Wang,
    Thank you for a phoenomenal statistical ride! It was good in 2004 but better in 2008(but ,of course nothing to do with outcome!).
    But two thoughts
    1)Would your use of a t-dist have improved your estimated outcome i.e. tied states:NC, IN,
    MO?
    2)PEC, 538, Pollster appear to have performed exceptionally well but if there had been more polls especially in IN would this not have helped accuracy?
    3)ND for a few, recent weeks was given by a few pollsters as leaning Dem. This appears to be due to an outlier(s). But, it did threaten the accuracy of PEC, 538 ,Pollster, forecast electoral vote et al?

    So, could PEC, 538, pollster nudge the polling companies to poll more frequently in less likely swing states? e.g. 2012, perhaps GA, ND, MT, AZ? Or, am I over-estimating the experts influence?

  • Sam Wang

    Mason – click the word “predictions,” which takes you to predictions.

    Mr. Alcock – A t-distribution is symmetric, and would not turn a positive margin into a losing probability. It indicates less certainty in the case of large margins.

    Indiana was extremely close to tied. It takes four times as many polls to reduce an error bar by half. Such a flood of data seems unlikely.

    North Dakota was sparsely polled, but the anomalous results came from partisan polls. Even so, enough data were still available to make a prediction.

    A shortage of polling data? You have to be kidding.

  • Mason

    Sam, I don’t find this on your site: One thing I can’t readily find on your site: the predicted Obama percentages in each state. I must be overlooking that on your “predictions” page.

  • Nicholas J. Alcock

    Dear Sam,
    Thank for your reply. You suggested moving to a t-dist fom a n-dist. If it doesn’t help
    in forecasting accuracy will you move to this in 2012?

    Wasn’t the problem with ND the lack of polls because using your median poll methodology
    the partisan polls would have been excluded?
    So, isn’t lack of polls a problem? A median poll of one poll outlier is the outlier poll.

  • Ben

    At 3:00 PM EST on the 4th Intrade was showing 364-174. They’re two for two on elections.

  • Mathphysto

    Thanks again for your work, Sam, it was a very successful cycle. Three questions for you…

    1) Has there ever been any consideration of using ARMA/GARCH/ARFIMA/etc models of the time-series polling data for states? Not to predict winners (your approach does well, as do others), but to identify states that make for optimal investments of campaign resources? I’m envisioning polling data, broken down by demographics, and fed into econometric models. Probably this is already done by campaign strategists and the like?

    2) I’ve heard of chaos theory approaches (e.g., analyzing Mondale’s polls from the 1984 Dem primaries), and wonder if you’ve heard of any current uses of this approach.

    3) Now to be really cynical… with the success of Lichtman’s Keys for the White House, do you think it’s even really worth doing all this polling analysis? It seems like the candidates and campaigns generally just wash out and the underlying mechanism for election is the incumbent administration’s performance. Short of one side having a ‘secret weapon’ in their campaign (perhaps like GARCH-type investment analysis) to upset the equilibrium, there doesn’t seem to be much value-added by detailed poll analysis. I’d like to believe otherwise because I enjoy it so much – please convince me! :-)

  • Mike L

    Prof. Wang,

    I believe that I deserve to be included as a candidate for the title of the most dog gone mavericky maverick due to having predicted that NE electoral vote for Obama on your final pre-poll thread—-and you said I’d “gone rogue’!

    Seriously, muchas gracias again for a most splendid job and for your generous giving of so much time to respond to our posts.

    Cheers!
    Mike

  • Sam Wang

    Ben – That’s revisionism. Earlier that morning it was at 375 EV.

  • Matt McIrvin

    Lichtman’s take on structural prediction seems to me to have the same issue as Nate Silver’s take on state polls: it’s more complicated and arbitrary than it probably needs to be. Does anyone really believe that all those components have exactly equal weight? Most of them are also suspiciously subjective, especially when making “post-dictions” to justify the model.

    And then there’s PV/EV splits in close elections. Lichtman predicted that Gore would win in 2000, and then argued that his model predicted the PV winner correctly. Which is all well and good, especially if you think the election was stolen, but Sam Wang apparently made the more interesting and useful prediction that the election would come down to Florida.

    I’m stunned at how well Sam did this cycle. I had a little imaginary bet of sorts going; I thought the state poll aggregator sites would be within 20-30 EV of correct, unlike others who predicted massive Bradley Effects. But since my own gut was telling me it would be closer than Sam was saying it would be, I paid attention to more conservative aggregates and my actual prediction was more lowball, about Obama 245. Instead the lesson seems to be the same as in 2004, that sophisticated probabilistic analysis is good but you should take the polls themselves at face value. Of course some individual states weren’t so spot on, but the aggregate was.

  • Hudson

    Remind me not to move to Arkansas.

  • Hudson

    My take on the best and worst pollsters of this cycle, based on popular vote percentage predictions on Election Eve:

    http://www.dailykos.com/story/2008/11/12/104733/07/259/659768

    However, what I would really like to see done (as explained in the piece) is to evaluate pollsters based on more than a final-day snapshot. Who was “right” on more days once it became an Obama v. McCain race? This is of course complicated… Do you base rightness based on margins, trends, or some other metric? And what do you do about undecideds?

    That is: If Gallup had a 4-point differential on some date four months before the election, with 16% undecideds, can one relate that meaningfully to the final result?

    The goal being: Who should I be paying attention to in January/July/October 2012?

  • Sam Wang

    Hudson – Evaluations of pollsters are not needed if a simple and effective approach is taken: take a median of all of them. By this means I arrived at a near-perfect estimate of the popular vote margin. You won’t be able to outperform that.

    As another example, Benihana’s approach to poll quality this year did not improve his predictions in any individual state or in the overall outcome. So an elaborate scoring system is totally unnecessary.

  • Aaron Andalman

    It is meaningless to compare your personal prediction to the output of other website’s models. I believe 538 was reporting the mean of all simulations? What was the mean of your EV distribution? Alternatively you could compare the mode and median of your two models.

  • Johnny P.

    you need to give the same number of significant digits for your prediction, nate’s prediction and the vote count. Either pick 2 or 3. I think it should be 3. Otherwise if Nate says 52.3 and the vote is 52.5, that gets rounded to 53, and Nate seems off by more than you, when that may not be the case.

  • Sam Wang

    Aaron Andalman – You’re being a bit brash, but all right.

    FiveThirtyEight’s mean and median are almost certainly equal, at 348.5 EV. Our final single-day snapshot median (352 EV) and mean (352.8 EV) were better, as was my actual prediction (364 EV). In the case of the mode, as I explained above, we both got 50 out of 51 and 353 EV, a tie. So we won two out of three, and tied the third. To put it succinctly, we win.

    There are some problems with how you have posed your question. If you would like a much longer answer, see here.

  • Sam Wang

    Johnny P. – If you insist: the vote was 52.7%, which is closer to 53.0% than to 52.3%. So we did better.

    You are missing the broader point: Anyone can make this prediction by taking the last pre-election polls, calculating their median, and turning it into the stated percentage. It’s not hard. The lesson is that there is not a point in doing anything more complex.

    But there is a problem with your question. Stop to think whether it really justifiable to give precision of +/-0.1%. The two-candidate margin in final pre-election polls had a standard error of about 1.0%. In such a situation, giving too many significant figures conveys a false sense of precision that does not actually exist.

  • Larry

    Dr. Wang,

    Isn’t saying you won in this election not really anything to be proud of from a statistical point of view? I understand that being correct is important but when there is a clear winner in the election it seems to be less important. Your numbers on the Franken/Coleman race would be more interesting at this point.

    Respectfully.

Leave a Comment