Princeton Election Consortium

A first draft of electoral history. Since 2004

A first look at post-election evaluation

November 5th, 2008, 5:58am by Sam Wang


Good morning! First, congratulations to President-elect Barack Obama. It’s a historic day. His campaign inspired millions. But also condolences for his grandmother’s passing – what a loss, and at what a time.

Let’s take a quick look at the performance of the Meta-Analysis and my other predictions. Preliminarily, things look quite good.

The EV total. The final snapshot was Obama 352 EV, McCain 186 EV, with a 68% confidence band of [337,367] EV. My personal prediction was Obama 364 EV, McCain 174 EV. With a tentative result of Obama 364 EV, McCain 174 EV, it looks like we may have come in on the money.

This is now the second time that a pure Meta-Analysis of state polls has predicted the winner and the EV breakdown within the confidence interval (2004 story here). Additional assumptions beyond state polls appear to have been unnecessary, at least this year.

Individual states and the mode. The median of the last-3-polls-and-last-week-of-polling got the outcome right in 47 out of 47 cases where the margin was greater than 1%. Of the remaining four, three were ties and one was incorrect (North Carolina). That makes 47 out of 48. The tie-breaking procedure I adopted yesterday morning got three out of four undetermined races right, assuming that the current leads stand in the three closest states, Missouri, North Carolina, and Indiana. This gives an overall record of 50 out of 51 states correct. The exception was Indiana, which Obama is leading by 0.9%. That made the mode of the snapshot, Obama 353 EV, off by 11 EV.

Overall polling bias. I gave you tools for finding a difference between the results and the polls. Based on the final EV total above, the bias seems to be about 1% in favor of Obama, (68% CI +/-1%). That matches well with the average difference in individual state true margins, Obama +0.7 +/- 0.8% (mean +/- SE, n=51).

So although the bias is not significantly different from zero, it is also consistent with my assessment that the Bradley effect would be 0% and the cell-phone effect would be Obama +1%, for a net effect of Obama +1%. So let’s put the Bradley effect to bed.

I’ll have more assessments, including the Presidential popular vote, the Senate, the House, and turnout later this morning. In the meantime – I’m pleased.

Tags: 2008 Election

34 Comments so far ↓

  • JohnL

    Doctor-Professor-Sam, thanks for running the numbers so well yet again.

  • Ed

    Your work is commendable. Thanks for providing some fabulous analysis and hope you find time for a well deserved vacation. My belief in the American people was strongly reinforced last night.

  • SJC

    Great site, thanks Sam!

  • jedwards

    Great job! Congratulations (to us all). Many thanks for all of your amazing work.

  • Bowen Alpern

    Wait. Isn’t a +1% poll bias toward Obama what would have been expected for a (slight) “Bradley effect”?

  • Patrick

    The Bradley effect is the idea that some people will lie to pollsters and say they are voting for a non-white candidate, when they actual plan to vote for the white candidate. This means that, had it existed, it would have been -X% for Obama compared to the polls.

  • Ahar

    Great stuff as usual, Prof. Wang! But, I do disagree (slightly) with putting the “Bradley effect” to bed… Although nationally there clearly was no net Bradley effect, McCain significantly overperformed in some Southern states — MS, GA, SC, LA, AR, TN and perhaps others — so in the South, its perhaps conceivable that some sort of local “Bradley effect” resulted…

  • Alysson

    As a non-statistician, I’ve been referring to web site frequently (I mean — really, really, really! frequently) over the last few weeks to keep my nerves down. Thank you — especially for being right! I have two questions:
    1) Why isn’t Georgia being called when it seems to be so far in favor of McCain
    2) Any ideas why the polling in North Dakota was so off?

    Thanks again, Alysson

  • DavidP

    Looks like recounts to me in NC (2,000 lead out of 4,000,000), MO (6,000 lead out of 2.8m) and possibly IN (30,000 lead out of 2.6m).

    In MN as of this moment 200 votes separate Franken and Coleman. It will be a while before we know the Senate…

  • Sam Wang

    Ahar – It is conceivable, but you have to be careful to impute too much to a single election, which is how the Bradley effect meme has persisted despite evidence to the contrary.

    Also, your data are not sufficient to support the statement. In the Southern states [AL SC TN MS GA LA AR], returns so far suggest discrepancies of [-3 -2 0 1 4 7 10], for a median of 1+/-2%, not statistically significant from the other 44 data points.

    Alysson – Georgia might be still tabulating the early voting. I haven’t kept track of where they are.

    North Dakota had very few polls and the most recent ones were partisan (D). I get a discrepancy of 8% (tie vs. McCain by 8), which isn’t all that bad considering the circumstances.

    Thank you for reading! Whatever will you do now with all your free time?

  • Sam Wang

    DavidP – Good point. It might be premature for me to claim victory. In any event, the Meta-Analysis was bouncing between 350 and 370 EV throughout October.

    Agreed that the Minnesota nail-biter isn’t resolved yet.

  • Ron E.

    Congratulations on the accuracy of your estimate, Sam. I was one who believed that Obama would overperform state and national polls for various reasons and perhaps break 400 EV. Clearly that didn’t happen and in fact for the most part the polls were dead on as was your model. Next time I’m sticking with whatever you predict. :)

  • Sam Wang

    Ron E. – From my end, it probably helps that if I do put my thumb on the scales it should stay light enough to stay in the 68% CI.

  • Mike C.

    Congratulations on your prediction, and bravo on your formula! This is a great site, and as a non-statistician I have learned a lot from visiting here.

    Ten minutes split between here and fiverthirtyeight is more valuable than 10 hours of CNN, CNBC, Fox, etc.

  • John W

    Mike C makes a great point. Like many of us, I followed this race obsessively — ask my wife! Blogs like this one and electoral-vote.com simply blew away the web and TV coverage from big media, which focused on trivialities and consistently made the race seem much closer than suggested by the data. Thanks Sam!

  • blair alef

    On the polling bias, the most interesting thing I see is the degree of bias in the hardcore Republican and Democratic states. To be understandable because these states are polled much less but a number of these state overperformed their poll numbers by 10 points or more for both candidates. This has no effect at all on EV’s but give a good indication of how motivated both sides were when election day came.

  • Glenn

    Dr. Wang: Thanks for providing the excellent analysis, and more importantly, for sharing a common sense approach to watching and participating in the general election — particularly during the last weeks.

  • blair alef

    Important point for the future: In the six most closely watched battleground states (VA, OH, FL, IN, MO, NC) the poll bias was only 0.4 versus the 0.8 national.

  • Evans

    I’m very curious about your thoughts, at this point, about the remaining senate races to be called: AK, MN (570 votes!), GA, and OR. It seems to me like 60 seats still being an open question should be a bigger story than it is so far today…

    My first thoughts are: MN recount kind of needs to happen regardless with such a close result. GA, with the Atlanta area polls still missing 1-2% of results (in counties like Clayton which break more than 4:1 for Martin) , could very well knock Chambliss under 50%. AK… what?!? They elected a felon? What even happens then? OR, why have only 3/4 the votes been counted there? They didn’t want to work longer hours?

    Additionally, as a NC native, with Obama up by 12,000 votes with 100% reporting, any thoughts on why the race hasn’t been called there yet? MO is understandable, as a 6,000 vote margin is twice as close, but I don’t know what’s left to be counted in those places.

    Finally: Today is a happy day for the country. I am proud to be an American for the first time in my adult life, and I think it’s a sentiment shared by the masses of my age group that have watched Bush, a president elected before many of us had the right to vote, condone terrorism of prisoners in Guantanamo, waste our money in a war and further waste it with irresponsible tax cuts without cuts in spending, dissolve many long-standing alliances and villify western europe in the press, disrespect world opinion and the UN, etc. Here’s to 8 years of waking up to the news each day without fear that our government has taken yet another inane step of brinksmanship overnight, but rather with the expectation of the well-thought, metered responses of a true intellectual in power!

  • Byron

    Professor Wang,
    Although I do not know the difference between a Fore Brain from a Brain Fart, nor a cosign from an exit sign, I deeply appreciate your “side” job. I do have a couple of questions. What impact do you believe the successful work of Nate Silver’s and this site may have on individual polling firms in the future, if any. Has polling improved in recent years, or are the last two years just “lucky?”

    By-the-by, I sent many of my worried friends to your site for assurance.

  • blair alef

    Sam,
    I can post a couple of sentences about how ‘successful’ the 538 sit was this year if you want.

  • gprimos1

    Dr Wang,

    As part of your wrap up, I would be interested in seeing some charts summarizing the polling bias across all states, not just battleground. Even if the median bias is close to 0, I get the feeling there is some story to tell deep red/blue states. Could a test of absolute bias abs(predicted – actual) = 0 reveal anything interesting?

  • blair alef

    I can send this in to you too if you want it.

  • Dr. J

    Hi Sam–A huge THANK YOU for this website! I am not at all surprised that your analyses performed so well (meta-analysis has always seemed to me to be the way to go…), and it was very helpful to have such an authoritative source available–as so many others have already observed. Much better than standard news outlets. You have performed a major public service.

  • bks

    Wow! It’s almost enough to make me believe in statistics. Excellent site.

    –bks

  • Observer

    Another historic moment:

    We have a constitutional law professor as president-elect!

    (Not seen since — substantively, crediting him with the Federalist Papers — James Madison)

    – a happy but dazed attorney

  • blair alef

    On poll bias, doesn’t the increased number and increased frequency of polling in highly contested states increase the accuracy of the polling aggregate for that state? It seems like this should be true, and from the way this year’s margins lay out it looks like it is true.

    If so, it is a nice, inherent safeguard against poll bias: that states with increasing importance naturally end up with more accurate polling aggregates.

  • Nikhil

    I’m sure someone has already sent this to you, but just in case they haven’t:

    http://www.xkcd.com/500/

    Thanks for all the blogging, it’s been fun to follow along! The cheat sheet was especially helpful last night. Can’t wait to see what happens in Minnesota with the Senate race; it seems we may have to wait up to a month…

  • Fokko

    While the state polls pretty accurately predicted who would win which state, there are immense errors in the percentages. Of course the biggest errors are in states which do not really matter (because the winner is so clear), and battleground states are much better predicted.

    I’d like to know your comments on whether you think state poll data can use state polls like this and trust them as much as you do; it seems the errors are much larger than your prediction method allows for, but mostly in states for which it does not matter. (Though, for example in Arkansas the polls gave McCain a 10% lead, and the error is also 10%; it’s just that the error is in the right direction this very save state indeed went to McCain).

  • Sam Wang

    Fokko – Based on some analysis I am doing now, I believe your interpretation is incorrect. It appears that the differences are not large compared with error bars. So my working answer is that the polls have a very good track record, and appear to have statistical properties consistent with this empirical fact.

  • Oz Observer

    Evan @ 19
    Firstly, as to your being proud to be American,and your reasons for not having been so, as a non American I am glad to be able to be pro Yank for the first time in eight years, and for the same reasons as you!
    Secondly, am I giving too much credit to Alaskans to think that they might want Stevens to be expelled from the Senate, and for them then to have a chance to elect a more palatable Republican(Palin, perhaps) in the special election, rather than have a Democrat for six years?

  • Observer

    Oops, slight error: Bill Clinton also taught Con Law (at U. of Arkansas), before winning the governorship in Arkansas.

  • Observer

    Oz Observer: It appears that the Senate will expel Steven under its rules. Then there will be a special election. No idea how that will turn out in Alaska, but I do expect that Palin will stay in the governorship rather than run for the Senate seat.

  • Henry Levenson

    Prof. Wang,

    Thanks for all the work the Princeton Election Consortium accomplished.

    It became sort of a wake up for me each morning over the past two weeks to reassure myself that today, November 5, would be the beginning of a new President named Obama and the visible end of a dysfunctional administration in Washington.

    It was nice speaking to you several weeks ago about meta analyses as applied to other subjects.

Leave a Comment