Princeton Election Consortium

Innovations in democracy since 2004

Outcome: Biden 306 EV (D+1.2% from toss-up), Senate 50 D (D+1.0%)
Nov 3 polls: Biden 342 EV (D+5.3%), Senate 50-55 D (D+3.9%), House control D+4.6%
Moneyball states: President AZ NE-2 NV, Senate MT ME AK, Legislatures KS TX NC

FiveThirtyEight’s blurred sense of time

September 5th, 2012, 8:41am by Sam Wang

Yesterday I reported early indications of a negative post-convention bounce for Romney. Some of you cited a claim by FiveThirtyEight of a positive bounce. However, that interpretation is incorrect. Because of limits in his statistical approach, he is reporting cumulative change since the end of July, a period dominated by the event I reported to you over a week ago: the post-Ryan-VP bounce.

A more accurate interpretation of recent events is as follows: Romney’s post-Ryan-VP bounce has slipped quite a bit toward pre-Ryan levels. The GOP convention bounce itself is zero at best and possibly negative.

First, let’s look at the data cited there.

Nate's bounce error

This list contains eight before-and-after comparisons: five state polls and three US tracking polls. However, the pre-convention state polls were done ending on July 23 through August 5. So they are reporting the total net change since August 5. What else happened since then? Paul Ryan’s VP nomination:

Ryan and Palin bounces in Meta-margin

At the time, I told you that Ryan-VP led to a three-point bounce for Romney. Yesterday’s report at FiveThirtyEight confirms this finding, but without information about when the bounce occurred.

What about the three national polls? They are more closely spaced in time, and report bounces of  (positive means a pro-Romney bounce) +6.0%, +5.5%, and -2.0%, for an average of +3.2 +/- 2.6 % (mean +/- SEM), giving p=0.15 by one-tailed test — not statistically significant. So they are not sufficient to answer the question of whether or when a bounce occurred.

The fundamental problem here is that paired comparisons are very hard to come by. With that constraint, before/after comparisons must by necessity have poor time resolution. In contrast, by pooling of all state polls using the Meta-analysis, we were able to resolve the post-Ryan VP bounce in time.


Finally, a postscript on why the state poll Meta-analysis is so sensitive. Basically, it is an extremely effective means of reducing noise. Uncertainty goes down proportionally to the square root of N, where N is the number of polls. At any given moment we are using dozens of polls. The resulting estimator has an equivalent precision of about 0.2% in margin.

I know this to be the case for three reasons: (1) In 2004 and 2008, the Election Eve Meta-analysis came within 0 EV and 1* EV of the final outcome. (2) The typical daily fluctuation of the EV and Meta-margin history are quite small. (3) When there is a turning point, for instance with the Bain attacks and the Ryan VP nomnation, the results of the Meta-analysis change in a largely monotonic fashion, i.e. the amount by which they jitter up and down is <5 EV and <0.5%.

*In comments, Christian reminds me that attaining a 1-EV error required variance minimization over the final several weeks of the campaign. The final single-day estimates were 360/363/364/352 EV.

Tags: 2012 Election · President

26 Comments so far ↓

  • Christian

    Hi Sam, I’ve been enjoying your recent posts. I have a question abou the Meta-Analysis. In the 2008 results, you left a note that the election-eve snapshot was noisy. It predicted Obama EV = 352, but you had to do a variance minimization to hit Obama EV = 364. Do you apply a variance minimization for every Meta-Analysis result? If not all, what are the criteria for its use? These questions are in reference to your postscript. Thanks!

    • Sam Wang

      I apologize, and thank you for this correction. I had forgotten that such an averaging was necessary. I see here that the day-by-day EV estimator was 360, 364, 363…and landed at 352 EV, as you remind me. The difficulty was the “lumpiness” of the distribution since IN and MO had 11 EV each. So I am claiming too much. The confidence of the single-day estimator clearly depends on Electoral College quantization.

      It appears that some averaging is needed. This gets back to what I found a few weeks ago regarding the fact that opinion is correlated for up to 40 days. A better approach would be to take that into account. Variance-minimization was an approximation to that, kind of seat-of-the-pants. How to do better?

  • Matt McIrvin

    Meanwhile, Todd Akin’s anti-bounce appears to be ending; he’s at parity with McCaskill in recent polls.

  • Sam Wang

    For Akin need one more data point I think?

  • Olav Grinde

    …ah, if only Dr Wang could obtain polls on request.

  • Leo

    I have to say, although I really enjoy both of your sites, and have always found the little tit-for-tats between you kind of silly, that this criticism of 538 seems to be right on. It is pretty obvious that the method he has used to adjust for convention bounce is flawed and he is trying to cover up that flaw by finding a bounce where there is none.

    • Sam Wang

      Leo – totally reasonable on your part. To be honest, I don’t particularly enjoy it either. But when fanboys/girls cite a statement that is statistically subpar, it’s hard to hold back. He gets you interested, I try to correct the wrong analysis.

      I’ll get back to my own math shortly.

      Wait…he doesn’t link to me, does he? I don’t read his site regularly.

  • Matt

    The funny thing is Nate’s own model is showing that there was no meaningful bounce. His version that is based only on polling has been flat while his version that is correcting for something that isn’t there is showing an Obama bounce.

    I do appreciate what he is trying to do, if only as an experiment. I mean his prediction model is trying to tell the future, after all, and it is reasonable to claim a lack of a Romney bounce means that Obama is more likely to win. I am skeptical that he can take everything into account in a mathematically meaningful way. So yes Obama is probably some unknown percent more likely to win because Romney failed to get momentum coming out of his convention. Is he the couple percentage points likelier to win that Nate’s model has calculated? I’m not so sure of that.

    I also appreciate Dr. Wang’s straight forward approach. I think his simpler is better approach is probably right, especially given how robust his simpler model has proven to be. I think his critiques of Nate have helped me understand the problem better than I had before with my very basic understanding of statistics.

  • badni

    I know that most of my statements are statistically subpar, but I believe it’s just because of a string of bad luck.

  • Terry

    I am a former researcher and current practicing forensic psychologist, I am continually amazed at the rather obvious discrepancy between cited data (e.g. testing results) and stated conclusions.

    Evidenced-based practice requires consistency. Nate blew it on this one; he doesn’t get a pass.

  • Leo

    Sam — I don’t think he links to you know from his NYT site. I thought that he had done so back in 2004 when he was new kid on the block but I could be wrong about that.

  • Leo

    Oops, I meant 2008.

  • wheelers cat

    Nate never links Dr. Wang.
    Sometimes the statistics nerdcore does in the comments.

  • xian

    there was one link or maybe just a citation, usng Sam’s name, I think, not the PEC label, a few weeks ago.

  • wheelers cat

    xian, no cite. PEC was mentioned in the average of poll aggregators post.

  • Ralph Reinhold

    Dr Wang: I would think, because the sample sets are different sizes, that the proper way to average polls is to resynthesize the quantities, sum the totals and then get the new percentage.

  • Sam Champion


    Two off-topic questions:

    To what extent can very large and asymmetric expenditures on negative advertising this late in an election cycle (e.g. what we will see from the Romney campaign and his SuperPacs) move the needle on performance? Is this accounted for in your model?


    • Sam Wang

      Sam Champion – The only input to the calculation is polls. I regard everything else as being “upstream” of polls or just adding noise. Read this to get my take on stuff like what you are describing.

      Campaign spending might be informative in two ways: (1) filling in missing data, as in Senate/House races; and (2) predicting future movement. At the level of the Presidential race it seems unlikely to have an effect. Think about it. Do you think that Obama lacks for a platform?

      I have written about all this over the last month…

  • Matt McIrvin

    There’s something undeniably odd going on with the national polls: the rising trend toward Romney has completely closed the gap, and he’s even with Obama or slightly ahead, while he remains behind in EV.

    This isn’t Republican convention bounce; my best theory is that it’s continued Ryanmania. The rise started when Romney picked Ryan.

    But it barely registers in the EV aggregates or the Meta-Margin, so it might be mostly red-state Santorum or Gingrich voters coming home and making the places unbelievably red, with little effect on the actual election.

    Still, Ohio and Wisconsin are now registering as toss-ups, so maybe the convention did give the Republicans a little Great Lakes bounce after all.

    It will be interesting to see what happens in the post-DNC polling, given the amazing reception Michelle Obama’s and Bill Clinton’s speeches got. It’s hard to keep it all in perspective and remember that speeches and conventions have temporary effects at best.

  • Matt McIrvin

    …More generally, if there’s anything the history of this campaign demonstrates, it’s that the 2000-inspired lore that the Electoral College favors Republicans is not necessarily true.

  • wheelers cat

    Matt, couldnt it just be robopoll bias?

  • Matt McIrvin

    There’s probably some robopoll bias in there (Rasmussen has Romney ahead by 3), but that doesn’t explain changes over time, since the robopolls have been heavily weighted in those averages all along. And some of the polls showing Romney even with Obama aren’t robopolls.

  • xian

    Matt, the old electoral college lock dated from the Reagan/Bush period, when California was still a red state (imho).

  • Matt McIrvin

    Yeah, one thing that the Electoral College obviously does is amplify clear majorities into massive blowouts. And in the Reagan/Bush days, that was what Republicans had.

Leave a Comment