Princeton Election Consortium

A first draft of electoral history. Since 2004

What’s up with the national polls?

August 8th, 2012, 9:09am by Sam Wang


National polling averages are contradictory: Pollster gives Obama by +0.2%, TalkingPointsMemo gives Romney by +0.1%, and RealClearPolitics gives Obama by +3.5%. Who’s right, and why are these measures fluctuating from day to day?

Much of the discrepancy is explained by my post on the problem with averaging:

These measures bounce around far less when using medians. RCP data since July 16th give a median of Obama +4.0 +/- 1.3% (estimated SEM, n=5), consistent with the Meta-margin calculated above, which is based on state polls. And a few weeks ago, the RCP median was at Obama +2.0%, again matching the Meta-analysis.

A second answer is the use of a long time window. These other sites’ engines achieve a smoother average by looking over a longer time window. The use of medians would allow them to achieve better time-sensitivity.

Third and finally, note that Pollster and TPM national-poll aggregators are dominated by many data points from a few polling organizations. It is a clear methodological error to rely so heavily on any one pollster. At a minimum, all data from the same pollster within a given time period should be combined to make one contributing data point. (Update: Matt Dickinson points out in comments that Pollster in fact does do this. My error!)

Update: I forgot to mention two more issues. It is not necessarily the case that national polls are a more accurate gauge of the race than state-level polls. At any given moment there are fewer pollsters operating at the national level, which means that they are subject to greater fluctuation. Also, national polls sample from all states, which are politically diverse. This presents a difficult sampling problem.

Tags: 2012 Election · President

9 Comments so far ↓

  • Tapen Sinha

    Two points.

    1. Rasmussen, either implicitly or explicitly, is part of the Republican Party machinery. By polling more heavily the Republicans, they sway the outcome. That gives a boost to the Republican candidate in the cross section data.

    2. Adding time series data for the same poll without weighting seems to go against the grain of any kind of time series analysis.

  • Matt Dickinson

    Sam,
    Mark Blumenthal at Pollster.com changed their tracking survey so that it only includes one data point per pollster – this includes the Rasmussen daily tracking poll. See:
    http://tinyurl.com/d6ogzkg
    This should address some of your concerns regarding Rassmussen’s undue influence on their trend line.

    • Sam Wang

      Thank you for pointing that out. I was just suggesting this to another major aggregator. Median-based statistics would help as well.

  • Sean

    RCP was pretty damn accurate in their averages in ’04 & ’08, so, I’m inclined to believe Obama is up roughly 4 points or so. Why TPM has decided to count more than one Rasmussen and Gallup poll is beyond me – that certainly will shift the numbers a bit.

    In fact, I’d take Rasmussen out of the average altogether. It’s clear they’re more pro-Romney than almost every poll. I guess there’s an off chance they might be accurate, but when you’re going against the trend in polling, the odds don’t favor that.

  • Matt McIrvin

    It’s interesting that RCP, the nominally conservative site, is giving Obama the bigger margin.

    I respect their aggregation a lot and get the impression they’ve always been very accurate, though I dimly remember some criticism in earlier cycles for leaning on polls that had Republican house effects. Their mix may not have actually been wrong, though, which is that same old question of what you pick as your center if you try to adjust for house effects.

  • Tapen Sinha

    Did you notice today how much TPM has shifted? That tells you how unstable their estimates are.

    • Sam Wang

      Poll aggregation, properly done, should get rid of such large shifts. There is very little pressure to do so. If an estimate of the race never changes…there’s no news! Compare that with the estimator here, which changes slowly and steadily, if at all. It’s not exactly riveting viewing. Obama’s Meta-margin is still +4 to +5%, same as it’s been for several weeks…zzzzzz.

      In my view, it would be better if people with more statistical smarts were in charge of aggregators like the TPM PollTracker. Too bad; it’s a missed opportunity.

  • Matt McIrvin

    What really drives me nuts is the multi-paragraph news story about what’s supposedly driving a shift in a single poll result.

    Also the treatment of a single poll’s “margin of error” (the statistical uncertainty arising purely from sample size) as if it were a hard threshold separating real and not-real differences.

    Also… well, a lot of things, actually.

Leave a Comment