Princeton Election Consortium

A first draft of electoral history. Since 2004

The economics of reporting polls

October 17th, 2008, 11:49am by Sam Wang


The only thing happening in the Meta-Analysis is a slight, slow widening of Obama’s lead. Some of you want to know about individual polls, such as a recent Gallup national poll showing Obama ahead by only +2% (standard likely-voter model) or +6% (high-turnout model). I confess that I tend to ignore individual polls because of the statistical variability. So it didn’t occur to me to care about this particular data point. Obama is still crushing McCain, period.

But there is a lesson to be learned here: It is not in the interest of individual pollsters or media organizations for you to have the most accurate possible picture of the horserace. Here is why.

Uncertainties such as the margin of error can be reduced by taking more samples. An individual pollster can halve the margin of error by surveying 4 times as many people. It’s a square-root relationship: N samples lead to a sqrt(N)-fold reduction in uncertainty. The same is true for combining polls, with the added advantage of reducing the effects of methodological variation. Thus the value of poll-aggregation sites like this one. Meta-Analysis worked extremely well in 2004 and 2006, and is likely to do so again this year.

So why don’t more pollsters or media organizations aggregate polls? The CNN Poll of Polls is a start, but it’s an exception. Two forces encourage bad horserace reporting:

Competition among pollsters. It’s not in the interest of individual pollsters to say “average my results with the others.” It’s also not advantageous to collect a larger sample once the margin of error meets industry standards.

The hungry media beast. With news budgets on the decline, it’s costly to report real news. Why pay for investigative reporting when you can buy a poll and report the horserace? Within the area of poll reporting, market forces discourage high accuracy. For example, commissioning a survey of 4 times as many people would reduce uncertainty by a factor of two. But why pay 4 times as much for data that generate a lower likelihood of an apparent – and reportable – swing?

For these reasons, media organizations aren’t motivated to report polling results with the maximum possible statistical power. The Meta-Analysis of State Polls is pure data reduction, basically a more general version of averaging. As a result, the top-line result is very steady. This a case where a blogger-hobbyist can add value. We use the polling/media system to provide added value – for cheap.

Which brings us back to costs…

If you like what I do, you can go to the ActBlue page (or the NRSC). Or, in my regular life, take a look at my book, Welcome To Your Brain: Why You Lose Your Car Keys But Never Forget How To Drive. [Amazon.com] It’s an introduction to how your brain really works, complete with useful tips and debunkings of myths. For instance, alcohol doesn’t kill brain cells. Playing Mozart to babies doesn’t make them smarter. Exercise is better for your brain than Sudoku. And – your brain is set up to trick you into thinking that Barack Obama is a Muslim. Check it out!

Tags: 2008 Election

26 Comments so far ↓

  • Danny Ainge

    Sam…love your book. What’s weird for me is that I know and understand many of the things you mention about the brain, yet I still do them. Lazy I guess…

  • Sam Wang

    Jim Wheaton – I would characterize that statement as statistical malpractice. Instead of a sampling error, CNN needs to place a confidence interval on the average, which is the point of reporting sampling error in a single poll. More general concepts in statistics are the standard error of the mean (approximately a 68% confidence interval), and the 95% confidence interval (similar to the famous Margin of Error). This is basic statistical good practive.

    On this site I am constantly reporting these kinds of quantities. In the case of a poll of national polls, if one uses 10 polls, as in my most recent post, the 95% confidence interval is Obama leading by 5 to 9 percentage points.

  • Jim Wheaton

    The article below, from CNN shows a “poll of polls” but claims that it “does not have a sampling error”. Is this an example of laziness on their part, or choosing polls that are statistically incompatible in some way?

    -Jim —-
    (CNN) – As Election Day inches closer, Barack Obama continues to hold a significant lead over John McCain, according to CNN’s average of several recent polls.

    The Illinois senator now holds an 8-point lead over McCain in the latest CNN poll of polls, 50 percent to 42 percent. That lead is 1 point larger than it was in Wednesday’s poll of polls.

    The national general election poll of polls consists of four recent surveys: Fox/Opinion Dynamics (October 20-21), Reuters/C-SPAN/Zogby (October 20-22), Gallup (October 20-22) and Diageo/Hotline (October 20-22). The Poll of Polls does not have a sampling error.

  • Sam Wang

    Chris has a point. To put it in a less partisan manner, there is a considerable advantage to polling being done by many organizations, each with its own view of defining priorities and likely votes. An aggregate of pollsters performs much better than any one pollster.

    Again, I point to the remarkable power of poll meta-analysis in 2004 and 2006 in advance prediction of outcomes. It would be hard for polling by one organization to reproduce that – or to inspire trust, as indicated by Chris.

  • Trevor

    Mr. Wang:
    I’ve been enjoying your analysis since the conventions. Whenever I feel over-drama’d by cable news I come back here for a reality check.
    In return, I ordered a copy of your book through my local b&n today.

  • Chris

    David Shor: By the same logic couldn’t you say that any publicly available information is a public good? (Indeed, isn’t that one of the original rationales for copyrights – they’re the Enclosure Acts for information?)

    Yet, somehow, the idea of a Ministry of Information fails to reassure me. Particularly at present – imagine what kind of heck of a job a Bush administration polling agency could do. RCP’s cherry picking would be minor by comparison.

  • David Shor

    I think this skirts the real economic issue here: Polls are a public good. Information generated by polls are enjoyed by everyone, but paid for by a few.

    By basic economics, both sampling sizes and number of polls will be under-produced.

    There is really a compelling case that Public Opinion Polls, like the census bureau or ACS, should be done by a government agency.

  • Peter H

    Hi Sam,
    Thanks for your insight. Very enlightening. How do you feel about the work being done at FiveThirtyEight.com?

    BTW, I’m a musician, but have been a brain hobbyist since the mid seventies. I live two blocks from Borders, and I’m on my way to pick up your book. Do you think science will every crack consciousness? I asked Pinker at Borders and he hypothesizes that it is by its very nature beyond human understanding, similar to Colin McGinn’s view. Someone asked, “What about Edelman?” and Pinker just said, “Not even close.” What do you think?

    Regards,
    Peter H

  • MattF

    Gallup is, I think, making a start with the RV/LV problem by giving results from a number of different models. I guess that one would like to estimate the RV/LV uncertainty in a more systematic fashion (perhaps by simulation)– but maybe the uncertainty turns out to be unpleasantly large. One can only wonder.

  • Matt McIrvin

    Couldn’t the difference between Gallup LV1 and LV2 be indicative of a large systematic error in all the RV polls, which would be immune to most meta-analysis? As far as I can tell it has to do with demographics: traditional “likely voters” are older and less likely to be Obama supporters. So Obama’s lead actually depends heavily on youth turnout, which has a spotty enough track record to make Dem strategists very nervous. If the young ones flake out again, many of these polls could be overstating Obama’s support.

    I think that’s part of why the Obama campaign is starting to ring the alarm about complacency; their GOTV depends on energy and they want their people a little worried. Analysts like you, who say that Obama basically has it in the bag and people should consider channeling money elsewhere, worry them very much.

    But the thing is, they also have an incentive to act worried even if you’re right! I’m not sure what to think, myself…

    And then there’s the contrary take of some of the fivethirtyeight people, who think that an “inevitability” narrative actually does more good for Obama than harm, because people like to vote for winners, and despair (for Republicans) is more damaging than complacency (for Democrats). Could be…

  • blislife

    Say, Sam, I assume you or others on this blog have heard of the brilliant polling expert, Alan F. Kay. (http://www.alanfkay.com). I notice his articles stop in 2006. Has anything happened to him? His analysis of the polling world and his social solutions offer some of the best thinking around. Anyone with any knowledge of Mr. Kay: I’d love an update.

  • Andrew Alef

    You have some methodologies that are different than Mr. Silver’s on fivethrityeight. Currently, you have somewhat different predictions of the outcome. It might be fun for us lay people to learn what the differences are.

  • Fred

    I agree with the comments: thanks for your thoughtful, sane analysis. Keep up the excellent work.

  • ron

    Great site, I too have thought a “poll of polls” would be much more reflective of sate of the race, and tend to iron out one poll.

  • jefflz

    This may seem obvious and or inane, but does it make any sense to combine in a statistically valid fashion electoral vote projections, across all the poll analyses. (RCP, 538, polster.com, princeton) ?

  • Cindy

    Hi Sam!

    This is the first time I’ve been turned on to your site and I have to thank a HuffPoster for that. Thank you for a thoughtful analytical post on polling. Yes, I AM scared to death that the Republicans will once again “steal” this election; even more afraid that he might win without stealing it.
    I cannot believe there are still so many radically racist citizens in this country (in this day and age) that refuse to accept anything BUT the truth regarding Obama’s credentials and background. Just yesterday, my elderly neighbor’s sister (three of em, all in their eighties) siad she’d NEVER vote for “one of those African people”. Her two sisters agreed, no matter which facts I presented to them.
    These Gabor sisters are only the tip of the iceberg. As Obama has repeatedly stated, this is no time to take the win for granted.
    Thanks again – I will now become one of your regular readers – please don;t let us down, lol.

  • Sam Wang

    Currently, in national surveys conducted 10/12-17 the median margin is Obama +5.0+/-1.2% (n=7, range 4-10%), possibly smaller but not yet statistically significant compared with a baseline of Obama +6%. In any event, some closing of the gap would have been expected; it often occurs in the Presidential race at this stage of the campaign.

    State polls lag this time window by a few days. We will have a clearer picture of the true change on Monday, when many new state polls will be released.

  • Observer

    The plague of horse race reporting of this presidential electon cycle has been just awful. Any slightest movement in polls has been subjected to maximum narrative fallacy. Your site is a good corrective, but what we are really hurting for is a news organization that would publicize how little the other news outlets’ horse race stories really mean and how unsupportable their logic so often is.

    This can be a better country. Doesn’t mean it will be of course.

  • Lisa

    First poster, but lurker for quite a while (I myself am working with numbers in my job). Thanks for always great information and analysis.

    If you watched Michele Bachmann’s (R-MN) appearance in Hardball tonight where she out-Palined Sarah Palin — and if not yet, you should DEFINITELY check it out — could you consider sending money to her opponent Elwyn Tinklenberg?

    Thanks, I look forward to reading your next post.

  • Marginal Intellect

    Great post, Sam. Thanks for the site, I look at it every day. Polling is misunderstood but this year I feel even “marginal intellects” are beginning to figure it out.

  • Peter

    I, too, check the site every day, even if I don’t comment. Keep up the great work!

  • Glenn

    Mr. Wang… I write again to thank you for being one of few sources of reason, and thus comfort, throughout this seemingly interminable campaign.

  • gprimos1

    It is interesting to consider the effect of the different weighting methodologies. This article mentions that some polls seem to be making arbitrary assumptions about political party weights.

    Would the issues involved here have a significant effect on the meta-analysis?

    http://online.wsj.com/article/SB122420151553142939.html#articleTabs_interactive%26articleTabs%3Darticle

  • Ginny Mayer

    Thanks for the reassurance about individual poll movements as I anxiously await the latest Gallup every day and have noted with dismay that it has been showing McCain creeping up on Obama for several days now (with a “sigh of relief” today as Obama moved up one.)

  • Elizabeth Duvert

    Thanks for the information. I can’t buy your book directly, but I’ve asked our library to do so. Still check your site every day.

Leave a Comment