Princeton Election Consortium

Innovations in democracy since 2004

Outcome: Biden 306 EV (D+1.2% from toss-up), Senate 50 D (D+1.0%)
Nov 3 polls: Biden 342 EV (D+5.3%), Senate 50-55 D (D+3.9%), House control D+4.6%
Moneyball states: President AZ NE-2 NV, Senate MT ME AK, Legislatures KS TX NC

Gallup’s man misses the point

November 17th, 2012, 1:04am by Sam Wang

(Update, Nov. 19: now with Gallup’s performance shown in graph form.)

Gallup editor-in-chief Frank Newport appears to be on a campaign against poll aggregation. In a recent essay (‘Polling, Likely Voters, and the Law of the Commons, he writes:

It’s much easier, cheaper, and mostly less risky to focus on aggregating and analyzing others’ polls. Organizations that traditionally go to the expense and effort to conduct individual polls could, in theory, decide to put their efforts into aggregation and statistical analyses of other people’s polls in the next election cycle and cut out their own polling. If many organizations make this seemingly rational decision, we could quickly be in a situation in which there are fewer and fewer polls left to aggregate and put into statistical models. Many individual rational decisions could result in a loss for the collective interest of those interested in public opinion.

Oh, please. Considering Gallup’s performance in estimating the national race, this could be interpreted as a defensive move by this year’s equivalent of the Literary Digest poll (‘Landon by a landslide,’ George Mason University).

Newport misses the positive value that we bring to his activity. It is too bad, because what we do can ultimately increase the relevance of his organization. Here’s why.

First and foremost, poll aggregation is not like other forms of news aggregation. News aggregators like the Huffington Post basically recycle stories. Those of us who examine groups of polls add value for both reader and pollster:

  • For the reader, we cut through the noise. Individual polls contain two kinds of error arising from (a) inherent limitations of sampling, and (b) systematic errors made by individual pollsters. By using robust statistical tools, we reduce and cancel these errors to obtain a far superior result.
  • For the pollster, we offer a benchmark for future performance. Paul Starr pointed out to me recently that a likely reason for the improvement in political opinion polls since the 1930’s has been the fact that polls are easily compared with election results. Until recently, this comparison was limited by statistical sampling error. Now, aggregators can grade a pollster’s accuracy to within 1 percentage point.

Newport does not fully acknowledge the second point. Regarding his own organization’s performance he writes:

The “gap” difference was….well within the statistical margin of error and underscore[s] the accuracy of random sampling today.

Actually, no. Thanks to aggregation, we can say with great specificity that Gallup’s national October numbers (Romney ahead by 2% to 6%) were systematically off by 4-8% from the true margin at the time, Obama +2.0% (“A final unskewing,” November 12th). No wonder he doesn’t like us. Underneath the bluster and threat, I believe that Newport’s real problem is Gallup’s own poor showing.

The red curve indicates Gallup’s data, plotted with 1-sigma error bars. The black curve is my best estimate of the true Obama-Romney margin, based on all available national surveys (“A final unskewing,” Nov. 12). The last data point is off by about 4.0% (2 sigma), and the three data points before that are off by more.

In fairness, it was not only Gallup whose national numbers were off. National polls as a group were biased by an average of 2.4 +/- 0.4% toward Mitt Romney. State polls were a superior source of information: our Popular Vote Meta-Margin did far better than the national Romney-vs.-Obama average in predicting the national vote.

Newport is correct that poll aggregation does devalue the news value of any single poll. That is the point of the activity. It’s why I started doing it in 2004. I was driven to distraction by breathless stories on single polls. This year, I almost blew a blood vessel when I saw the entire front page of USA Today dedicated to a single Gallup poll that was an outlier. Let’s face it, news organizations love outliers. If aggregation kills that kind of story in the future, our entire nation wins.

Despite Newport’s complaints, my own view for the future of his field is bright. Aggregators like PEC bring focus to their activity and add a new dimension. However, now they have to be nimble. They can’t get stuck in a rut reporting only topline numbers. That low-hanging fruit will soon be gone.

But there are many ways they can improve their game. For example:

Focus on crosstabs. Much of the richness in polls comes in the details: knowing that young voters tilt Democratic, or that many Romney supporters would rather identify themselves as independents than as Republicans. Those details carry endless news interest.

Watch one another. This year PPP missed a big story by failing to report a sudden plunge in support for Todd Akin (R) in his Missouri Senate race to unseat Claire McCaskill (D), after his “legitimate rape” comment (“Akin sheds 8 points overnight to near-tie,” August 12). They could have caught that story if they had been willing to compare their own results with other pollsters. This pridefulness should stop.

Develop new products. The most interesting polls this season were products like the RAND longitudinal survey, in which the same respondents were surveyed repeatedly. Gallup itself had some fascinating results from their tracking poll, which showed an overnight jump in President Obama’s job approval rating after Michelle Obama’s speech (“Michelle Obama, the Great Persuader,” September 9th). Let a hundred flowers bloom!

Learn from the crowd – but don’t be afraid to go the other way. Pollsters can learn not only from each other’s biases, but also from where the polls are fielded. In the five weeks after October 1, showed 97 national polls. That is complete overkill. Some of that effort would have been better spent on downticket Senate or House races – or even nonswing states, which received so little love this year.

Go local! Many pollsters conduct national surveys as loss leaders. They do it for the media exposure. But there is plenty of publicity to be had in other races. If pollsters swooped in there, it would garner them publicity – and even help slow the decline in local journalism. That would be a win for pollsters – and add much-needed diversity to our media culture.

And here we come to an irony: the Gallup organization is rich in expertise, and is a leader in adding value in interesting ways. If they continue to do that – and stop complaining about the new kids on the block – they can maintain their relevance. I wish them every success.

Update: To emphasize a point above, for organizations like Gallup, I am under the impression that most of their income comes from other polling, not what aggregators analyze. Therefore I believe that the availability of public polling data is not under threat. However, their brand did take an undeniable hit this year.

Tags: 2012 Election

176 Comments so far ↓

  • wheelers cat

    Its BACK!
    check out Dr. Wang’s twitter stream for what hes been up to while the site was down.

  • Henry Rosen

    To Sam Wang:

    First of all – thanks! For your tireless focus on fact and math. While Nate Silver is the public face of polling aggregation, you and some others have helped refine it to solid science.

    Second, I would like to see more analysis from you about the RAND American Life Panel. It seems to me that they did a fantastic job; their final prediction was a 3.32% margin for Obama. Turns out the final margin was 3.5% – and may go higher. But I think the way their poll worked very accurately reflected the (relatively minor) ebbs and flows of the campaign.

  • Alan Cobo-Lewis

    Did you realize (maybe you did, but I didn’t) that George Gallup is buried in Princeton Cemetery?

  • A New Jersey Farmer

    I used to make calls for Gallup when they were in Research Park on 206. Fun times.

  • Hayford Peirce

    I see that Mitt’s percentage of the total vote has now cracked the magic 47.49999% barrier and is at 47.36%. Which means, rounded off, he is at the delicious figure of 47 percent!

    Also: Obama’s margin over Mitt, the believer in internal polls, is now 3.6%….

    • Pat

      Yep, and now the margin is Obama 50.94% – Romney 47.32%.
      Dave Wassermann @Redistrict suggests there may be about 1 million votes left to be counted, mostly from New York. Assuming those remaining votes split the same way that they have so far in the state (i.e. let’s add 620,000 votes to Obama and 360,000 to Romney), that would make a final margin of 3.8% (Obama 51.03% – Romney 47.23%), very impressive compared to what the national polls predicted.

      Damn, no one is here anymore… Sam, I guess we will need at least our monthly blog post, so that not everyone disappears…

    • Sam Wang

      Pat, it is unavoidable that there won’t be that many people here. I have some ideas that I want to float, and will post them from time to time.

      One point on my mind is the fact that as per Wasserman, even state polls were a bit off. 1% is within acceptable limits, but 2% would be worth understanding why.

  • mediaglyphic

    A lot of lurkers still here, just not too much to post!!

    Any thoughts on what the polls are saying about Obama’s stance in the fiscal cliff?

  • Olav Grinde

    Looks like Reid & Co are pressing filibuster reform.

    My only question is: why not do this now, before New Year’s? For starters, the Democrats can ensure the Senate finally fills those judicial positions that GOP obstructionism has kept vacant all to long.

    • Suja

      Olav, the votes are probably not there to override the expected filibuster. They can allegedly use the ‘constitutional option’ on the first day of the new session in January to pass legislation through a simple majority.

  • Dave Gikow

    I don’t think that PPP “missed a big story” by not reporting the Akin poll result — I think that was a conscious decision to reduce the chances of Akin withdrawing from the race before the deadline. Just like Rasmussen suddenly showed Akin getting killed in an attempt to convince him to withdraw.

    And I applaud PPP’s decision if my assumption is correct – while I don’t expect PPP to alter results to affect the outcome like Rasmussen does, as a Democratic pollster I am glad that they made a conscious decision to bait Akin into staying in the race.

Leave a Comment