(Update, Nov. 19: now with Gallup’s performance shown in graph form.)
Gallup editor-in-chief Frank Newport appears to be on a campaign against poll aggregation. In a recent essay (‘Polling, Likely Voters, and the Law of the Commons‘, Gallup.com) he writes:
It’s much easier, cheaper, and mostly less risky to focus on aggregating and analyzing others’ polls. Organizations that traditionally go to the expense and effort to conduct individual polls could, in theory, decide to put their efforts into aggregation and statistical analyses of other people’s polls in the next election cycle and cut out their own polling. If many organizations make this seemingly rational decision, we could quickly be in a situation in which there are fewer and fewer polls left to aggregate and put into statistical models. Many individual rational decisions could result in a loss for the collective interest of those interested in public opinion.
Oh, please. Considering Gallup’s performance in estimating the national race, this could be interpreted as a defensive move by this year’s equivalent of the Literary Digest poll (‘Landon by a landslide,’ George Mason University).
Newport misses the positive value that we bring to his activity. It is too bad, because what we do can ultimately increase the relevance of his organization. Here’s why.
First and foremost, poll aggregation is not like other forms of news aggregation. News aggregators like the Huffington Post basically recycle stories. Those of us who examine groups of polls add value for both reader and pollster:
- For the reader, we cut through the noise. Individual polls contain two kinds of error arising from (a) inherent limitations of sampling, and (b) systematic errors made by individual pollsters. By using robust statistical tools, we reduce and cancel these errors to obtain a far superior result.
- For the pollster, we offer a benchmark for future performance. Paul Starr pointed out to me recently that a likely reason for the improvement in political opinion polls since the 1930′s has been the fact that polls are easily compared with election results. Until recently, this comparison was limited by statistical sampling error. Now, aggregators can grade a pollster’s accuracy to within 1 percentage point.
Newport does not fully acknowledge the second point. Regarding his own organization’s performance he writes:
The “gap” difference was….well within the statistical margin of error and underscore[s] the accuracy of random sampling today.
Actually, no. Thanks to aggregation, we can say with great specificity that Gallup’s national October numbers (Romney ahead by 2% to 6%) were systematically off by 4-8% from the true margin at the time, Obama +2.0% (“A final unskewing,” November 12th). No wonder he doesn’t like us. Underneath the bluster and threat, I believe that Newport’s real problem is Gallup’s own poor showing.
The red curve indicates Gallup’s data, plotted with 1-sigma error bars. The black curve is my best estimate of the true Obama-Romney margin, based on all available national surveys (“A final unskewing,” Nov. 12). The last data point is off by about 4.0% (2 sigma), and the three data points before that are off by more.
In fairness, it was not only Gallup whose national numbers were off. National polls as a group were biased by an average of 2.4 +/- 0.4% toward Mitt Romney. State polls were a superior source of information: our Popular Vote Meta-Margin did far better than the national Romney-vs.-Obama average in predicting the national vote.
Newport is correct that poll aggregation does devalue the news value of any single poll. That is the point of the activity. It’s why I started doing it in 2004. I was driven to distraction by breathless stories on single polls. This year, I almost blew a blood vessel when I saw the entire front page of USA Today dedicated to a single Gallup poll that was an outlier. Let’s face it, news organizations love outliers. If aggregation kills that kind of story in the future, our entire nation wins.
Despite Newport’s complaints, my own view for the future of his field is bright. Aggregators like PEC bring focus to their activity and add a new dimension. However, now they have to be nimble. They can’t get stuck in a rut reporting only topline numbers. That low-hanging fruit will soon be gone.
But there are many ways they can improve their game. For example:
Focus on crosstabs. Much of the richness in polls comes in the details: knowing that young voters tilt Democratic, or that many Romney supporters would rather identify themselves as independents than as Republicans. Those details carry endless news interest.
Watch one another. This year PPP missed a big story by failing to report a sudden plunge in support for Todd Akin (R) in his Missouri Senate race to unseat Claire McCaskill (D), after his “legitimate rape” comment (“Akin sheds 8 points overnight to near-tie,” August 12). They could have caught that story if they had been willing to compare their own results with other pollsters. This pridefulness should stop.
Develop new products. The most interesting polls this season were products like the RAND longitudinal survey, in which the same respondents were surveyed repeatedly. Gallup itself had some fascinating results from their tracking poll, which showed an overnight jump in President Obama’s job approval rating after Michelle Obama’s speech (“Michelle Obama, the Great Persuader,” September 9th). Let a hundred flowers bloom!
Learn from the crowd – but don’t be afraid to go the other way. Pollsters can learn not only from each other’s biases, but also from where the polls are fielded. In the five weeks after October 1, Pollster.com showed 97 national polls. That is complete overkill. Some of that effort would have been better spent on downticket Senate or House races – or even nonswing states, which received so little love this year.
Go local! Many pollsters conduct national surveys as loss leaders. They do it for the media exposure. But there is plenty of publicity to be had in other races. If pollsters swooped in there, it would garner them publicity – and even help slow the decline in local journalism. That would be a win for pollsters – and add much-needed diversity to our media culture.
And here we come to an irony: the Gallup organization is rich in expertise, and is a leader in adding value in interesting ways. If they continue to do that – and stop complaining about the new kids on the block – they can maintain their relevance. I wish them every success.
Update: To emphasize a point above, for organizations like Gallup, I am under the impression that most of their income comes from other polling, not what aggregators analyze. Therefore I believe that the availability of public polling data is not under threat. However, their brand did take an undeniable hit this year.