I get asked a lot about why the various aggregators differ from one another. After all, we all start with the same polling information. Today I will give a general sketch of how and why we differ – and what I view as the strengths and weaknesses. I’ll restrict myself to organizations that I am more familiar with. It’s Sunday, so not that much math.
The two big distinctions to make among the various approaches are:
(1) Do we use polls only, or do we bring in predictive indicators (e.g. economic variables)? There are many “pure aggregators”: Electoral-vote.com, Electionprojection.com, Pollster.com,and RealClearPolitics. If what you want is polls only, any of these sites are good. The Princeton Election Consortium uses polls only, and gets them from Pollster.com. A few online prognosticators bring in economic variables too, most prominently FiveThirtyEight and Votamatic.
(2) Do we take a snapshot of current conditions only, or do we attempt a future prediction? I’ll review three organizations that have been making predictions all season: FiveThirtyEight, Votamatic, and the Princeton Election Consortium.
I’m leaving out political scientists who use predictors only, such as Alan Abramowitz, Ray Fair, and the University of Colorado people. As I’ve written, I categorize their models as tools to test ideas about how voting preferences are shaped. All of them do well at “post-dicting” past events. They might get the next election right, but if they don’t…so what? Make another model. This activity is research, with emphasis on the “search.” There’s nothing at all wrong with it. But it’s most useful before the election season starts. In the storm, you want the person with the instruments, not the person with the almanac.
I’ll go through the various models, going gradually farther away from polling data.
Polling data: Electoral-vote.com, RealClearPolitics, Pollster.com. All of these sites present polls with relatively little additional processing. Electoral-vote.com, by Andrew Tanenbaum, exemplifies the first wave of aggregators. He gives simple tabulations of state polls, with the electoral vote total determined by the most recent poll. To reduce poll-to-poll fluctuation, RealClearPolitics adds simple averaging. They also leave out partisan pollsters. Pollster.com uses more sophisticated smoothing methods, and has remarkably good user tools to allow the construction of customized graphs. In all cases, the electoral vote count is a simple total, assigning each state one possible outcome. This is the mode of the distribution.
Pro: Gives a quick look at the race with a minimum of filtering. Averaging gives a sharper picture of any individual state.
Con 1: The total electoral estimate still fluctuates because all states are reduced to a single combination of outcomes, which usually corresponds to the single highest point on the histograms here at PEC or at FiveThirtyEight.
Con 2: The use of averaging allows an extreme outlier poll to pull the average disproportionately in one direction. This can be an issue where polls are sparse. Smoothing is not optimal for revealing sudden shifts such as the effect of Debate #1.
Snapshot plus state-polls-only based prediction: Princeton Election Consortium. Like the sites listed above, we also offer a snapshot, shown in the topline of this website. The history of the snapshot is plotted in the right column. However, our methods wring considerably more information from the data. Our fundamental output is two core numbers: the EV Estimator and the Popular-Vote Meta-Margin. They are very high-resolution measurements at a single point in time. Think of them as an electoral snapshot or an electoral “thermometer.”
How we get these numbers requires a little explanation. Briefly, in each state we do more than ask “who’s ahead?” Instead we calculate a win probability from the median of recent polls, an outlier-rejecting approach. Using a simple math trick, we then take these 51 probabilities to calculate the exact distribution of all outcomes (2.3 quadrillion). The middle of the distribution is the EV Estimator.
The Meta-Margin takes advantage of the fact that the EV Estimator calculation is very speedy (takes much less than 1 second to run). It relies on a core tool, the bias variable b. It is easy to shift all polls over by a fixed amount b. There are several reasons to care about this: (i) polls in different states tend to move together – correlated variation; and (ii) polls may all be biased by some amount, which can also be simulated by varying b. The Meta-Margin is defined as what value of b would lead the electoral college to be a perfect tossup. It’s just like a margin, which is why it’s in units like Obama +2.6%.
Finally, we also use b as a way to game out future scenarios, and make a prediction. If we think polls can move by up to 1% in the future, then we can add up all the possibilities from b=-1% to b=+1%. The red and yellow strike zones (which are almost gone as of today) are calculated this way. Based on past elections, we can estimate what b might be.
The reason I am going off about b is that it is my way of thinking of contrasts between the Princeton Election Consortium with FiveThirtyEight. In some sense, my assumption that b has a narrow range accounts for why the two sites give different re-elect probabilities for President Obama.
Pros: Makes near-maximal use of existing state polls, whose track record using PEC’s methods is excellent. Uses medians to reject outliers. Converts Electoral College mechanisms to a Popular Vote margin, an intuitive quantity. The low noise allows accurate identification of swings in the race.
Cons: Doesn’t use national polls. Doesn’t correct for house effects. Assumes that state polls are, as a group, unbiased (though this does have support from 2004-2008).
Hybrid model: FiveThirtyEight. In addition to state polls, Nate Silver uses other variables – national polls and econometric indicators – to infer a likely election result. He used this approach to predict winners in the 2008 Democratic primaries, in that case including demographics and more. He was able to fill in some missing-data problems.
For his Presidential model, he takes several approaches. One type of variable is econometric indicators, which informed his calculation earlier in the season. His current calculation uses national and state polls, with fuzz factors to account for the possibility that these polls could contain systematic errors.
I’ll be brief without getting too far into the weeds. He takes a very conservative approach to estimating win probabilities, in the sense that he builds in ways that effectively reduce the certainty of any particular outcome. In addition to being conservative about single-state probabilities, it appears that he puts a lot of credence into the possibility that national and state polls could be off by a substantial amount. Recently he said that this cautious approach accounted for much of the 16% probability of Romney winning the election.
Let me express this idea in terms of my bias variable b. The 16-percent idea is approximately equivalent to saying that there might be overall (i.e. all pollsters combined) systematic errors in national and state polls that could drive b as high as 5% in either direction (a 95% confidence interval), given today’s Meta-Margin. However, as I pointed out the other day, b doesn’t affect state outcomes very much in most cases, since these races, even in swing states, are usually determined by a larger margin. Also, based on my analysis in 2004-2008 (where data are abundant), b for state polls is smaller on average, 1-2%. It’s larger for national polls.
Eventually, I believe that a suitable way to measure b in past elections is to perform aggregation separately on state polls and national polls, then compare actual national popular vote and EV with national-poll margin and the Meta-Margin that I have defined. This might be hard to do for earlier elections, where polling was sparser.
Pros: Takes into account national polling data; corrects for individual pollster biases; takes a conservative approach to the uncertainties.
Cons: Likely to overcount uncertainties (look at the error bars). The use of national polls may reduce accuracy of state-level Electoral College outcome. Uses econometric variables even after direct measurements (polls) are available. Time resolution not as good as a pure-state-polls approach.
Polls with a predictive prior: Votamatic. Drew Linzer’s project is a fresh approach to the problem of combining econometric variables. In his case, he uses an econometric model for long-term prediction to set up a “prior” expectation of how the race will unfold, then uses this to guide the interpretation of polling data.
As you can see, the model fluctuates hardly at all. It seems to really have an affinity for Obama 332 EV, Romney 206. At some level this is a good feature: if a prediction is accurate, it shouldn’t vary much. However, I am a bit concerned because this suggests that the prior is drawn very restrictively. In other words, it is set to ignore or shape incoming polling data. The predictive value of such a model depends a lot on the validity of the prior.
My own inclination for a model like this is to use it to fill in “missing data” problems. Many states are underpolled, such as Texas or Vermont. A strong prior can give us expectations for what would happen there. Although those outcomes are not in doubt, the vote-share is not known. This would be a good test. Another example of a missing-data problem is Senate or House races, the latter being a significant prediction prize.
Pros: Very stable prediction; keeps prior and polling data separate; high level of analytical rigor.
Cons: Dependent on the validity of the prior; doesn’t reveal much about the dynamics of the race.
As you can see, these models each have their own uses. To my own taste, I’d use them as follows.
Seeing polling data for one’s own envelope analysis: Electoral-vote.com, Pollster.com, or RCP.
Sharp snapshot of the race as it evolves over time: Princeton Election Consortium.
Least-confident “conservative” prediction with all conceivable rational caveats: FiveThirtyEight.
Stable, model-driven view: Votamatic.
I should also note all of these sites also have their own flavor of commentary. Drew Linzer has done fascinating recent work getting into examining individual pollsters and looking for “skew” or “bias.” Electoral-Vote.com gives a very good daily survey of the scene at all levels, and highlights polls of particular interest. And of course FiveThirtyEight’s Nate Silver made his bones in part by the data-driven play-by-play commentary that made him famous in 2008.
It is certainly possible that I have not put my finger on key differences between these approaches. I imagine many of you are big fans of the other sites, and can offer alternative interpretations or corrections in comments.