Princeton Election Consortium

A first draft of electoral history. Since 2004

About the Meta-Analysis (FAQ)

(Updated on 19 September 2012 by Sam Wang)

The right-hand sidebar features a meta-analysis directed at the question of who would win the Electoral College in an election held today. Meta-analysis provides more objectivity and precision than looking at one or a few polls, and in the case of election prediction gives a highly accurate current snapshot. In 2004, the median decided-voter calculation captured the exact final outcome. In 2008, the final-week decided-voter calculation was within 1 electoral vote.

Calculations are based on recent available state polls, which are used to estimate the probability of a Democratic/Republican win, state by state. These are then used to calculate the probability distribution of electoral votes corresponding to all 2.3 quadrillion possible combinations. For a popular article about this calculation, read this article and the follow-up.

Is this Meta-Analysis a prediction of what will happen on Election Day?

The basic analysis does not; it is a snapshot of conditions today. Between now and Election Day, think of the Meta-Analysis as a precise snapshot of where the race stands at any given time. In late October the Meta-Analysis should come quite close to the actual outcome.

Starting in 2012, this site also provides a prediction (see essay 1 essay 2) based on the current year’s polls and the amount of variation observed in similar past races. This is a true prediction for November. It has the specific advantage of not relying on poorly-justified assumptions such as econometric conditions. It relies only on polls, which are the only direct measure of opinion. The approach taken in both popular and political science models introduces more noise than signal, as discussed in this essay.

What’s different about this analysis in 2012 compared with 2008?

The main difference is the addition of a prediction for Election Day as described above.

What was different about this analysis in 2008 compared with 2004?

In 2008, three major changes were made.

First, the Meta-Analysis relies entirely on the well-established principle that the median of multiple state polls is an excellent predictor of actual voter behavior. On Election Eve 2004, a calculation based on this principle made a correct prediction of the electoral vote outcome. Additional assumptions were unnecessary and unwarranted. In 2008 the calculation is kept simple – and therefore reliable.

Second, the calculation is automated to allow tracking of trends over time. This allows the Meta-Analysis to be used to identify changes in voter sentiment as seen through the lens of actual electoral mechanisms.

Third, instead of focusing on battleground states, we are tracking all 50 states and the District of Columbia.

In the Meta-Analysis, how can you possibly go through 2.3 quadrillion possibilities? Wouldn’t that take forever?

The Meta-Analysis doesn’t actually calculate the probability of every combination of states one at a time. At a rate of going through a million combinations per second, that process would take over 71 years. Yet repeated simulation is exactly what other sites do – though they only do thousands of simulations, not quadrillions. Such a laborious approach means that they can only approximate the expectation based on a set of win probabilities.

Instead, the Meta-Analysis uses an overlooked method to calculate the probability of getting an exact number of electoral votes, covering all ways of reaching that number given the individual state win probabilities. This is a much easier problem – it can be solved in less than a second. Here is a simple example.

Imagine that there are just two states. State 1 has EV1 electoral votes and your candidate has a probability P1 of winning that state; in state 2, EV2 electoral votes and a probability P2. Assume that EV1 and EV2 are not equal. Then the possible outcomes have the following probabilities:

EV1+EV2 electoral votes (i.e. winning both): P1 * P2. EV1 electoral votes: P1 * (1-P2). EV2 electoral votes: (1-P1) * P2. No electoral votes: (1-P1) * (1 – P2).

In general, the probability distribution for all possible outcomes is given by the coefficients of the polynomial

((1 – P1) + P1 * x^EV1) * ((1 – P2) + P2 * x^EV2) * … * ((1 – P51) + P51 * x^EV51)

where 1…51 represent the 50 states and the District of Columbia. This polynomial can be calculated in a fraction of a second.

Why don’t other projection sites use your approach?

Three reasons.

First, the Meta-Analysis is unlike, say, fantasy baseball, where a lot of enjoyment comes from thinking about individual scenarios. We take no interest in specific scenarios; we want the median outcome that takes into account all possibilities. This gives the most precise possible answer, but it lends itself poorly to color commentary.

Second, the treatment is somewhat mathematical. Hobbyists at other sites may not have the expertise to take the polynomial shortcut, which is made possible by the fact that the Electoral College follows a relatively simple system in which EV are added up. Certain aspects of the Meta-Analysis are original and may someday be published.

Third, a properly done calculation reduces noise. This, in turn, reduces variation – and opportunities for commentary. Most media organizations want more commentary, not less.

What polls do you use? When do you exclude a poll?

We use all available state polls, with a preference for likely-voter polls over registered-voter polls when both are released. We do not exclude any poll.

For the current snapshot, the rule for a given state is to use the last 3 polls, or 1 week’s worth of polls, whichever is greater. A poll’s date is defined as the middle date on which it took place. In some cases 4 polls are used if the oldest have the same date. At present, the same pollster can be used more than once for a given state. From these inputs, a median and estimated standard error of the median are used to calculate a win probability using the t-distribution.

What do you think of my favorite/despised pollster?

Appropriate aggregation methods remove the need to dissect individual polls. For example, median-based statistics correct for outliers. Also, human beings engage in motivated reasoning, and look more critically and closely at polls with which they disagree. Avoiding this bias leads to more accurate results. For these reasons, commenting on individual pollsters is usually not productive.

Indeed, good aggregation has the potential to free up mental (and media) space for information about more important topics than individual polls.

When do updates occur?

Every day at midnight, 8:00am, noon, 5:00pm, and 8:00pm.

Your estimate fluctuates less than other sites. Why is that?

It’s the power of meta-analysis. Even though individual polls may vary, aggregating multiple polls per state reduces uncertainty. Calculating the entire distribution of outcomes does an even better job. As a result, the Electoral Vote estimator on this site typically doesn’t move much. This is in fact the point of the analysis – to get past the vagaries of day-to-day poll reports. Rigorous meta-analysis is sometimes less exciting to watch than a site that varies every day, but in our view it’s the best way to present polling data.

Your calculation could be used to give a win probability. Why don’t you show this?

The uncertainty at any given moment is small enough that the results of an election held today would not be in much doubt. At any given moment, a current-poll win probability is typically greater than 95% for either candidate. Because this quantity is the wrong one to focus on, it is not given.

The greater uncertainty comes from changes that may happen over time between now and Election Day. This can be used to derive a true November-win probability. Some challenges in estimation are discussed here.

From day to day, a very useful quantity is the Popular Meta-Margin, defined as how much swing would have to take place to generate a near-exact electoral tie. The Popular Meta-Margin is equivalent to the two-candidate difference found in most single polls. It has many uses because it tells you where the race stands in units of voters. Errors in polling such as cell phone user undersampling and third party candidates are in these units, and therefore can be compared directly to the Meta-Margin.

For those who still insist upon getting a probability from the Meta-Analysis, it can be computed by pasting current histogram data into a spreadsheet and summing rows 270 through 538.

Why should I believe the Meta-Analysis? In 2004, didn’t it predict a narrow Kerry victory?

Actually, the method was fine, but its inventor, Prof. Sam Wang, made an error. In the closing weeks of the campaign, he assumed that undecided voters would vote against the incumbent, a tendency that had been noticed in previous pre-election polls. Compensating for the “incumbent rule” had the effect of putting a thumb on the scales, lightly – but unmistakably – biasing the outcome.

Leaving out this assumption, the prediction in 2004 was exactly correct: Bush 286 EV, Kerry 252 EV. In retrospect, it’s clear that the incumbent rule is subjective and cannot be relied upon. You can read about the confirmation of the prediction in the Wall Street Journal (pre-election story here). A second confirmation came in 2006, when, using a related but simpler method Sam expected the odds of a Democratic takeover of the US Senate were 50-50, a higher chance than predicted by either pundits or electronic markets. Indeed, that event did end up occurring. Finally, in 2008, and Presidential and Congressional calculations did extremely well.

Overall, the analysis is kept as simple as possible as a means of avoiding unintended bias. Both data and the code for doing the calculations are freely available. That way, anyone can check the results. Everything was open in 2004 as well; readers provided lots of useful feedback, such as this exchange.

State polls are done less often than national polls. Does that introduce a delay into your analysis?

Yes. As of early August this delay is about two weeks in key states. The delay will diminish dramatically as the campaign season progresses. A correction based on national polls is possible, but adds considerable uncertainty to the estimate.

What is the Popular Meta-Margin?

The Popular Meta-Margin is the amount of opinion swing that is needed to bring the Median Electoral Vote Estimator to a tie. It helps you think about how far ahead one candidate really is. For example, if you think support for your candidate is understated by 1%, this can overcome an unfavorable Meta-Margin of less than 1%. If you think that between now and Election Day, 1% of voters will switch from the other candidate to your dude, this is a swing of 2% and can compensate for a Meta-Margin of 2%.

What if I think that polls are biased against my candidate? Do you provide a tool for me to see how a bias changes things?

One tool is the Popular Meta-Margin (see above). Another tool is the map in the right-hand column, which comes in flavors that show single-state probabilities with a 2% swing toward either candidate.

What are jerseyvotes? And can you explain the “Power Of Your Vote” table?

Jerseyvotes, invented at this site in 2004, are a way to measure the power of individual votes to sway the election. Conceptually, jerseyvotes are distantly related to the Banzhaf Power Index, but normalized to the power of one individual. If you have ten times as much influence over the national win probability as a voter in New Jersey, your vote is worth 10 jerseyvotes. Sadly for the hosts of this site, one jerseyvote is not worth very much.

The Voter Influence table in the right-hand sidebar displays information about the ten states currently with the highest jerseyvotes, plus New Jersey for comparison. The jerseyvote statistic for each state is listed in the “Power” column, and they are normalized so that the most powerful state has power equal to 100. (Originally, this power statistics was normalized so that NJ voters had power equal to 1, hence the term “Jerseyvotes”.) The current polling margin, as determined by the meta-analysis, is also displayed for each state. For example, if the meta-analysis indicates that NJ is currently polling 50% for Obama and 44% for McCain, then NJ’s “Margin” column would read “Obama +6%.”

In your future prediction / current snapshot, the probability is very different from the InTrade price. Why is that?

It is wrong to interpret InTrade prices as true probabilities. Those prices reflect what a number of bidders think to be the win probability. InTrade bidders tend to be underconfident in evaluating polling data. On Election Eve, even a 5 +/- 1 point lead for a candidate is often insufficient to drive a share price above $0.90. However, it is true that the candidate with an InTrade price above $0.50 is usually the leading candidate. The issue is analyzed further here.

I wrote a comment but it does not appear. What happened?

The site is moderated to shape the discussion. Our audience includes a wide range of numerically-oriented professionals and academics. Many are also partisans of various stripes.

Here are just a few examples of comments that can get deleted or delayed:

  • falsehoods
  • abuse
  • perseveration on one data point such as a single poll or pollster
  • opinions given without evidence, especially quantitative evidence
  • long comments
  • questions that are already answered somewhere in the left sidebar
  • diversions from the point at hand that are not interesting

Of course, all of these statements can find a home at unmoderated sites, of which there are many.

Several other sites emphasize the possibility that states tend to vary together, so that if one poll is off, then others will be off in the same direction. Why don’t you include that in your model?

Assumptions should only be added to a model if they make a difference in the outcome.

For a snapshot, adding covariance makes very little difference. For instance, let us make the assumption that all polls move together between the last day of polling and Election Day by a random small amount (up to 1%, say), and the random amount is unbiased. In this situation the median EV estimator does not change by a measurable amount. The uncertainty in the final outcome, as measured by 95% confidence interval, gets a little wider. But that’s it. Thus, since covariance has no substantive effect, it is left out.

For a prediction, the answer is more nuanced. In this situation, the change between polling day and Election Day could be considerable. Now, the way that the change is modeled affects the shape of the distribution. However, the median is still the same. In this case a simple and effective way to vary long-term change is to covary all states together by a random amount. This is at the core of the prediction, a feature that was introduced starting in 2012.

This site has discussed the subject of non-independence here and, most recently, here.

What do you do with third-party candidates? Can such a candidate shift the outcome?

We take whatever the pollster gives us as the margin between the two leading candidates. Third-party candidates tend to fade in the finish in a system with two dominant parties. Some pollsters give third-party results and some don’t. This kind of detail might help, especially for analyzing local/state races where third-party local candidates run strong, such as Maine.

 

69 Comments

69 Comments so far ↓

  • Robin

    I just found out about your site. Thanks. One favor. I have very little experience in statistics and would love to understand in regular people’s terms how you do the staistical measurement.
    Can you help?

  • Jim

    Do you have a Figure comparing your 1/3 day ahead predictive interval with your estimate of the state of the race 1/3 days later? I’d be curious to see this to evaluate your suggestion that you might be overestimating the 95% Credible Interval. I apologize if this is somewhere on the website — I didn’t see it immediately.

  • Guy

    In your FAQs, the probability of EV1 electoral votes is a function of P2, as well as P1. For a real-world example does this mean that: if [1] EV1 is, say, Massachusetts and EV2 is Texas; and [2] P1 is the Democrat probability in MA and P2 is the Republican probability in TX — then the geographically/demographically/etc. disconnected TX Republicans would has some influence on EV1? Just a random thought: To keep a State’s preference within that State, have you ever considered a Poisson distribution (variance=mean) where lambda-1 is related to P1, and then crank over k1=0-100; keeping the other 50 distinct areas constant, then change P2 to P2+0.01, repeating over k1=0-100, and so on?

    • Kyle

      Guy: That is 100% correct? If the candidate wins EV1 and loses EV2, he receives EV1 electoral votes. So the odds that he wins exactly EV1 electoral votes is (directly!) influenced by his performance in EV2.

    • Googa Mooga

      In the formula Sam gives, P1 and P2 are completely independent.

  • Bill

    Great site Sam–been following you for several weeks now–I thought at one time you had posted a paper detailing the EV meta-analysis but now I can’t find it.

  • CS

    Has the model been tested with 2000 polling data? This year’s election is going to be quite close like the 2000 election and it would increase one’s confidence in the model if it can predict the outcome of the 2000 election based on the state polling data of that time. That would be a better indicator of the model’s robustness than the predictions of the 2004 and 2008 elections as these were easier to predict.

  • charles carey

    In FL and OH long wait times are being reported. That may lead to beingessentially disenfranchized. Is there any way that your model can reflect that outcome?

  • Paul C

    What happened today? Nov 5 meta margin is down 0.6% from Nov. 4!

  • Neminem

    Hi can someone briefly expand my understanding of the polynomial, perhaps with a simple example? Struggling at the moment. What is “x” in the polynomial? What does the product of the terms in the polynomial respresent?

  • Matt

    This is such a great site! Is there anyway I could get the code for computing the jerseyvotes index for WA? I like to give my mom a hard time about our vote not counting in this election (regardless, it’s still a duty I perform), and it would be great to show her some hard data supporting my assumptions.

    Good luck on your big night tomorrow!

  • Mark Tardiff

    I’m keen to read the article and follow-up for your calcuations, hot linked in the second paragraph:

    For a popular article about this calculation, read this article and the follow-up.

    The links are broken. Could you send me the articles?

    Thanks.

  • 538 Refugee

    Be interesting to see how the model holds up to the final numbers. Tough luck on the Florida coin flip. I told you Gravis saying tied meant Romney would lose it. My son told me that Gravis has NO credibility left and they make up their data anyhow. I know there was a link here about them but never saw a follow up after someone claimed to have ‘busted’ them.

    • 538 Refugee

      Sorry, must have hit something wrong and got this posted in the wrong thread.

  • Art Brown

    Aha! Finally found your “About the Meta-Analysis” /methods/ page. It was well worth the effort, but I wonder if you could add a link to it here or on the front page (apologies if it’s already there). [ … it was there all the time, under the blogroll. Sorry…]

  • Brian

    What are the chances that the Kansas Supreme Court will drop the Democratic candidate off the ticket for the Pat Roberts Senate seat race? That decision may ultimately decide the Senate balance, and does the 5 point lead for independent Orman factor the Dem. candidate being on or off the Nov. ballot?

  • Susu

    Rachel clued me in. I now am watching this site every day. Thanks!

Leave a Comment