Princeton Election Consortium

A first draft of electoral history. Since 2004

All estimates point toward HRC>50% probability. What determines the exact number?

November 6th, 2016, 11:31pm by Sam Wang


Three sets of data point in the same direction:

  • The state poll-based Meta-Margin is Clinton +2.6%.
  • National polls give a median of Clinton +3.0 +/- 0.9% (10 polls with a start date of November 1st or later).
  • Early voting patterns approximately match 2012, a year when the popular vote was Obama +3.9%.

Based on this evidence, if Hillary Clinton does not win on Tuesday it will be a giant surprise.

There’s been buzz about the Princeton Election Consortium’s win probability for Clinton, which for some time has been in the 98-99% range. Tonight let me walk everyone through how we arrive at this level of confidence. tl;dr: With a more conservative assumption (see discussion) the PEC approach gives a probability of more like 95%. I will also give a caveat on how it is difficult to estimate win probabilities above 90% – and why fine adjustments at this level might not matter for my goals in running this site.

An obvious contrast with PEC’s calculation is the FiveThirtyEight win probability, which has been in the 60-70% range. As a prominent outlier this season, FiveThirtyEight has come under fire for their lack of certainty. Its founder, Nate Silver, has fired back.

Let me start by pointing out that FiveThirtyEight and the Princeton Election Consortium have different goals. One site has the goal of being correct in an academic sense, i.e. mulling over many alternatives and discussing them. The other site is driven by these factors, but in addition by monetary and resource considerations. However, which is which? It’s opposite to what you may think.

Several weeks ago I visited a major investment company to talk about election forecasting. Many people there had strong backgrounds in math, computer science, and physics. They were highly engaged in the Princeton Election Consortium’s math and were full of questions. I suddenly realized that we did the same thing: estimate the probability of real-world events, and find ways to beat the “market.”

In the case of PEC, the “market” is conventional wisdom about whether a race is in doubt. If a race is a certain win or a certain loss, it is pointless to put in money and effort, assuming that the rest of the market is in the game. On the other hand, if a race is in doubt, then it may be moved by a little extra push. Think of it as “math for activism.” This point of view heavily influences my calculations.

>>>

Now think about the FiveThirtyEight approach. I don’t want to get into too much detail. Although they discuss their model a lot, to my knowledge they have not revealed the dozens of parameters that go into the model, nor have they released their code. Even if they did, it is easy to make errors in evaluating someone else’s model. Recall Nate Silver’s errors in his critique of PEC in 2014. So let me just make a few general comments. I am open to correction.

Their roots are in detail-oriented activities such as fantasy baseball. They score individual pollsters, and they want to predict things like individual-state vote shares. Achieving these goals requires building a model with lots of parameters, and running regressions and other statistical procedures to estimate those parameters. However, every parameter has an uncertainty attached to it. When all those parameters get put together to estimate the overall outcome, the resulting total carries greater risk of accumulating uncertainty that is hard to keep under control.

For this reason, the Huffington Post claim that FiveThirtyEight is biased toward Trump is probably wrong. It’s not that they like Trump – it’s that they are biased away from the frontrunner, whoever that is at any given moment. And this year, the frontrunner happens to be Hillary Clinton.

And then there is the question of why the FiveThirtyEight forecast has been so volatile. This may have to do with their use of national polls to compensate for the slowness of state polls to arrive. Because state opinion only correlates partially with national opinion, there is a risk of overcorrection. Think of it as oversteering a boat or a car.

In addition to all this, it has been said that the amount of available polling has decreased or may be of lower quality. That would increase uncertainty as well.

With all that prelude (whew!), let me explain how the Princeton Election Consortium achieves such a high level of confidence.

>>>

We start by generating the sharpest possible snapshot, based on state polls. State polls are more accurate than national polls, which at this late date are a source of unnecessary uncertainty.

For each state, my code calculates a median and its standard error, which together give a probability. This is done for each of 56 contests: the 50 states, the District of Columbia, and five Congressional districts that have a special rule. Then a compounding procedure is used to calculate the exact distribution of all possibilities, from 0 to 538 electoral votes, without need for simulation. The median of that is the snapshot of where conditions appear to be today.

Note that in 2008 and 2012, this type of snapshot gave the electoral vote count very accurately – closer than FiveThirtyEight in 2008, and missing only Florida in 2012.

This approach has multiple advantages, not least of which is that it automatically sorts out uncorrelated and correlated changes between states. As the snapshot changes from day to day, unrelated fluctuations between states (such as random sampling error) get averaged out. At the same time, if a change is correlated among states, the whole snapshot moves.

The snapshot gets converted to a Meta-Margin, which is defined as how far all polls would have to move, in the same direction, to create a perfect toss-up. The Meta-Margin is great because it has units that we can all understand: a percentage lead. At the moment, the Meta-Margin is Clinton +2.6%.

Now, if we want to know what the statistical properties of the Meta-Margin are, we can just follow it over time:

This variation over time automatically tells us the effects of correlated error among all states. Uncorrelated error is cancelled by aggregation under the assumption of independence; what is left is correlated variation. The problem is solved without any regression. Hooray!

As I have noted, the Presidential Meta-Margin tends to move on a one-to-one basis with the Senate Meta-Margin and the generic House ballot. That suggests that downticket effects are powerful, and also that the snapshot calculation does a good job of separating correlated from uncorrelated change.

To turn the Meta-Margin into a win probability, the final step is to estimate how far the results of tomorrow’s election will be from today’s Meta-Margin. As a community, pollsters have pretty good judgment, but their average estimate of who will vote may be off a little. In past years, the snapshot has been quite good, ending up within a few electoral votes of the final outcome. That is equivalent to an uncertainty of less than one percentage point.

Here is a table for the last few Presidential races:

“Actual threshold margin” is estimated using voting thresholds for the several states that were just enough to put the winner over the top. Note that these errors are not symmetric: there seems to be a tendency for the winner to overperform his final Meta-Margin. So it is not clear that Meta-Margin errors are symmetrically distributed. That means we can’t just use the average overperformance – that might be an overestimate of the amount of error that would work against the front-runner. However, the sample is too small to be sure about this.

Another way to estimate Meta-Margin error is to use Senate polls. Here’s a chart from 2014 (look at the Presidential column only):

The directional median indicates a bonus that favors one party over the other. Over the last six Presidential election cycles, the absolute value of the error (i.e. ignoring whether it favors Democrats or Republicans) is 0.6%, really small.

To turn the Meta-Margin into a hard probability, I had to estimate the likely error on the Meta-Margin. For the home stretch, the likely-error fomula in my code assumed an Election Eve error of 0.8% 0.5% on average, following a t-distribution (parameter=3 d.f.). The t-distribution is a way of allowing for “longer-tail” outcomes than the usual bell-shaped curve.

So…there’s only one parameter to estimate. Again, hooray! However, estimating it was an exercise in judgment, to put it mildly. Here are some examples of how the win probability would be affected by various assumptions about final error:

As you can see, a less aggressive approach to estimating the home-stretch error would have given a Clinton win probability of 91-93%. That is about as low as the PEC approach could ever plausibly get.

I have also included the prediction if polls are assumed to be off by 5% in either direction on average. It is at this point that we finally get to a win probability that is as uncertain as the FiveThirtyEight approach. However, a 5% across-the-board error in state polls, going against the front-runner, has no precedent in data that I can see.

Bottom line: Using the Princeton Election Consortium’s methods, a less aggressive assumption (sigma=1.1%) leads to a Clinton win probability of 95%.

>>>

As I said at the top, my motivation in doing these calculations is to help readers allocate their activism properly. Whether the Presidential win probability is 91% or 99%, it is basically settled. Therefore it is a more worthwhile proposition to work in Senate or House campaigns. Get on over to IN/MO/NC/NH/WI, or find a good House district using the District Finder tool in the left sidebar.

Update: see this exchange, which suggests that a more reasonable uncertainty in the Meta-Margin is 1.1%, giving a Clinton win probability of 95%. However, to state the obvious, I am not going to change anything in the calculation at this point. That would not be kosher!

Tags: 2016 Election · President · Senate

165 Comments so far ↓

  • mediaglyphic

    Prof Wang,
    Thanks for this, its nice to see it all summarized once more.

    Very Puzzling to hear Nate Cohn, say there is no “empirical” evidence for your approach on Zakaria yesterday. Also someone needs to explain them how to pronounce your name. Zakaria may be a Yalee but he has no excuse!

    • Charles Bouldin

      A way to sort out some of this discrepancy is to look back previous Presidential election predictions and compare the -actual- accuracy of the forecast with what the stated error bars would have predicted.

      For instance, in 2012, Nate called 50/50 states correctly and in 2008 49/50. His own stated error predictions would have estimated 3-5 states “wrong”, so his error bars are clearly very generous, probably far too generous.

      Whether is is 583, PEC or Upshot, that’s the way to establish an accurate error assessment, by comparing actual accuracy with predicted accuracy.

      Of course, we have few data points (3 elections) so there’s not a lot of data, but this is at least a logical starting point.

    • Sam Wang

      Cohn: That must be personal. He should know better than that.

    • Ed Wittens Cat

      I watched the segment– Enten and Cohn both seemed pretty desperate to defend 538 model.
      PEC model is transparent– unlike Silver’s magical secret sauce.

    • AySz88

      To be clearer, Cohn said said something like there isn’t [enough] empirical evidence for 99%+ certainty, not that the approach doesn’t use empirical evidence.

  • JL

    Prof. Wang:
    Fan comments are useless; nevertheless, I am a huge fan. It’s a little depressing that your low-profile, meticulous, transparent approach, confident in its own expertise and uninterested in manipulating the emotions of its readers, is so rare. Thank you for providing reliable election predictions as well as a refuge from all the media noise!

  • Tom Gavin

    Excuse me if this is a dumb question, but does the 2.6 shift mean that that much movement would be required to shift the popular vote from Clinton to Trump or the Elctoral vote. (Dumb and winning is k with me )(:-)

    • BrianTH

      At a MM of 0, each would have an equal chance of winning the electoral vote.

    • Arun

      One way to interpret this.. how likely is it that all states (or even just the group of swing states FL, NV, NH, and NC) will move 2.6% consistently towards Trump in two days? Answer looking at the MM graph – very unlikely.

  • Alan Neff

    I’ve done a lot of survey research but generally relied on very simple correlations to tease out meaningful relationships among the data. Your presentation of your complex model with so much transparency is admirable, and you’re not afflicted with a commercially driven inclination – albeit understandable – to generate click-bait. As one of the other commenters said above, this is good science – in execution and openness. Thanks.

  • Joel

    Sam, you mentioned Silver’s roots in fantasy baseball. Did you ever come across Tom Tango’s MARCEL? It was a straightforward player projection system that seems to make not-so-subtle mockery of another all-capitalized, acronymed projection system (you might guess who developed that one).

    http://tangotiger.net/marcel/

  • ned rollo

    An excellent site! If only I understood anything that’s being discussed, it would surely be even better (for me.)

    I must say that such predictive endeavors are a serious challenge and a natural magnet for serious mathematicians. I appreciate your good work; thank you.

  • Arun

    Thanks for the note at the end which clarifies how the uncertainty in the meta margin influences the final win probability. I’ll make the point again that while the calculation applies the meta margin equally to all states, a few close large states will have the biggest impact, most will not show a meaningful change at all. I do think with the rather limited data set of elections, the more conservative error model makes more sense. Besides, a win probability of 91-95% is in slightly better alignment with other forecasting models, such as the NY Times Upshot model.

  • JSchmoe

    I appreciate this write up.

    I’d had my own qualms about the “>99%” figure (such figures are better left to ‘the sun will come up tomorrow’ type predictions), so I’d already applied my answer to Drew Linzer’s informal twitter poll

    https://twitter.com/DrewLinzer/status/795365771987730432

    and decided that “>90%” is the right way to interpret any figures that fall in that realm.

    And I’d noticed that the PEC website’s brief “About Us” page does have the following quote, which get’s to Sam’s statement about why fine tuning this figure is not important to PEC’s objective:

    “… originally motivated by the fact that in a close race, one can make the biggest difference by donating at the margin, where probabilities for success are 20-80%. ”

    Again … this was a helpful clarification, so thanks.

  • Ken S

    Is this your reply to Silver’s comments on ABC that any models predicting a 99% chance of a Clinton victory doesn’t pass “the commonsense” test?

    PEC is the only one at 99%.

    Silver’s comments were a not so veiled jab at you and your work.

    • Sam Wang

      When they go low, we go high.

    • Jake Ingram

      Sam, if the neuroscience thingy doesn’t work out for you, you just might make it as a stand-up.

    • Alex P

      What’s funny is a lot of his state probabilities don’t pass the “commonsense test” I want to know how he got simulations that have Trump winning a New England state outside of Maine.

    • Vasyl

      HuffPost model is also up there (just recently it was 98.8%)

    • Roger

      538s fat tails gives unrealistic output. For example, 538′s distribution has recently projected a 7% chance that Clinton gets less than 200 electoral votes as well as a 3% chance she gets over 400. Both those situations have almost 0 chance of occurring, therefore the 538 model’s output should be telling them that something is wrong.

    • AySz88

      I’ve been seeing criticism of PEC from quite a few sources, not just Silver, so I wouldn’t chalk this up as a direct response to just the one comment.

  • Scott

    The standard deviation of the meta-margin since July 1st is 1.1%, coincidence?

  • Runner

    In my mind, the best way to translate the very sound math, statistics and science used by PEC is as follows, using a legal term understood by most Americans:

    PEC finds Beyond a Reasonable Doubt that Hillary Clinton will be the next President.

  • Amitabh Lath

    To explore the space of different uncertainty estimates have you tried PDFs other than t-distributions? I know a gaussian is probably too narrow but it would be nice to see it done for comparison reasons. Maybe a log-normal or Landau or Poisson distribution?

    • Stephen Hartley

      not sure Poisson would give you anything Gaussian wouldn’t given their similarities

  • Vasyl

    >It’s not that they like Trump – it’s that they are biased away from the frontrunner,
    538 model is leaning towards an uninformed mean (i.e. the best guess of a chance of meeting a dinosaur if you know nothing about them is indeed 50-50)
    And the fact Trump is both uninformed and mean is purely circumstantial and coincidental.

  • Oscar

    I understand the contrast with 538, but why does PEC remain so far at odds with Betfair, which has Clinton at 82%? Are you sticking to what you said in http://election.princeton.edu/2012/09/24/predictions-a-brief-comparison-with-intrade/, that “Bidders overestimate both current and future uncertainty, even in cases like a Presidential re-election race where movement is small”?

  • Angie

    Thank you so much for your hard work. As someone else noted, your site is “an island of sanity.” In the last two months especially I have found great comfort in your work.

  • Seth Gordon

    Why three degrees of freedom (as opposed to, say, 49)?

    • Sam Wang

      That parameter sets the length of the tail. It’s not used as actual degrees of freedom. d.f.=49 is indistinguishable from a normal distribution. If you want a longer tail, then use a smaller d.f. For example, if d.f.=1, then the result is within 1 sigma of the mean only 50% of the time.

    • anonymous

      I see the MM moved down .1. And NH,NC are white. Will you have a final states pick tonight, Sam?

  • Dawolf

    If the evidence is that the election eve error should actually be 1.1%, perhaps for the next election that is the calibration which should be used (adjusted for this election as well of course).

    Great work as always, bit to me a 91% probability *feels* closer to what the polls are showing: Obama overperformed his national polls by about 3% in 2012 and a similar error in the other direction should make this incredibly tight or a Trump win so 99% just seems too confident to me.

    Having said which, early voting results are great for Clinton generally, so I do think she’s going to win comfortably.

  • A New Jersey Farmer

    Thanks Sam. The news media loves a story that will keep readers guessing every step of the way. It’s great to come here and see actual data and analysis.

  • Sridhar

    I know some forum members were worried about the SEPTA strike in Philadelphia and the election turnout.Good news on this front overnight.A deal was reached to end the strike and services should gradually resume from Monday AM and should be fully operational by dawn on Election day.

    • ravilyn sanders

      S@Sridhar // Nov 7, 2016 at 5:03 am: Thank you Sridhar. Good news keeps coming. I was the one who started neurotically worrying about it here and in kos, eventually he-who-shall-not-be-named-but-is-biased-against-the-${front-runner} actually made a posting about it. Not good for my blood pressure.

      Then Comey came through. Now you.

    • Matt McIrvin

      That’s great news!

      I think the single strongest argument for early voting is that it allows people whose Election Day plans might be affected by uncertain events (such as metropolitan commuters looking down the barrel of a transit strike) to plan ahead and ensure their votes. But Pennsylvania doesn’t have it (yet).

    • bks

      Matt, the single strongest argument for early voting is that when you get a phone call from someone about the election, or someone comes to the door about the election, you can simply say, “I already voted.” No discussion, no psychodrama.

  • Iain Roberts

    Great post, and great blog. I think there are a couple of things widely misunderstood about a high-confidence prediction.

    1) Like Asimov’s psychohistory, these models assume initiative on the part of voters and campaigns. If Clinton supporters work hard to ensure a victory, her chances are >95% (depending on model assumptions). If they stay home, her chances plunge. The model does not devalue the importance or difficulty of their work.

    2) There’s a distinction between risk and return on investment. If you believe President Trump would be a disaster, then even a 5% chance of Trump victory is scary, for the same reasons as a 5% chance of contracting a terrible disease. But moving Trump’s win probability in either direction, to (say) 4% or 6%, would be extremely costly if it could be done at all. There are more competitive races where additional resources (money, volunteer time) would be much more useful. Again, the model does not detract from the scariness of a possible Trump presidency.

    • Kearny

      I’m sure I’m not the only one, but I will continue to donate all of my resources to HRC campaign (instead of other, closer races) until it’s officially over. A 5% chance of Trump is too much for me.

  • Ed Wittens Cat

    Just beautifully done Dr. Wang.
    PEC is an island of sanity and math in this dadaist election season, full of melted clocks and burning giraffes.

  • Will Nemirow

    Thanks for the explanation!

    One critique: you are citing old early voting numbers. Things have moved towards Trump relative to Romney in almost every state except Nevada.

  • Davey

    Thanks…very interesting read. Also, thank you for putting to math what seemed sort of intuitive…that this was a pretty stable and relatively boring election, despite the very odd nature of the campaigns and coverage.

  • AJS

    In line with Sam’s perspective, what I tend to do is just categorise the prediction – i.e. 40-60% is a toss up, 60-75% is a good chance, >%75 is pretty certain, >90% is about as certain as you can be of anything in life (except…). OK maybe >95% as you get info close to the event. But does that last step make a meaningful difference in psychology (stress/relaxation level).

    Potential systematic error can still occur – but these have to add up to a net effect (often can cancel each other out to some extent). Such as unexpected turnout in a specific group – early voting patterns provide some insights on that.

  • Anthony Shanks

    Probably one of my favorite clips of this election season:

    https://www.youtube.com/watch?v=NjilH4XY18s

    Where the CNN interviewer tries the “but what bout the trump supporters not being captured by the polls” argument and Sam dismisses it by saying

    “oh no no no”

  • Ruth Rothschild

    Thanks for this post, Sam, and for explaining your method. Much appreciated.

  • Yuval Peres

    Hi Sam,
    Where can we find the actual Math representing how you compound from state win probabilities to an overall win probability?

    That is where correlation between states enters and some compounding procedures could represent hidden independence assumptions.
    Thanks,
    Yuval

  • Reginald

    This is one of the most interesting articles that I’ve read on this site, almost up there with the enlightening posts on how and why the race has been so stable.

    PEC has always been the most transparent service of this kind. It’s unsurprising Dr Wang’s methods be favored by wall street quants and other finance types.

    I’m probably preaching to the converted here, but I’ll offer my 2 cents on why I don’t use 538

    0. The analysis. Communication about numbers is just as important as the numbers themselves. I used to read 538 but all of Silver’s articles basically boil down to the same thing: provocative title, reiteration of what’s already posted in their model, and 4-5 inconclusive paragraphs of filler. I even read his book a few years ago and it was more of the same.

    1. High uncertainty aversion. In many professions, particularly data driven ones, you are forced to make evidence based predictions when there is less data than they wished you wish you had. You don’t have this data because your subject is fairly obscure, so you do your best and you hopefully make your boss happy. But an election at the presidential level is the most highly polled event in the United States. Many highly skilled statisticians give you new information several times *a day*. How on earth does a professional analyzing this come back with anything less than 90% certainty a few days before an election?

    2. Obvious click bait trolling. The best example of this is Silver attempted to quantify the level of panicking each party should be doing just last week on a scale from 1-10.

    There are many reasons to use Dr Wang’s site, but listing them here would seem too fan boyish. But it’s obvious that all that I criticized 538 for cannot be said of PEC.

    • Charles

      “How on earth does a professional analyzing this come back with anything less than 90% certainty a few days before an election?”

      I’m baffled by this, too. How can he have Hillary at +2.6% nationally, similar to what other models have, similar to the 2-7% lead she’s had literally the entire race, and still project a 1-in-3 chance that she loses? It would be absolutely unprecedented for her to lose that lead at the death.

      “Obvious click bait trolling.”

      Silver makes it very easy to dismiss his model in this regard. He is the only one predicting a greater than 20% chance of a Trump victory, the only one who has been repeatedly sounding alarm bells for months, the only one writing empty, panicked articles about how Trump was briefly leading in one of his three models, etc. I wasn’t born yesterday. It’s not a coincidence that this is all coming from the man who 1. utterly failed to predict Trump in the primaries, and 2. is the only one of these stats guys (that I know of) whose livelihood is based on traffic generated almost exclusively by his presidential model.

      I, for one, won’t be surprised if 538′s model flips a swing state or two at the last minute and gets the electoral vote exactly right.

    • gumnaam

      Regarding the first point, a high uncertainty can exist even in the case of abundance of evidence, e.g. when the gap between the two candidates is small. It is not the case here, but it happened in 2000 and 2004.

  • Jeff W

    Seems to be an attack on Twitter going on. Can’t get Sam’s Twitter feed or anyone else at the moment. Hope this isn’t a test run for something on election day. Maybe it is just where I am at…

    • Eric Walker

      I am in inland eastern Washington State and am seeing the same thing. Cannot even Ping it.

    • TW

      Yes, twitter was down but it seems to back up now. We should not rule out the possibility of this being another DDoS drill. Not trying increase the already high levels of anxiety…but anything is possible. Hollywood concepts have usually come true in the past.

    • Frank

      “As goes Twitter, so goes the Nation.”

      -General George Washington, at the Battle of Friendster

  • Ryan Casey

    The title of your post implies that you’ll explain whether and why a win probability of 99% is reasonable. Then, after very in-depth explanations of the model (some of which I will admit to not fully understanding), you show that even the small change of a 0.8% assumed error to a 1.5% brings the win probability from 99% to 91%. Now, I fully agree that statistically, that’s a relatively insignificant difference. But given the (apparent) subjective nature of choosing 0.8% instead of something like 1% or 1.5%–numbers that seem to have a better historical justification–and the fact that this makes such a sizable difference in your win probability, I’m pretty curious what your intuition is and why, and whether you’ve used this same approach in 2012, 2008, and 2004. Thanks!

    • Sam Wang

      Didn’t do it in 2004 or 2008. In 2012 I used other parameters, hadn’t worked through this reasoning in anything like the current form.

      In summer 2016 I had to figure this parameter out. I am unable to recount the thought process that went into estimating the parameter at the time. I did not anticipate that it would be controversial. But if we really want to do it more rigorously, here we go:

      Take all the Senate and Presidential errors I listed, and convert them into the direction of favoring the Presidential winner. That gives [-2.1 -1.0 -0.2 0.2 0.2 1.1 1.2 1.3 2.3 3.4]. The median of that gives an estimated SD of 1.1%, with skewness 0.01 and kurtosis 2.54 (not bad, since a Gaussian has a kurtosis of 3). That would suggest a final error of 1.1% instead of 0.8%. Then, if I keep the t-distribution that I have now (that’s why I cared about the kurtosis), the current Meta-Margin of Clinton +2.6% gives a win probability of 95%.

      To state the obvious, I am not going to make any changes at this point for 2016.

    • Emanuele Ripamonti

      I think Ryan nailed down my objection: it is very difficult to get the meta-margin uncertainty. While 0.8% (or 1.1%) are perfectly justified in terms of getting the best estimate, it is entirely possible that the actual value is higher. For example, if you look at the 4 data points for presidential race, one of them is 2.3%, and you could naively say that 1 time in 4 the meta-margin is off by ~2%, which gives a (gaussian?) sigma in the 1.5-2% ballpark. Sam’s approach is obviously much better, but data can’t rule out such an uncertainty (OTOH, I’d say they DO rule out anything above 2%).

    • BrianTH

      I definitely agree that Sam should NOT change his estimates on the fly right now, but I think it is great he is publishing his alternative calculations for different assumptions, for much the same reason it is helpful to publish his +2 maps: it helps people to better understand the sensitivity of the model to that assumption, and then they can adjust their own sense of the probabilities as they feel is warranted.

    • Eli

      That type of empirical estimation would seem reasonable to me. And for sure, in most elections nobody would care, e.g. if G.W.Bush’s win probability is 1% or 5%. But when the candidate is spectacularly unqualified, it scales the expected value. You got unlucky on that.

    • MartinM

      I think you’ve dropped a minus sign, Sam; you’ve got 1992 down as +1.1%, should be -1.1%, no? That makes the median 0.2%, with skewness of 0.3, and kurtosis of 2.3, approximately.

  • David

    Sam, I’ve been a fan of the blog for years. The last part of this post is frustrating. In the most atypical election in living memory, you’re choosing an average error that is well below each of the last four races. You say that a 5% error “has no precedent in data that I can see,” which at your degree of confidence is more absence of evidence than it is evidence of absence.

    There is an enormous difference between an 90% chance and a >99% chance. Unpredictable events like natural disasters, terrorism, or poorly timed transit strikes can be safely ignored at 90%, but they have to be considered at >99%. Projecting a >99% chance is like risking at least tens of thousands of dollars to win $100. I can’t imagine anyone making that kind of a bet on an event that hasn’t even happened ten times.

    Your probability of victory strikes me as being of a lower quality than the rest of your output.
    It isn’t falsifiable and it lacks empirical or methodological rigor. Why not rest on the strength of your EV, margin, and Senate predictions — and give a range like 85-99% instead of >99?

    • Sam Wang

      Is your argument that the weirdness of the race should make me estimate a parameter differently? How do I know which way an emotional race will drive the error? Not buying your argument. Also, error bars on probabilities are tough for people to parse.

      But yes, I agree that I might change something in the future. My current inclination would be be to use a lower d.f. for the tails, i.e. on Election Eve end up with tcdf(MM/0.8,1) or tcdf(MM/0.8,2) instead of tcdf(MM/0.8,3). That’s about it.

    • David Roher

      Thanks for the reply! I’d say that the weirdness of the race increases the uncertainty around polling. Imagine if this were 1860 or 1920 — I’d expect more error if pollsters had to account for a 4-way race or a huge number of voters with no history, respectively.

      I’m not saying to throw out the book, especially given that this doesn’t actually look like an unusual election in many ways. But it, among other things, would stop me from going >99.

    • alurin

      The surface content of the election is atypical, but there’s no evidence that it is atypical from a polling perspective. Polls worked well in the primaries; specifically, they were good at nailing T—p’s performance. There’s no data-driven reason to doubt that polls are accurately capturing voting intentions, just emotion-driven reasons.

  • Scott Matheson

    Looks like the meta-margin yellow band is narrowing steadily while the ev estimator yellow band is not. Anyone know why?

    • A DC Wonk

      My guess is that it’s simply a function of time — that is: the amount of time between the date of prediction and the date of the election.

      In other words, the yellow band would look larger today if the election were scheduled for next week. But since it’s tomorrow, there’s just a small range of movement we can expect in only 24 hours.

  • Michael

    Sam. We’ve reached a breaking point: How confident can be be of a Clinton victory, which is essential for continuation of the republic as we know it. Let’s stop dancing around.

  • MarkS

    OK, I’ve embarrassed myself here before by not understanding some basic point, but it seems to me that the methodology of going from EVs to meta-margin to win probability (by estimating an error on the meta-margin) is not internally consistent, at least deep in the tails where we are now. The current EV histogram appears to give Trump at least a 2% chance of winning today. Random-drift should increase that probability, not decrease it. Yet the election-day random-drift win probability for Trump is quoted as <1%.

    What am I missing?

    • Sam Wang

      The histogram’s not used to estimate win probability. Think of the width of that histogram as showing the standard deviation (spread of all possibilities) as opposed to the standard error (how well we know where the midpoint of the histogram is).

    • Matt McIrvin

      I don’t get this response: we don’t really care about the standard error, do we, except as a way of broadening the distribution of all possibilities?

      I think what a lot of people are thinking is that the distribution for a “random-drift” prediction should be something like your histogram for uncorrelated errors, convoluted with some distribution for the error in the center of the histogram. That can only make the spread wider, not narrower, so it ought to increase the probability of an improbable tail event.

      Just eyeballing it, you’ve currently got NC and NH as tossups and Florida and Nevada light blue.

      Say we assign a probability of 50% to the tossups and 75% to the light blues, we get about a 1.6% chance of an incredibly narrow Trump win. That’s not too far from the size of your histogram’s left tail and I suspect it’s close to the effects currently dominating your histogram.

      That depends heavily on the assumption of no correlation. If there’s a chance that some of that error is really a systematic polling miss with a symmetric distribution, that means the real distribution of possibilities is wider, and Trump’s chances should only go up from there, right? They shouldn’t go down.

    • Matt McIrvin

      …Now, that said, if all these excited reports about Nevada are true and Nevada is a lock, then I think 99% is actually perfectly reasonable–with maybe a 6-10% chance of a nail-biter in which we have to wait for Nevada to know who won. But PEC’s model explicitly does not incorporate any of that.

    • Matt McIrvin

      (And, also, I recall being burned by excited anecdotes about early voting in previous cycles, so I’m still a little skeptical, though the NV numbers seem so overwhelming that it makes me less skeptical than usual. I’m skeptical about the relevance of the early-voting reports in other states.)

    • MarkS

      I agree with Matt McIrvin’s comments. Random-drift should broaden the histogram while leaving its median in place. That would raise Trump’s win probability.

      But as I understand Sam’s response, if we processed the histogram through the random-drift algorithm with zero time for drifting, the Trump win probability would go down instead of up.

      This does not make sense to me. My conclusion is that I should look at the median EV graph with the gray 95% CL band, and do my own probability estimates from that.

  • Colin

    Fantastic article. I was just wondering today what your confidence is in your analysis and how you reach your high confidence.

  • David E. Davidson

    Isn’t the relevant-to-activism metric not event probability, P, but P*C, where C is event cost? And I’ll take Hillary with only 49 Senators, over Trump with 51, every day.

    p.s. Also, why use the Senate margin (smaller error = ~0.6%) rather than the presidential margin (~1.45%)? This seems akin to the “turn-out” factor from 2004.

    • Sam Wang

      Maybe…but at least it is not a first-order bias toward either candidate, as was the case in 2004. Now it is a second-order bias toward certainty.

      Truly we are in the advanced stages of this kind of activity.

  • A

    Such a wonderful, clear and concise explanation of the model.

    I consider this art as much as science, and it truly soothes the soul.

    Cheesy, I know–but true.

  • SK

    Do you account for any higher uncertainty from the larger percentage of undecided/third-party voters this year?

  • Nick B

    Thank you for sharing the details of your model. I found it fascinating that even the most conservative approach here still leads to a probability of a Clinton win at ~91%.

    I think Nate’s 538 model is hedging too much on general uncertainty regarding this race. I find it interesting that he thinks the race is too uncertain (high % of undecideds and third party share), while you think the uncertainty is at ‘normal’ levels (at least, compared to previous elections). I can’t help to think Nate’s ’2% chance of Trump winning the GOP primary’ discussion has forced him to take a more conservative approach regarding his model this election.

    • Stephen Hartley

      Nate is a pundit now. He likes going on TV and his money is made by generating traffic. He has a vested interest in the election having a certain narrative. In addition, he likes to use his ‘subject matter expertise’ to fuel discussion of the election (see the months of ‘Trump can’t win’) and that leads him to bake his ‘insights’ into his model.

      His work isn’t bad but there are forces on it that create a push towards a certain kind of answer – the fact that his model has moved in that direction (‘the race is close OMG! Volatility!) suggests to me he hasn’t been able to keep his science walled off from those influences.

      Sam is a not-for-profit venture – his motivations will be driven towards being as accurate as possible with no other influences towards profit etc. That’s helpful in crafting an objective view I think.

  • Patrick E.

    The “allocating activism” argument makes me feel better about staying home in Indiana to do canvassing this weekend when I was sorely tempted to volunteer in neighboring Ohio instead since the Presidential race is closer there. To me defeating Trump is by far the most important thing we need to do in this election, but the Senate and gubernatorial races in Indiana are very close. Plus we have local races to consider. I was torn, but I ended up helping the weary Democratic party activists close to home.

  • Ketan

    This is a complex discussion, so I hope I’m not oversimplifying but, let’s talk Florida.

    538 gives FL a 52% chance for Republicans. PEC gives 31% (and HuffPo gives 11%). That’s a huge spread for a well studied state.

    The difference is about actually about 2%. 538 thinks Trump is ahead by 0.5% and PEC thinks he is behind by 1.5%.

    I think this is almost entirely due to the “window” i.e. how PEC/538 discount older polls. (Recent polls in FL are mostly blue/tied, but the ones from last week of October were mostly red.)

    Sam, would a different window change your numbers for Florida? (PEC is essentially 7 days for close states.) How strongly do you feel your window is sized right?

    • Sam Wang

      The time window doesn’t matter. I am pretty sure what does matter is that the FiveThirtyEight approach generates a ton of uncertainty at the single-state level as part of the process of predicting those states.

      Note that single-state November win probabilities don’t even come out of my approach at all. In fact, I had to concoct a parallel calculation so that The Upshot could carry those: I applied random-drift to the single-state snapshots. Such a step is probably baked into the middle of the FiveThirtyEight process, which adds 56 doses (states+DC+districts) of uncertainty.

    • AySz88

      I wonder, wouldn’t the window choice change the high-frequency response of the meta-margin? So the meta-margin’s walk over time wouldn’t capture uncertainty that occurs on the order of a few days. (Though, any such movements would have to be transient and self-reverting, or otherwise they’d be accounted for in the random walk, right?)

      So, some concrete examples of things that behave that way are…. weather (meh), day-of-week effects (meh), and the coming and going of “news cycles”. I guess the “news cycles” thing is somewhat plausible – salience of the issue-of-the-day might last on the order of a few days, and decays quickly – and I noticed that 538 discusses recent news *a lot* in their discussions. But how often does the “news cycle” change (i.e. away from the typical blabber) in the last few days of a Presidential election?

    • Ketan

      I think that the focus on the difference in confidence (99 vs 65) is misplaced; and it could well be true that their low confidence is justified for their model! Not arguing that.

      But! You are predicting 225 trump EV (median). They are predicting 243 (average). At 225, it takes multiple battleground states to surprise for Trump to win, but at 243, it only takes two (e.g. NV + NH).

      I believe that looking at differences between the sites brings insights and helps find errors. Another poster on this site pointed out that the MS margin is incorrect (+3 should be +7-10); and it is more likely a bug i.e. not what you designed.

      I don’t know that the polls are signaling a clear margin for NH, FL, NV, and NC. I don’t know why that is. I would have thought the more polls there are, the more averages, medians, and more complex filtering would converge.

      Thanks for this post; it really helped me think about thinks more clearly.

  • Michael

    You are a gentleman and a scholar.

  • Meta Analyst

    Thank you for this post, I had been looking forward to this in order to better understand the differences. One question: can you explain in more detail how you model the correlations between the state polling errors. You mention that it is somehow automatically handled, but it is not clear. I suspect that a large part of the difference in the two models (yours vs. 538) is how state correlations are modeled.

    • Sam Wang

      I don’t model the correlations at all. It is done automatically by the procedure, and then the variation in the Meta-Margin over time gives an empirical measure of the correlated error. There isn’t much more to say unless you are ready for the actual math.

    • Debbie Leung

      “There isn’t much more to say unless you are ready for the actual math.”
      If you have time, please do explain the actual math how the variation in the Meta-Margin over time gives an empirical measure of the correlated error. This issue has generated a lot of discussion between myself and my friends (all with math/physics background). Thanks in advance.

    • Paul

      I’m curious about this too. Can I see the math?

    • Sam Wang

      Imagine a simple scenario in which state i’s margin is

      M_i + national_swing + noise, where M_i is a constant, national_swing varies over time, and noise is uncorrelated across states.

      The all-state Meta-Margin at any moment will tend to give a result of

      M_i + national_swing + noise/sqrt(N_eff)

      where N_eff is the number of states N, reduced by a factor having to do with the fact that only closely-contested states contribute noise. N_eff should be something like N*mean(abs(erfc(M_i)-0.5)), where erfc() is an appropriately scaled cumulative error function.

    • Debbie Leung

      Many thanks for taking the time to explain this! Thanks for making the code and other aspects of the calculation open. I’m really looking forward to your publication on this.

  • Ezra

    The most recent polls seems to be increasingly favorable to Democrats. She has recovered somewhat, and today’s FBI announcement might also move the needle a tiny little. Maybe Sam won’t be eating the bug after all. (Hoping for the best: President Hillary Clinton with >=50 seats in the Senate. I don’t care less whether Sam eats the bug…. Fingers crossed.)

  • David D.

    Thanks so much for this post.

    Can you say a little bit more about why you chose an Election night error of 0.8%? The chart you provide comparing poll-based metamargin to actual margin says past error in the last four elections was ~1.0%, 1.3%, 1.2%, and 2.3%, or an average of 1.45.

    Based on that past performance, doesn’t an error model of 1.5% seem more reasonable?

    • Sam Wang

      Parameter estimation is just something this kind of work requires. In this post was some of the data that went into the estimate.

      Whereof one cannot speak, thereof one must be silent. -Wittgenstein

    • Tony Asdourian

      Thanks Sam for a great post.

      David, those errors you refer to are only relative to the “several states that put the winner over the top”. I don’t know how off the Meta-Margin was on average in those years if you include all the states. Conceivably, the average MM error could be lower?

    • Eli

      I appreciate your openness, but is there any more that can be unpacked out of your intuition? Do you have an intuition that the “winner gets a bonus” pattern is real so the positive winner margins over-reflect an unobservable?

      In all four Presidential elections, the actual abs error was more than 0.8%. Are you blending those with the Senate data, or what other information brings that parameter down?

  • Arun

    Sam,

    Thanks for the detailed explanation. Aren’t most states “safe”, in the sense that moving the margin in them by fairly large amounts would not change the win probability significantly? In that sense, isn’t the meta margin controlled mostly by a few states: FL, NC, and NV in this election? So it isn’t so much that all polls would have to move, just the ones in a couple of the larger states, to get the median EV and win probability to move.

  • Alice Gibb

    Thanks so much for the clear explanation of your straightforward and highly accurate approach.

  • Shawn Huckaby

    Thank you so much for this. The fact that you openly share every aspect of your model and process is the very definition of good science.

    Bravo!

  • Tom sinclair

    Mr Wang,
    Thanks for your analysis.

  • Dave Kliman

    I think although past performance is not necessarily an indicator of future predictions, it’s reasonable to point out you’ve guessed the electoral vote count most accurately–closer than Nate at least once, than most anyone else, since you’ve been doing this.

    • Mark Z

      ‘Guessed’ is such an ugly word. How about ‘predicted’ or ‘calculated’? :-)

    • GC

      Thanks Sam for running the site again this year. You’ve been gracious in answering comments and explaining how this works.

      What’s exciting to me is seeing how PEC holds up come November 9th, since the model can actually be tested in a real way (i.e. we know the code). This makes it more valuable than certain other sites that rely on black boxes and (IMHO) too much hand-waving.

    • Dave Kliman

      Yes. Predicted.