Princeton Election Consortium

A first draft of electoral history. Since 2004

All estimates point toward HRC>50% probability. What determines the exact number?

November 6th, 2016, 11:31pm by Sam Wang


Three sets of data point in the same direction:

  • The state poll-based Meta-Margin is Clinton +2.6%.
  • National polls give a median of Clinton +3.0 +/- 0.9% (10 polls with a start date of November 1st or later).
  • Early voting patterns approximately match 2012, a year when the popular vote was Obama +3.9%.

Based on this evidence, if Hillary Clinton does not win on Tuesday it will be a giant surprise.

There’s been buzz about the Princeton Election Consortium’s win probability for Clinton, which for some time has been in the 98-99% range. Tonight let me walk everyone through how we arrive at this level of confidence. tl;dr: With a more conservative assumption (see discussion) the PEC approach gives a probability of more like 95%. I will also give a caveat on how it is difficult to estimate win probabilities above 90% – and why fine adjustments at this level might not matter for my goals in running this site.

An obvious contrast with PEC’s calculation is the FiveThirtyEight win probability, which has been in the 60-70% range. As a prominent outlier this season, FiveThirtyEight has come under fire for their lack of certainty. Its founder, Nate Silver, has fired back.

Let me start by pointing out that FiveThirtyEight and the Princeton Election Consortium have different goals. One site has the goal of being correct in an academic sense, i.e. mulling over many alternatives and discussing them. The other site is driven by these factors, but in addition by monetary and resource considerations. However, which is which? It’s opposite to what you may think.

Several weeks ago I visited a major investment company to talk about election forecasting. Many people there had strong backgrounds in math, computer science, and physics. They were highly engaged in the Princeton Election Consortium’s math and were full of questions. I suddenly realized that we did the same thing: estimate the probability of real-world events, and find ways to beat the “market.”

In the case of PEC, the “market” is conventional wisdom about whether a race is in doubt. If a race is a certain win or a certain loss, it is pointless to put in money and effort, assuming that the rest of the market is in the game. On the other hand, if a race is in doubt, then it may be moved by a little extra push. Think of it as “math for activism.” This point of view heavily influences my calculations.

>>>

Now think about the FiveThirtyEight approach. I don’t want to get into too much detail. Although they discuss their model a lot, to my knowledge they have not revealed the dozens of parameters that go into the model, nor have they released their code. Even if they did, it is easy to make errors in evaluating someone else’s model. Recall Nate Silver’s errors in his critique of PEC in 2014. So let me just make a few general comments. I am open to correction.

Their roots are in detail-oriented activities such as fantasy baseball. They score individual pollsters, and they want to predict things like individual-state vote shares. Achieving these goals requires building a model with lots of parameters, and running regressions and other statistical procedures to estimate those parameters. However, every parameter has an uncertainty attached to it. When all those parameters get put together to estimate the overall outcome, the resulting total carries greater risk of accumulating uncertainty that is hard to keep under control.

For this reason, the Huffington Post claim that FiveThirtyEight is biased toward Trump is probably wrong. It’s not that they like Trump – it’s that they are biased away from the frontrunner, whoever that is at any given moment. And this year, the frontrunner happens to be Hillary Clinton.

And then there is the question of why the FiveThirtyEight forecast has been so volatile. This may have to do with their use of national polls to compensate for the slowness of state polls to arrive. Because state opinion only correlates partially with national opinion, there is a risk of overcorrection. Think of it as oversteering a boat or a car.

In addition to all this, it has been said that the amount of available polling has decreased or may be of lower quality. That would increase uncertainty as well.

With all that prelude (whew!), let me explain how the Princeton Election Consortium achieves such a high level of confidence.

>>>

We start by generating the sharpest possible snapshot, based on state polls. State polls are more accurate than national polls, which at this late date are a source of unnecessary uncertainty.

For each state, my code calculates a median and its standard error, which together give a probability. This is done for each of 56 contests: the 50 states, the District of Columbia, and five Congressional districts that have a special rule. Then a compounding procedure is used to calculate the exact distribution of all possibilities, from 0 to 538 electoral votes, without need for simulation. The median of that is the snapshot of where conditions appear to be today.

Note that in 2008 and 2012, this type of snapshot gave the electoral vote count very accurately – closer than FiveThirtyEight in 2008, and missing only Florida in 2012.

This approach has multiple advantages, not least of which is that it automatically sorts out uncorrelated and correlated changes between states. As the snapshot changes from day to day, unrelated fluctuations between states (such as random sampling error) get averaged out. At the same time, if a change is correlated among states, the whole snapshot moves.

The snapshot gets converted to a Meta-Margin, which is defined as how far all polls would have to move, in the same direction, to create a perfect toss-up. The Meta-Margin is great because it has units that we can all understand: a percentage lead. At the moment, the Meta-Margin is Clinton +2.6%.

Now, if we want to know what the statistical properties of the Meta-Margin are, we can just follow it over time:

This variation over time automatically tells us the effects of correlated error among all states. Uncorrelated error is cancelled by aggregation under the assumption of independence; what is left is correlated variation. The problem is solved without any regression. Hooray!

As I have noted, the Presidential Meta-Margin tends to move on a one-to-one basis with the Senate Meta-Margin and the generic House ballot. That suggests that downticket effects are powerful, and also that the snapshot calculation does a good job of separating correlated from uncorrelated change.

To turn the Meta-Margin into a win probability, the final step is to estimate how far the results of tomorrow’s election will be from today’s Meta-Margin. As a community, pollsters have pretty good judgment, but their average estimate of who will vote may be off a little. In past years, the snapshot has been quite good, ending up within a few electoral votes of the final outcome. That is equivalent to an uncertainty of less than one percentage point.

Here is a table for the last few Presidential races:

“Actual threshold margin” is estimated using voting thresholds for the several states that were just enough to put the winner over the top. Note that these errors are not symmetric: there seems to be a tendency for the winner to overperform his final Meta-Margin. So it is not clear that Meta-Margin errors are symmetrically distributed. That means we can’t just use the average overperformance – that might be an overestimate of the amount of error that would work against the front-runner. However, the sample is too small to be sure about this.

Another way to estimate Meta-Margin error is to use Senate polls. Here’s a chart from 2014 (look at the Presidential column only):

The directional median indicates a bonus that favors one party over the other. Over the last six Presidential election cycles, the absolute value of the error (i.e. ignoring whether it favors Democrats or Republicans) is 0.6%, really small.

To turn the Meta-Margin into a hard probability, I had to estimate the likely error on the Meta-Margin. For the home stretch, the likely-error fomula in my code assumed an Election Eve error of 0.8% 0.5% on average, following a t-distribution (parameter=3 d.f.). The t-distribution is a way of allowing for “longer-tail” outcomes than the usual bell-shaped curve.

So…there’s only one parameter to estimate. Again, hooray! However, estimating it was an exercise in judgment, to put it mildly. Here are some examples of how the win probability would be affected by various assumptions about final error:

As you can see, a less aggressive approach to estimating the home-stretch error would have given a Clinton win probability of 91-93%. That is about as low as the PEC approach could ever plausibly get.

I have also included the prediction if polls are assumed to be off by 5% in either direction on average. It is at this point that we finally get to a win probability that is as uncertain as the FiveThirtyEight approach. However, a 5% across-the-board error in state polls, going against the front-runner, has no precedent in data that I can see.

Bottom line: Using the Princeton Election Consortium’s methods, a less aggressive assumption (sigma=1.1%) leads to a Clinton win probability of 95%.

>>>

As I said at the top, my motivation in doing these calculations is to help readers allocate their activism properly. Whether the Presidential win probability is 91% or 99%, it is basically settled. Therefore it is a more worthwhile proposition to work in Senate or House campaigns. Get on over to IN/MO/NC/NH/WI, or find a good House district using the District Finder tool in the left sidebar.

Update: see this exchange, which suggests that a more reasonable uncertainty in the Meta-Margin is 1.1%, giving a Clinton win probability of 95%. However, to state the obvious, I am not going to change anything in the calculation at this point. That would not be kosher!

Tags: 2016 Election · President · Senate

165 Comments so far ↓

  • Tim Scherr

    Sam,

    I have watched your site re elections since 2000. I remember that you called 2004 for Kerry, and then seemed shocked when Bush won. At that time you tried to determine what went wrong. It appears you still don’t have a good model, as predictions of Trump-Clinton were completely wrong. I have detected a strong bias in favor of democrats over the years. This seems to have permeated your model. It’s clear you need a new model for the polling, the margins of error are clearly much greater than your fundamental assumptions. Perhaps you need someone that is a republican to create a similar model, and then determine why they are different. The truth is in there somewhere.

    Best,

    Tim Scherr

  • Bill Sawyer

    Thanks for a very informative discussion of your analysis. I particularly appreciate your transparency regarding your approach to your model design. Thanks also for making clear the difference between your method and that of 538.

  • A

    As someone who played a mobile game where you draw different rarity characters with a 9% and a 1% chance, there is a very big difference between events occuring with 91% and 99% probability.

    I’d be mildly surprised if a 9% chance event happens, while I’d be pretty surprised when a 1% chance event happens. To get a feel for the difference, its like once every week and a half vs once every season of the year. Or if you’re talking about something that occurs every 4 years, it is (on average) once every 44 years (say twice in your lifetime) vs once every 400 years (we’re 232 years into this presidential elections thing). That is a huge difference.

    There is also a huge world of difference between 1% and 0.1% events, but it is unreasonable for anyone to be able to confidently differentiate between those. From your 95% estimate, I feel like saying >99% is too confident… (my initial reaction to 538 is that 70% is an underestimate…) Maybe what you mean by 91% and 99% are similar is that a prognostigator will have a difficult time telling those apart (similar to how it is unreasonable for anyone to tell the difference between 99% and 99.9%)

    • Brendan

      From the perspective of an individual human, there may not be that much difference between something that happens once every 44 years and once ever 400 years. If you start paying attention to politics at age 8 and live to be 84, you’ll see 20 elections. If something happens 10% of the time then, as you say, it would be expected to happen in 2 elections, i.e., twice in your lifetime. If it happens 1% of the time it will probably happen 0 times. I agree there’s an important difference between something that never happens and something that sometimes happens, but I think for many people it’s not an earth-shattering difference, assuming that you can tell if something “almost” happened (e.g., if the 1%-longshot candidate wins, but just barely).

  • DonC

    Personally the numbers promise a precision that can’t be delivered. I’d be OK with ranges and/or descriptions. Things like toss up, likely, highly likely, virtual certainty, and so on.

    On the other hand the numbers don’t do any harm.

  • Roger2

    Loads of respect to Sam. I’m not even a frequent visitor here nor am I a statistician.

    This is one of the most fascinating things I’ve read in a long time; essentially, we have a small confession of voodoo and a sort of revised prediction in the 91%-95% neighborhood has suddenly popped up.

    Simultaneously, we have the very strange statement that 538 “has the goal of being correct in an academic sense, i.e. mulling over many alternatives and discussing them,” while PEC is all about “is driven by monetary and resource considerations.”

    This is absolutely fascinating and is actually a sort of compliment to 538, while also an admission that “numbery truth” is not necessarily PEC’s mission here.

    Wow! Well, if that’s the case then I don’t think PEC belongs on the New York Times aggregate thingy, because there is a sort of “good faith” concept there that everyone is “trying to nail it.”

    Whereas PEC is actually trying to direct resources and predict where people should put their money, which is a noble cause. Just not like the others.

    Anyway, Sam and Silver are both facing some heat. I, for one, don’t like to see people railing on Silver because it’s pretty clear his work is very good, as is Sam’s.

    Finally, the whole thing about the regressions– I’m not really convinced that something is more accurate because it is more elegant. If anything, simulations across the sciences are telling us that sheer processing power can also take us to interesting places.

  • J.R. Mole

    I see, as usual on PEC, two discussions going on.

    One is essentially how likely it is that polling aggregates miss the actual results by at least 2.5% in Clinton’s favor. This is an interesting discussion.

    Clearly a model that assigns a <1% chance of that happening is absurd. One that assigns around a 30% chance says, more or less, that sigma for this kind of polling miss is around 2.5% or that a normal distribution is quite a bit too narrow. Sam makes a good case against either of these.

    The 5-10% chance from using the MM with more realistic assumptions seems reasonable, bearing in mind the simplifying assumptions made to get there.

    The second discussion is an emotion-driven one revolving around such things as "pundits", "clickbait" and a clear animus toward Nate Silver. This is lamentable, particularly since it tends to obscure much of what's excellent about this site, such as the emphasis on where best to focus one's efforts. But hey, we're all human.

    • Roger2

      I don’t see much animosity here toward Nate Silver.

      Sam is basically saying that 538′s goal is to process every grain of data into the most accurate prediction possible, where PEC isn’t even trying to predict the election, but rather help people decide where to put their resources.

      It’s a tacit acknowledgement that they are apples and oranges, and a totally far out thing to say.

      Can’t believe you guys aren’t seeing what Sam is saying!!!!

    • J.R. Mole

      +roger2

      I see the thoughtful analysis, but I also see a steady stream of digs on the order of

      “Recall Nate Silver’s errors in his attempted critique of PEC …”

      in this article, to say nothing of the comments section. I’m not in academia myself, but I’m pretty sure most of the academics I know would take “attempted critique” as fighting words. This certainly gives the impression that Sam intends the same order of slight, but I hope to be corrected. As to comments, check out the six occurrences of “click” in this recent thread:

      http://election.princeton.edu/2016/10/30/a-test-of-the-polarization-hypothesis/

      Sure, that’s not universal, but it’s definitely there, and lamentable.

      I can only speculate on why Sam’s otherwise-sharp reasoning seems to go fuzzy when 538 enters the picture — my guess is that this is simply a product of having a normal human brain — but it does. For example

      “When all those parameters get put together to estimate the overall outcome, the resulting total is highly uncertain.”

      No, it’s not. 538 used the same high-uncertainty methods to give Obama a 90% chance of beating Romney in 2012, and the 538 forecast never had Obama trailing. This year it gives Clinton a 70% chance, and only gave Trump a >50% chance for a very brief period after the GOP convention — at which point Trump was, arguably, ahead in the polls.

      But never mind that. The implication that you can’t build up a certain result from uncertain parts is false in general, whether or not it’s true in this particular case.

      To be clear, my intuition is that Sam is right and that Clinton’s chances are closer to 90-95%, but that’s just my intuition based on reading both explanations. I would just enjoy reading it a lot more without the side digs and the pretense that there’s no “punditry” going on.

      It’s not like any of this is really falsifiable. If one chart says that Clinton’s chances have oscillated between 50%- and 90%- over the course of the election and the other has it in the 90%+ range most of the way, what’s to pick? The only real way to know would be to pause time at any particular point, run the election a few dozen times and continue after neuralyzing everyone involved.

      Short of that, we’re left sifting through the limited data points we have (lots and lots of polls but few elections and lots of potential parameters to sort out). I personally don’t see how we can be more precise than “Clinton’s chances are considerably better but Trump’s are not negligible”, which is why a 99%+ chance doesn’t pass basic sanity for me.

  • Bridget Blasius

    Sam,

    There is something that I need you to do for me when this whole thing is over. I really want a Sam Wang vs. Nate Silver epic rap battle. I think you’ll own it. Do it for America.

    -B

  • josh f

    Sam thanks for alternative explanation of differences between pec/538. As somebody who has worked in quant fi for 13 years and runs a company in space, I’d say embrace your strengths. Trying to claim that pec is more capitalistic than 538 is opinion-based spinning, which I believe you’ve stated is not a goal of pec commentary.

    Silvers rebuttal to underconfident argument was definately not an academic response (iow cussing at one’s peers :). Besides the emotional aspects, thrust of response was his model is empirical not subjective, implying that it can’t be wrong because he’s backtested it. This as well as thing he’s said in his book (and else where: http://fivethirtyeight.blogs.nytimes.com/2011/03/24/models-can-be-superficial-in-politics-too/) seems to he may believe that out of sample testing removes overfitting as a potential problem. On large data sets like sabermetrics where silver’s pre-538 PECOTA business operated, that is an assumption that will get you into less trouble than with small data problems like prez forecasting. I’m curious to see how far he’s off in his projections and how he responds to it, if it all.

    thx again for your great site

  • Arun

    A small nitpick. There is actually a fairly significant difference between a 91% prediction and a 99% prediction. For example, over four election cycles (the rough current lifetime of this model), a 91% event would have a 1 in 3 chance of not occurring. OTOH, the similar probability for a 99% event is only 4%.

    Other than that, I have Sam to credit for removing any remaining excitement and suspense remaining in this election. I also have to credit him for providing me one of the very few bright spots in this otherwise depressing election through the quality of his technical work and explanations.

  • Mike G

    Your link to the 2014 Silver critique and your response – doesn’t work.

    I’m quite curious to read it, as I’m sure are other readers – any chance you can post both?

  • Scott H

    This method assumes a likelihood distribution whose width is solely determined by drift error (0.8% per day) and you know MM today with 100% confidence. But there is also polling error in the measure of MM. The likelihood distribution should be “what is the pdf that we will see on Nov 8 given the probability distribution of MM today?” The likelihood function could take both the error in MM and estimated drift in to account, as others have said, by convolving the distribution from expected drift with the distribution from poling error (from EVs). Then multiply by the prior.

    In the end, it doesn’t change much – Clinton is 97% instead of >99% if nothing else changes.

    http://www.scholarpedia.org/article/Bayesian_statistics#Prediction
    “Prediction in frequentist statistics often involves finding an optimum point estimate of the parameter…and then plugging this estimate into the formula for the distribution of a data point. This has the disadvantage that it does not account for any uncertainty in the value of the parameter, and hence will underestimate the variance of the predictive distribution.”

    • UI Supporter

      Since http://electionanalytics.cs.illinois.edu/ uses the same type of Bayesian estimator, which was published in 2009 (work done in 2006-2008), their estimates are nearly the same (see the very strong Republican swing for the worst case for Clinton.) As they say, imitation is the greatest form of flattery. The only differences are the parameters.

    • Sam Wang

      Imitation? Am I cited in that 2009 paper for work I did in 2004?

    • UI Supporter

      Sam Wang // Nov 8, 2016 at 1:57 am

      Imitation? Am I cited in that 2009 paper for work I did in 2004?

      That is great. Can you provide the reference and citation for your work in the open literature?

  • Ashok

    What a difference four years makes!

    During the 2012 elections, I didn’t know about PEC, or for for that matter, any of the poll aggregator sites. During the closing stages, Nate Silver got talked about a lot in the mainstream media because of his sharp piece criticizing the media for saying that the election was close. To the media, it looked like he was going out on a limb with the Obama victory probability in the high 80s (I think). The Washington Post had carried a whole slew of pundit projections and they were all over the place. After the results, the media pundits were livid: principally because they had egg on their faces and Silver was essentially vindicated.

    This time around, fortunately I came to know of PEC from my friend’s son who’s pursuing a doctorate in Poli Ssci at Princeton (he’s taking time off from writing his dissertation to get his hands dirty at CBS!). And I see a complete change of scenery in the prediction drama, what with PEC (briefly) carrying Clinton at 100%! And Nate Silver is on some other part of the limb, going by the Josh Katz picture (https://twitter.com/jshkatz/status/793553664724140032 ).

    Tomorrow night, how can we determine who, if any, is vindicated? I wonder whether the following is a fair decison rule:

    . If Trump wins, 0% would surely look silly, but a 32% call won’t fare much better. Both will be buried in the stampede of the likes of Investors Business Daily.

    . If Clinton wins, but Sam has to eat a bug, I suppose one could say 538 is vindicated.

    . If Clinton wins with a healthy margin — hopefully, in a landslide — one can rightly pooh pooh 538.

    • David Abbott

      Clinton winning in a landslide is more consistent with 538 than PEC. PEC’s confidence interval is so tight that any big deviation from the expected outcome in either direction would call it into question.

  • Bill

    What about some correlated polling error? Say a bunch of surveys’ likely voter models are off in a similar way, and that means Trump over performs his polling in the rustbelt?

  • novum

    Sam, I just wanted to say thanks for the site, and the wonderfully informative discussion in the threads (which I always tell people to read when I post about PEC). As someone who is trying to learn the science of stats/probability modeling etc, this has proven invaluable, and oh, it also made me pick up your “Welcome to Your Brain”. Can’t wait to jump in.

  • ATF

    “… a more reasonable uncertainty in the Meta-Margin is 1.1%, giving a Clinton win probability of 95%…. I am not going to change anything in the calculation at this point.”

    I understand that changing the code this late in the game would be a bit silly, but you’re also sticking to your guns in part to keep those sweet low Brier scores, aren’t you? ;-)

    • Lorem

      Well, if we’re doing state-based Brier scores, then having appropriate uncertainty would actually be a good thing. Getting a state miss when you’re calling it at 95% would hurt a lot more than if you were saying it’s 85%.

  • llywrch

    Sam. I would say your comment about Nate Silver’s bias against the front-runner is the kindest thing said about his otherwise inexplicable generosity to Trump’s chances.

    Silver argues that the cumulative effect of the margin of error in the polls could disguise an unexpected Trump victory. On the other hand, more likely missampling of the Hispanic vote in Nevada, Texas, & Florida could be unexpected benefit for Clinton. Or the fact she has a much better ground game than Trump’s (or, to put it bluntly, she has one & he has practically none).

    Even some back-of-the-envelope calculations make a 30%+ probability for Trump impossible. Go to 270towin.com, & set the map to “2016 competitive”; this will set 9 states — & two districts in Maine & Nebraska — to toss-up. In other words, roughly a 50-50 chance for either candidate to win that state. Let’s simplify matters & assign the two toss-up districts in Maine & Nebraska to the victor of the rest of those states. Then let’s to go a little further & give Arizona, Iowa, & Georgia to Trump — which I’d say is a reasonable expectation. (Maybe Clinton can win one or all three; but for this quick-&-dirty calculation, I’m giving the benefit of the doubt to Trump.)

    Now it is obvious that unless Trump wins Florida, he cannot reach 270 EV — thus, all things being equal, his chances of winning are no better than 50-50. Next, a bit of playing around with the map on 270towin, it is clear that Trump also needs to win Ohio — reducing his chances to 1 in 4. (0.50 x 0.50= 0.25) A bit more playing around shows that even here there are combinations that don’t result in a Trump victory, so his actual chances are less than 1 in 4.

    Now one could argue that certain states — such as Pennsylvania, Wisconsin, or Colorado — aren’t 100% locked in for Clinton. True. But one could also argue that Arizona, Georgia, & Texas aren’t 100% locked in for Trump. Making special pleadings that we should ignore the results of our calculations is just another way to cherry-pick the evidence to make it fit the answer we want.

    And making the evidence fit the answer we want is ignoring the truth. We must work with the results we get, not the results we want.

  • Mike

    Since the predicted probability is so heavily dependent on the error estimate, why not estimate a 90% confidence interval on the error? Then you could publish the resulting probabilities at the top and bottom of the interval (i.e., something like 89%-99%).

    • Brendan

      This is akin to what I was thinking. Even if you can’t actually derive a 90% confidence interval, you could have a few different values for this parameter, creating a few different confidence intervals for the result (e.g., a series of shaded confidence areas in the meta-margin or EV prediction).

    • Lorem

      I think that would be poor presentation. Let’s say I’m a relatively uninformed viewer – which number should I pick from that range? Should I just go with whatever my preconceptions tell me?

      I think to justify giving a range, you must provide extremely clear guidelines of how one should select a number from that range and also a compelling explanation of why the viewer should do the picking instead of the forecaster.

  • Edward Allen

    PEC answers the question definitively: the polls show Clinton is ahead. However, while I think you have done a good job of arguing that 538′s approach adds uncertainty, doesn’t their method, which allows for heavily polled states to be used to evaluate poorly polled states, provide a good tool for the activist as well?

  • mike

    “closer than FiveThirtyEight”

    In 2012 you incorrectly called Florida for Romney. 538 got all fifty states correct. How exactly do you spin that as “closer” on the electoral count?

    • Rachel

      Because of how confident they were about it. For example, if 538 says that there’s a 50.5% chance of Obama winning Florida and PEC says 49.5%, neither has put much weight behind their decision, although if you decide to throw out probabilities then 538 was “right” and PEC was “wrong”. Then suppose there’s only one other state, which 538 also says Obama has a 50.5% chance of winning but which PEC says has a 90% chance of winning (which Obama then wins).

      Technically 538 called more states correctly, but wouldn’t you say that PEC’s predictions were more accurate?

    • Reed

      Sam was talking about final EV count, not state-by-state predictions. I’m not sure what he means with 2012 – 538 was indeed closer here, although FL, from my understanding, was polling so closely that it was a bit of a toss-up.

      2012
      538: 313-225
      PEC: 305-233
      Result: 332-206

      2008
      538: 349-189
      PEC: 364-174
      Result: 365-173

    • Sam Wang

      Yes, I stand corrected on the 2012 state-by-state. I actually found Florida too close to call at the time, but got other states correct individually.

    • mike

      You stand corrected…but you haven’t corrected that statement in the article?

      And there’s no way to say which level of confidence is “right” for a single election. Both sites predicted maps, one got them all right and one didn’t. So no, I wouldn’t say that PEC’s predictions were more accurate.

    • Emmy

      @mike Chill, he probably hasn’t and time to correct it in the article yet.

      Additionally, I agree with Rachel’s point — Mr. Wang’s model was more accurate than 538′s. Not to mention 538 fluctuates endlessly at any small information point

    • Sam Wang

      corrected. thanks, Mike.

    • Lorem

      There most certainly is a way to see which level of confidence is right for a single election – Rachel outlined it. You can also see it put into practice over here: http://rationality.org/2012/11/09/was-nate-silver-the-most-accurate-2012-election-pundit/

  • Jake Smith

    Hi Sam,

    I’m not quite as skilled at math as you are, but I have a BSEE with a minor in Math and MS in Optics. I run my own (private) prediction spreadsheet with data exclusively from SurveyMonkey and Ipsos/Reuters that I scrape from the CSV that you provide. I do this for what you could call Wall Street Bets. Independent of your analysis, my “meta-margin” would be 2.7% and that represents a 92.04% probability of a Clinton victory. Plus I have confidence that the bump will be for Clinton, since women and minorities tend to be disproportionately silent about their opinions and confederate flag-wavers tend to be disproportionately loud. So I also feel really confident that Clinton will win. My model is far less complicated, but also makes far fewer assumptions, and pulls lass data, which I think could possibly be a strength, if data gets outdated or convoluted.

    Anyway, here’s a link in case you would like to see what I have done: https://drive.google.com/file/d/0B7E3dK41rQKaa3FjOVZULTZmbzA/view?usp=sharing

    And to all the people complaining that 99% just sounds like this means “it cannot happen,” understand that there are many things that do happen that are far lower than 1%. He could show 99.993% and that still tells us far more about the likelihood of an unlikely event. Lightning doesn’t strike 1 out of every 100 people.

  • PaulC

    I, too, thank you, Sam. In particular I admire the sheer elegance of your work. The simplicity of the meta-margin construct, is a thing of beauty.

    But let me also remind you that you have expressed another important goal; namely, to create a scientific basis for responsible horse-race reporting by professional journalists, among whom there is an institutional bias to cherry pick individual polls seemingly to make races seem closer than they may be. “Poll-aggregation” has come to maturity since 2000 and is now a true force to be reckoned with.

    (Ultimately, by eliminating some number of horse-race noise articles you may be enabling media bandwidth to be refilled with policy signal articles.)

    Nate Silver is and was a pioneer who helped establish poll aggregation. Now, unfortunately, his web site seems to benefit more from the noise than from the signal. If Nate’s “way” has a bias, its toward retaining an overly complex and proprietary system whose complexity adds no accuracy, and whose added uncertainty is used to fuel the kinds of horse race noise articles it used to debunk.

    • Dennis

      I’d also say that the stakes are higher for Silver – i.e., his future livelihood depends on the correct calling of this race. If he ended up with a model that gave more certainty, but ended up wrong, his website is done.
      If his model injects more uncertainty, yes it brings more readers in (but I really think he is far too ethical to purposely skew results for readership), but it also protects against an outlier event that would be devastating to 538.
      A version of (perhaps unconscious) loss aversion.

    • Sam Wang

      Come to think of it, I am not even sure that the increased uncertainty arose as a result of a changed algorithm.

      For example, over there they’ve complained about the lower number of polls this year. That might make pollster-specific corrections more difficult, which would in turn inject more uncertainty than previous years – all without changing their method.

    • Roger2

      I just don’t see how accounting for a higher percentage of undecideds and a 2.6% lead qualify as an “overly complex system.”

  • Michael

    Is it possible that readers struggle to accept your calculations because they seem counter-intuitive? Maybe people are saying, “it just can’t be this simple! It has to be more complicated than this!” As if it weren’t complicated enough the way you do it.

    • Roger2

      I’m a good one to ask, since I’m not a statistician and not very involved at PEC or anywhere, really.

      The answer is no, it’s not because it seems counter-intuitive. It’s because there are a high number of undecideds and we don’t know who they are going to vote for, and also Clinton’s lead is not out of “polling error” territory, which is important in some states as we are shifting demographics and unsure of who is going to vote.

      Now, maybe there is some magic bullet to predict who the undecideds will vote for, but if somebody had that magic bullet, it would be Nate Silver, not Sam Wang, by virtue of their very different and respectable approaches.

      But Sam just told us that if we want to get toward the truth of the matter, for example if we were gambling rather than activist investing, we should rest on our laurels at 91%.

  • Geoff

    Thanks for this post, Dr Wang. Can one argue for your 0.8% (but perhaps a longer tail) by pointing to the “winner bonus” you’ve written about before? As you wrote a few posts ago, on Obama outperforming his poll numbers against McCain, there may be more “emotional reward” in voting for someone you think is ahead, and that this “implies that there is a hidden bonus for whoever is ahead.”

    The (admittedly few) numbers from recent presidential elections do suggest that if there’s error, it’s more likely to favor the leader.

    To engage in some emotional punditry, I think the optics of the final few days favor Clinton too, what with Comey-redux, Beyonce/LeBron/Springsteen. Trump’s not doing himself any favors wearing a baseball hat that shrouds his face in shadow.

  • Robin

    Sam,

    Thanks for the article.

    I have two questions:
    1. Do you agree there is more uncertainty in a race with a 2.6% average lead for Clinton with 12% undecided voters and a race with a 2.6% average lead for Clinton with 2% undecided voters?

    2. Does your model treat the two scenarios differently? It is not obvious from your comments.

    Thanks in advance for any response.

    • RTR

      Where are you getting 12% undecided? Looking at Pollster.com, nationally undecideds are averaging around 5-6% and the median looks to be lower than that. I see what you mean though, this is a couple points higher than undecideds at this time during the Obama-Romney election, which should suggest increases uncertainty in theory. However, I’d also imagine that trying to estimate how undecideds will break is something that would increase uncertainty. Lot’s of things seem like intuitive additions to a model, but at the end of the day, if it doesn’t expand the explanatory power in a quantitatively meaningful way, best to leave it out, right?

    • Robin

      @RTR

      Thanks for your reply.

      Both 12% and 2% are made-up numbers. I want to know if Sam thinks the distinction is relevant and if his model is sensitive to it.

  • Ed Wittens Cat

    i would just like to point out– how utterly unprecedented it is to have public intellectuals debating whether a presidential candidate (Trump) is a psychopath–
    https://www.salon.com/2016/11/06/why-donald-trump-scares-you-so-much-and-why-it-matters/
    or a sociopath–
    https://www.youtube.com/watch?v=NIL0w4BkHQU

  • Dave

    Hey Sam;I’m a politics junkie who visits all the sites for analysis.In comparing the meta margin of Clinton today(2.6) and Obama in 2012(2.3),she should win the popular vote by slightly over 4 % while receiving slightly less electoral votes.

  • andrew

    Go easy on Nate, in the latest election update he really started to lose it: https://medium.com/@and0/election-update-how-our-model-works-8acb97202277#.jwtgz16yu

    • 538 Refugee

      His language is deplorable considering his position. I can see a lot of media sites simply start ignoring him because they won’t want/need the association.

    • Sam Wang

      Um, that’s a parody.

    • 538 Refugee

      OK. Got me. That was my first impression so I checked the by line and then the page for disclaimers. Personally I think it crosses a line to use a by line like that. (Nate Sliver would have been a good choice here.) I did double check his Twitter feed though and he does drop f bombs there so the point still stands about media outlets. I only found one but the one I found was on a list of others I saw.

    • Roger2

      Yeah it’s hilarious but Nate Silver probably didn’t write that. It’s not posted on 538 and is rather posted by some other guy on Medium.

  • TeddyVienna

    At this point, I think the only thing that worries me is voter suppression.

  • 538 Refugee

    ” I am open to correction.” For new/occasional readers let me point out that this applies to EVERYTHING. Every aspect of this site and process can be considered ‘peer reviewed’. I think Sam had a different mechanism in mind when he used the word “consortium” in naming the site, but the reader/contributers have made it just that. This isn’t the first discussion of the model and I doubt it won’t be the last.

    The period between elections does get pretty slow but if you look up next to the meta margin you will see “RSS”. This lets articles show up in my mail client so I know when there has been a new post. A very handy thing when there are fewer articles being posted.

  • Tim L

    I am not a mathematician (architect by trade) so i really appreciate the breakdown – i find it incredibly interesting!

    Sam, i have a question. I am prefacing to say that while i think i follow your description, this is outside of my discipline, so i could easily be wrong. It seems to me that the method of generating the final percentage confidence prediction is very different in your model. It seems, based on their description, that 538 is baking in the uncertainty on both the state and national level, with a greater margin of uncertainty (going back to average polling eras that include priors farther back than just the last 16 years) and THEN running 10,000 simulations accounting for this uncertainty (i’m assuming going both ways); they have mentioned on twitter and in their podcast that they are running these simulations every time they refresh the model output. It seems to me that they are then just publishing the results of that simulation as a percentage outcome “prediction”. Does this seem correct (obviously, no one can know since they won’t disclose)? If so, it does not seem like a prediction at all to me. Conversely, if i follow your logic you are simply trusting the polls to generate an adequate median, establishing that for both the MM and EV calcs, and then applying a factor over time on how much you would reasonably expect the polls to move to generate a percentage confidence in that already established median. To me, this makes much more sense as a prediction of a future event a variable number of days away since by it’s very nature it would get smaller the closer one gets to election day.

    Now, i may be totally wrong, and i sure there is SOMETHING in “their” model which accounts for the time (or lack thereof) before the election for polls to move, but it seems to me, even as a layman, that this would have significant effect on making a prediction. Essentially, Sam, are you simply trusting the real data more than 538 and allowing it to speak for itself (and your confidence in it) while they seem to be trying to bake in uncertainty then just allowing it to spit that uncertainty back out?

    anyway, great work – super informative and interesting!

  • MJW

    It’s not that they like Trump – it’s that they are biased away from the frontrunner, whoever that is at any given moment.

    This is where I get confused.

    Assuming that 538 was running their model on the same parameters in 2012 as it is this year (which might be a hairy assumption, but without more transparency on their end it’s all I have), the above means that even with a bias away from the frontrunner, they had Obama at a 90.9% chance on election eve ’12. Whereas today HRC is hovering at 65% in their model.

    Doesn’t that all by itself suggest considerably more uncertainty exists this year?