Princeton Election Consortium

A first draft of electoral history. Since 2004

Looking ahead

November 9th, 2016, 12:53am by Sam Wang


Going into today’s election, many races appeared to be very close: 12 state-level Presidential races were within five percentage points. But the polls were off, massively. And so we face the likelihood of an electoral win by Donald Trump. At the same time, Hillary Clinton appears likely to win the popular vote. The Upshot’s model currently projects a Clinton lead of more than 1 percentage point. If that lead lasts, it means that more American voters preferred her to Trump.

At the moment, the NYT is projecting Trump leads of less than 1 percentage point in Pennsylvania and Michigan. Even without these states, Trump has at least 268 electoral votes (depending on some districts in Maine and Nebraska). We will see in the morning how these last few states and districts will be resolved.

In addition to the enormous polling error, I did not correctly estimate the size of the correlated error (also known as the systematic error) by a factor of five. As I wrote before, that five-fold difference accounted for the difference between the 99% probability here and the lower probabilities at other sites. We all estimated the Clinton win at being probable, but I was most extreme. It goes to show that even if the estimation problem is reduced to one parameter, it’s still essential to do a good job with that one parameter. Polls failed, and I amplified that failure.

Update: I believe a likely reason for polling failure is undecided Republicans coming home in the home stretch.

This election is about to create shock waves that will make the last year of campaigning look mild. We are about to see both houses of Congress under Republican control, quite possibly with a President Donald Trump. This comes in the face of a reasonably growing economy and a popular Democratic President about to exit the White House. It is difficult to reconcile these different facts.

Thinkpieces that have been written in the last few weeks have to be re-examined in a new light. Ezra Klein at Vox has written about the weakness in U.S. democracy, in which a weak Republican Party could nominate Trump, and partisan polarization gave him a shot at the Presidency. This one-two punch appears to have landed, hard. I was correct in documenting Trump’s rise in the primaries, an easier task for polling analysis because there, his lead was considerable.

I have written about the role of partisan polarization in getting voters to choose up sides, to the exclusion of even considering a vote for the other side. The chickens have now come home to roost. Exit polls showed that most voters felt that Trump lacked the temperament to be President, and that Clinton was seen as more qualified. Yet Trump seems to have rallied enough support to get overcome these factors. All Presidential nominees have had lower and lower approval ratings, and Clinton was no exception to the pattern.

Now we see where that long trend has led. One consequence is that more voters refused to support either major candidate. Neither Trump nor Clinton is headed for winning a majority of voters in Pennyslvania or Michigan. In Pennsylvania, the NYT projects that over 3% of voters cast their ballots for Gary Johnson or Jill Stein. In Michigan, the minor-party total was over 4%. In both cases, these numbers are considerably greater than the Trump-Clinton margin.

The coming years will be disruptive ones, to say the least. Whether you are Democrat, Republican, or neither, it’s going to be a challenging time ahead. It’s Donald Trump’s Republican Party, and maybe his Presidency too. The nation belongs to all of us. Good night.

Tags: 2016 Election · President

291 Comments so far ↓

  • Doug

    I’ve been thinking about your post about Hari Seldon. He missed the Mule. Still, you were the smartest person in the room with the early call for Trump during the primaries, and this isn’t even your day job!

  • Joec

    “POTUS OF AMERICANS” is correct. Hilary and Obama accepted the results of the election. In 2008, George W. Bush accepted the results and the two worked together to insure that the economic disasaster was not worsened.

    GWB for all of his faults understood the need for a peaceful transition. Unfortunatelly, the rest of the GOP set out to delegitimize Obama’s presidency despite two resounding majority elections in both the popular and electoral vote.

    Here we have a president-elect elect who did not win the popular vote majority.

    Paul Ryan and others are eager to grab the reins of power and proclaim a MANDATE for themselves.

    While the polling indicated an electorate thinking we are headed in the wrong direction, it did not differentiate what direction we should take. Once again the GOP is assuming a mandate.

  • Peter Wiles

    How useful is it to report the win probability anyway? I have an 83% chance of “winning” a game of Russian Roulette (one bullet in six chambers… seems like an apt analogy to the situation we find ourselves in). If I played one time and shot myself, would you be claiming that my probability calculation was wrong? The fact that Trump ended up winning does not invalidate the notion that coming into it, Clinton had a good chance of winning and it was hers to lose. He just bucked the odds. I don’t understand how a single event, namely a Trump win, automatically means that a model showing a smaller chance of a Clinton win was more correct than a model showing a higher chance of a Clinton win. A lucky Trump can win in both models.
    Lets continue the analogy. If two people play Russian Roulette, one dies and one survives. Can you tell me who had one bullet in their gun and who had 5? The outcome of one event doesn’t tell you anything about the underlying probability of that event!

  • HG

    A lot of people here (not Sam) owe Nate Silver a big apology. Like Romney supporters in 2012 they turned on him because he told them things they didn’t want to hear.

    Of course Silver got it wrong too, because he was working with the same polling data, but he was less wrong. We can only conclude he had the better model.

    Princeton professors are smarter than 99% of the people 99% of the time, but they are not smarter than 100% of the people 100% of the time.

  • Mark

    GIGO- garbage in, garbage out.

    It seems PEC’s math was solid, however the data was bad. There’s going to be a lot of reckoning over the next few months/years as to why it was bad, and I don’t know if there will be a simple answer. But if the polls were wrong then anything predicted using those numbers in good faith are going to be bad.

  • weichi

    Too early emotionally for post-mortems. And not yet enough data to help us understand the polling failure. It will become more clear in a few months.

    Sam, thank you for the site!

  • HG

    With respect, Professor, stick to neuroscience. The objects of that study don’t strategically lie to the people who are studying them. You can be more confident that you can accurately measure the things you are putting in your models.

    • Micheale Duncan

      The only person that predicted this outcome with any degree of accuracy was Michael Moore. I went to your site every day for an update. It kept me reassureed. I won’t do it again.

    • TK

      I think some criticism is justified here. But I think this is overly harsh.

      I think that the real lesson is that one needs to handle disagreements over methodology academically. I hope that this lesson is learnt, and we get to see some more integrity next election about disagreements over methodology – especially when those disagreements are voiced in interviews for news articles.

    • BA

      I think 2016 was an outlier. You had two candidates that we not liked. I don’t know if people behaved rationally in this election. I think there was a section of America who hated their options. But maybe the news about the emails, and the wikileaks, and the fact that Clinton seemed to ignore the blue collar white vote, is the reason why the models failed. I don’t agree fully with the methods Dr. Wang or Nate Silver but I think we need as many people as we can get. I don’t people think people lie to pollsters if the candidate they support isn’t controversial. Plus Trumps nearly constant attacks on the media and polling I think meant that Trump supporters may have not participated in polls. Political Statistics is hard and messy but someone has to do it.

  • Garry Garrison

    I seriously doubt that the vote counts were rigged or that the pollsters deliberately cooked the books. Something about the pollsters’ voter models were off, and the increasing difficulties of getting a good sample probably played a role. More speculative, maybe there were shy Trump female voters — embarrassed to admit they were voting for the sexual assault candidate. Meanwhile, I wish I’d paid more attention to the House generic preference chart. If you look at the unsmoothed Huffpollster chart, the Republicans started a climb around mid-October, and by election day, were pretty close to a tie – about where the national election came out.

  • Anthony

    Sam, you have been very quiet on here and on twitter. No updated post or new podcast yet? Do you plan to leave the country?

  • Paulie

    Looking at the raw pop vote and some of the demos, it appears that Trump pretty much got Romney Lite numbers. He still needs 1.5M votes just to get to Romney. Hillary is missing SIX MILLION of O’s numbers. And that can’t be explained by just low AA turnout. Sam’s overall idea that voters don’t change their minds, especially with Trump’s numbers, looks dead on. Apparently they change their minds about showing up at all though.

  • john bowman

    While this election was a shock, I’m not happy with the lack of performance of Sam Wang either. I’m not gracious in this matter, I’m upset I sought competence here and did not find it. I made decisions based on professed competence and I was wrong for having faith in such competence. I don’t believe Sam Wang and Nate Silver can claim any competence in projections whatsoever

  • Ready to Retire

    I can’t wait until the old steel plants get fired up, coal is tumbling down those mountains once again, 10mm undocumented are on busses back home opening up millions of jobs for Americans, Ford brings back those cars to Detroit from Mexico, Apple begins making iPhones in the States, we start rebuilding our infrastructure, tax breaks for everyone, and our deficit begins to lessen. It will be great.

  • GP

    The national polling organizations may have very well cooked their results. Statisticians and Mathematicians are by definition…well educated, “diverse” (whatever that means), safely employed within government or college institutions. It’s understandable that they (you) only see or hear what you want to see or hear in the numbers. We’re humans. Even Ted Cruz (shivvver). I trust that the people that run this organization and the people that visit its site are smart folks. Go out to the rural parts of the “rust belt”. If you can’t go, get on google street view and wander the streets of elyria ohio, or Clarksburg, WV, or Pecos, TX. Then reassess your surprise that a populist message resonated.
    I believe in the value and the importance of science and math and open trade. But, when we hear the phrase, “some workers will be displaced or left behind”..a red flag should go up. Fortunately, the dissatisfaction and anger from these segments was manifest in a series of primaries and a general election.
    If you want to do a study that will really scare you, do a correlative study of how many “uneducated whites” (as they’re called by so many) own modern, powerful firearms.

  • MAT

    I’m in North Carolina. I used to be a County Board of Elections member. The voting machines we use in our county are scans of paper ballots. I’ve been thru multiple hand to eye recounts and never once was the machine count off. Trump and downticket candidates hugely overperformed across the board here yesterday. It wasn’t due to hacked voting machines. It’s because he got more votes.

    There are millions of Obama voters that just vanished beteeen 2008 and now. Republican presidential candidates have seem to hit a ceiling of about 60m votes for the last few cycles, Trump being no exception. How did we lose millions of other voters and how do we get them back is the question I want to understand.

    • AySz88

      This may not be the “right time” while people are trying to strike a conciliatory tone, but the loss of those Voting Rights Act provisions looms very large. It’s something to look at.

    • Mike

      @AySz88 – I dont see any plausible way that a similar act can get passed in the next 4 years, and if RBG steps down while Trump is still president, we’ll probably never see it again. I’m trying to be realistic not pessimistic. Progressives need to focus on organizing within the extremely tilted framework that the rightwing will allow, not pretending that it might somehow be un-tilted any time soon.

    • Anthony

      @Mike – After Republicans largely take away our right to vote, what else left is there? I think that is what liberals are truly not grasping right now. No matter how bad Trump is the only way to fight him and other Republicans is at the ballot box. They are working on taking away that option.

    • JBR

      Four things.

      1. Two weeks ago, just before the exceptional and unwarranted Comey letter, Clinton was polling +5-6% nationally. This had fallen in knee-jerk fashion to 2-3% by election day, with a lot of voting happening in the meantime. If Clinton had received just 1% more total vote, that likely would have flipped key battleground states.

      2. A single bad call by Clinton’s support staff – especially one IT guy – led to Podesta’s gmail account being hacked by a targeted phishing attack… What ensued was the daily Wikileaks email drip.

      3. The Supreme Court’s rollback of key provisions in the watershed Voting Rights Act which led to the shuttering of hundreds of polling places across ten states (including North Carolina), disenfranchising or suppressing countless votes.

      4. Incessant vicious lies by Trump and his minions about everything – especially Clinton. His overall truthfulness scoring was only a fraction of Clinton’s.

      Bottom line: dirty tricks win again, common sense and decency loses.

    • Mike

      @Anthony — There are a LOT of people who live in the costal progressive states where there is little risk of the GOP coming to power and restricting voting. Progressivism will have to move primarily to the State level. But it can be effective there. Think of how California environmental regulations were able to push the whole automotive industry to enact cleaner standards.

    • Suvro

      Republican votes by the year:
      2008 59,930,551: 2012 60,934,407: 2016 59,022,040

      Democratic votes by the year:
      2008 69,438,983: 2012 65,918,507: 2016 59,245,315

      YUUUGE dropoff of Democratic voters this time around!

    • 538 Refugee

      I remember some derision of models showing a huge drop in the African American vote in polls having Trump ahead.

    • Arun

      Those comparisons are rather telling. Republicans vote in the same numbers year after year (give or take). The difference is in the Democratic vote. Obama was able to inspire people to come and vote for him. Clinton was not. On the Republican side, quality of candidate does not seem to matter – to a first approximation, McCain, Romney and Trump did similarly. Somehow, the Democrats lost enthusiasm.

    • Andrew

      Arun, I’m not so sure this is specific to Clinton. Same thing happened under Gore, and that has happened again and again historically. It’s darn hard for one party to get three consecutive terms in the WH.

      It does point to something though, how do Democrats get Democratic voters to vote reliably? As far as I can tell, this wasn’t just one group, African-American turnout was down, yes, which is perhaps lack of enthusiasm, and/or voter suppression. BUT 53% of white women voted for Trump (!). Hard to explain that one.

  • sukaeto

    Long time listener, first time caller here. I generally don’t post comments on blogs, but I felt compelled to do so now.

    Sam, thank you so much for what you do here. Please don’t be discouraged or feel discredited by what happened last night. Your model is legit, your methods are good and right and true. Even the best model will yield bad results when it’s fed bad premises.

    Please don’t give up, we need people doing the work you do here. It just also so happens that we need better polls.

  • Randy Haugen

    Makes no sense at all that almost All real polls had Hillary ahead,So I wonder if its simply a case of people lying to the pollsters? Sam thanks for all the work you did,you are a inspiration and hope you do this again in about 14 months for the 2018 election.

  • fred flint

    I think the bottom line is garbage in garbage out. If the polls are completely wrong to begin with you are not going to get the right answer no matter what you do.

    I don’t blame Sam at all. There is no recent historical precedent for all the polls being so wrong.

  • The Indomitable Ted

    Hi Sam,

    I’m sure we’re all feeling pretty shitty this morning, but a little levity and little analysis would be welcome. So please, when you’re ready, swallow down that bug and help us identify the sources of the apparent systemic error in the polls.

    For example, it seems to me, in retrospect, that those 5% undecideds may have predominately Republican leaners this entire time. Is there any plausible way for pollsters to gauge this lean in the future?

    It also seems that we ended up lacking quality state polls where actually needed them. So do you have any ideas on better identifying swing states in the future?

  • Rick L

    I coded up a time series of the Brier score for 538s “polls plus” and for PEC, for five battleground states (FL, NC, OH, NH – assumed HC win – and NV). As a reminder, 0 is a perfect score and 1 is 100% confidence in a result that didn’t happen.

    538 fluctuates from just above 0.2 in late July to ~0.26 in mid-August, down to 0.18 in late September, up to nearly 0.28 in mid-October, then drops to 0.18 on the day before the election. The final update bumps it up to 0.198.

    I’m not 100% sure on the PEC calculation. I couldn’t find the floor on the MAD that Sam uses, so I assumed it was 1%. Given that, the PEC Brier value starts at ~0.35 in late July, climbs to 0.6 in mid to late August, drops to 0.2 in early September, then spikes at 0.75 in early October before dropping to 0.25 in early November. It finishes at 0.32.

  • Mark F.

    Thanks, Sam, for your analysis and for admitting your HUGE error. We won’t expect you to eat that bug, btw.

  • Brett H

    Sam -

    First of all, thank you so much for your analysis. Silver hedged his bets based on intuition that the uncertainty was high, and he was vindicated. You were more impartial and trusted your data. I appreciate your dedication to statistical and mathematical rigor.

    The central tension that I saw as an observer of this site and the other major aggregators was between the feeling that we had that, somehow, this election was “different” and the evidence that we had (MM/EV predictor) that showed that this election was like 2012 and 2008. We arrogantly assumed that the lack of movement in the MM validated the polarization hypothesis. Well – Trump managed to take WI and PA. How did polling fail to capture this? This question will be asked again and again in the weeks and months ahead.

    If Trump succeeded by getting an unusually large amount of disaffected white voters to crawl out of the woodwork and go to the polls, this reminds me very much of LBJ’s first congressional election, where he knew he couldn’t beat the political insiders in the urban areas, so he concentrated on the voters in the isolated Texas Hill Country, where he grew up. None of those voters had ever been communicated to by the Austin elites. By simply making them feel heard, he earned their vote. The polls missed them entirely. He won.

    By contrast, HRC, as the establishment candidate, made no effort to court these voters. She disparaged them wholesale. Her “basket of deplorables” comment will go down in history like Romney’s “47%” comment — a tone-deaf assault on voters she desperately needed to win.

    Sorry for the rambling. I am trying to see this with a historian’s perspective – for it is far too scary to look ahead.

  • Mike Beers

    Sam, thank you for all your hard work, for providing a shared space for so many of us. It is honorable of you to apologize for underestimating the probability of Trump’s win. But no apologies were necessary.

  • Five Tool Player

    All statistics and predictions aside, I am devastated by the outcome of this election. The Cubs World Series victory does not assuage my deep, gut level feeling of disbelief and powerlessness in the slightest. In my view, the ~+4 Trump polling bias is most easily explainable (a la William of Occam) by the sizable chunk of closet Trump-supporters who were too embarrassed to admit their preference to another human being over the phone.

    • Gelatinous_Cube

      The Economist believes it was a failure in the likely voter models, not “shy Trump voters”. Last night here someone noted the high turnout which was followed by ominous speculation whether it would help or hurt Clinton. The ongoing inversion of the blue collar/elite voting pattern among whites is consistent with the theory that “high turnout” in the abstract is no longer good for Dems. It’s a question of who turns out.

    • Ash

      Conventionally high turnout is associated with democrats who vote only during the general election. Probably that theory needs some work. Total number of votes casted is around 10M lesser than 2008, and 5M less than 2012 general election. Looking at the razor thin margins in PA, WI and MI, I believe the Obama coalition wasn’t as energized as it was in previous two elections. This could have thrown off models like Economist/YouGov which rely on previous election turnout to predict future election. Yougov consistently predicted Clinton +2-5%.

    • GM

      Cube, this just isn’t true. The calls for increased turnout were wrong – Repub voters were down more than a million votes, and Dem voters were down well over seven million. The two stories so far are: voters did not turn out for Hillary, and she lost minority vote share (including women) when compared to Obama.

  • gumnaam

    This is certainly a time to take a moment and reflect a bit. There will be a lot to do later, but one cannot really blame a poll-based model for being wrong if the polls are wrong.

    • Brian

      Agreed, but one hand kinda washes the other.

      I know it’s a relatively minor “problem” compared to the world of bleep we are set to embark on, but America’s new favorite pastime of poll watching came to a sudden end last night.

  • KT

    Okay, I’ve slept on it.

    Now I really think the polling companies cooked the numbers. I do not know how or why it happened. It happened, but it could have been a number of reasons. Maybe their employees were lazy or failed to meet their goals, and they turned in fake or duplicate surveys. Maybe the companies deliberately skewed or made minor adjustments to their data because they had political interests and ties to the DNC. Maybe they “recycled” data: one polling company would use another company’s data, so if one poll was off, the other poll would also be off. It could be other reasons, or a combination of all of them.

    I do not know why they did it, but they did: the polling companies cheated and cooked their number. And the problem is not limited to one or two companies. It is systemic. It is an industry-wide problem. That is really the only logical explanation to how and why almost all the polls were so off (with a couple exceptions.)

    • KT

      Is there any third-party or outside entity that periodically/randomly audits the polls and data behind all those polls, or are we supposed to take all those polls on face value?? Someone needs to audit all those polling companies and look into their data.

    • Ready to Retire

      But the polls are only as good as the respondents. If enough lie, say undecided instead of Trump, how could a poll catch that, if in fact that’s what happened to some degree?

    • Mike

      It wasn’t the polling companies. Polls were correct in Illinois, but off by 9% in Wisconsin, which had massive voter suppression. No polling company can or should account for people who try to vote and are turned away or just have their vote not counted. The blame lies with the politicians that unconstitutionally took away the voting rights of ~2 million Americans.

  • Scott H

    To scientists, data and models aren’t a game where you’re cheering for a result. You do science to learn, not to be right. So what can be learned from this election?

    The PEC’s model theory was that you can predict the outcome of an election (and important states to allocate resources) assuming:
    1) Polls accurately measure electorate.
    2) The precision of the measurement is determined by the spread of the polls.
    3) Median polls remove outliers.
    4) Error in polling measurements between states is uncorrelated.
    5) The state of the race today will randomly walk until the election day. The amount of random walk is expected to be 0.4% per root day.
    6) The final error in measurement is expected to be 0.5 (changed to 1.1%) off of the last day’s polls.
    7) The final result is expected to be within 6% (sigma) of the average polling result through campaign.

    Based on outcome, what conclusions can be drawn?
    1) Polls, this cycle, did not accurately measure electorate.
    2) The precision of the measurement is probably still determined by the spread of the polls, but the accuracy was way off.
    3) Median polls remove outliers is probably still true, but can’t tell since no one was right.
    4) Error in polling measurements is correlated when demographic turnout is drastically different than expected since assumptions in likely voter turnout is apparently shared among all polls.
    5) This election provides no data on random walk since it the polls were so far off.
    6) The final error in measurement was in the range of 4-5%. 5 to 10x larger than expected.
    7) The final result is not at all in line with the average of polling.

    Next step is to study if this election is an outlier, or a new trend. If a new trend, through away bad assumptions. More data is needed. Unfortunately, we only get to test this once every four years, and in statistics “three points make a line.”

  • Shinny Objects

    After I recovered from nausea, it occurred to me that Trump, for all his many faults, is not a religious fanatic (Cruz, Carson, Santorum) or a GOP Cartel front-man (Bush, Walker, Kasich, Rubio) but a former NYC Democrat. I’m hoping some residual liberalism is still lodged in his brain.
    He’s least worst option of the original crew.
    Now, back to the nausea.

    • gumnaam

      Stages of grief: denial, anger, bargaining, depression, acceptance. You are either at bargaining or at acceptance, I hope it is the latter.

    • KT

      Donald Trump’s supporters voted for him because they wanted to take a country to the right. We lost. They won. He owes his supporters what he had promised to them.

      “Elections have consequences.”

    • KT

      And he does not owe us anything.

    • Shinny Objects

      Gum: Acceptance tyvm
      KT. He will be the POTUS of all Americans, whether they voted for him or not and owes all of us his best effort. If HRC had won, I’d expect the same, not limited to only her voters…

    • Rob

      His surrogate Omarosa is already talking about an enemies list so I think it’s rather quaint to assume he’ll happy govern for all of us. We didn’t elect someone who appears to have the greatest interest in democracy or governing.

      http://www.salon.com/2016/11/09/omarosa-hints-at-a-donald-trump-enemies-list-its-so-great-our-enemies-are-making-themselves-clear/

    • Shinny Objects

      Rob:”I think it’s rather quaint to assume he’ll happy govern for all of us. ”
      I don’t make any assumptions about Trump. I have hope and expectations… neither of which involve quaintness nor assumptions. I expect him to be awful. I hope he is not. See?

  • Dave Flanagan

    As the election approached it did appear, Sam, that you had fallen victim to the “Look at me” syndrome. ‘Me and Beyonce’, ‘Me on at 11:40′, ‘I’ll eat a bug’. God only knows where that silliness would have led if you hadn’t missed the target so miserably.

    • Wild Irish

      I don’t think that at all. We had no polls that incorporated the effect of the FBI’s latest announcement, two days before the election. And we still don’t know the differences between actual votes and exit polls for the states with big differences from published polls. There is still a very real possibility that gutting the VRA allowed suppression of a large number of minority votes–not that they didn’t try to vote, but that they were unable to.
      All of this needs to be sorted out before you start saying anyone fell victim to a syndrome.

    • BrianM

      Princeton no doubt took a bad hit here …hard to recover from such an off the mark prognostication. 538 it would seem won these stakes.

  • EricK

    Sam, I have very much enjoyed your analysis for the past several elections and hope you continue to do it. I admire that you made your code open-source and describe it in detail.

    I am unhappy with the election result, but hope that once the smoke clears you do some analysis of why the polls were so bad, and what might be done about it in future. I know you aren’t a pollster, but I appreciate your insights.

    • SK Platt

      Please, please take this site down. A 99% probability that came *wrong* is categorical *proof* that the methodology of this site is *wrong*.

      It is similar to preaching that evolution is wrong. This site is a disservice to the national dialogue in the present moment.

      Because if we really need to understand what happened, we need to go out and talk to the people in the rust belt.

    • SK Platt

      The polls were *not* off, as some readers are claiming.

      Prof. Wang’s methodology failed to account for the large number of undecided voters even on the last day this cycle (~ 13%).

      And undecided voters that late are precisely the kind to throw a brick.

      The model should have taken this uncertainty into account.

      If nothing else, it should have taken into account non-mathematical things like human psychology. That alone should have brought down the prediction from 99% to something realistic.

  • Emigre

    Dr. Wang please do not throw in the towel but continue this project when your valuable time allows it.
    Somewhat selfishly I am looking forward to your post mortem analysis to find out why almost all polls were off so much. … but then
    1. Why is Ann Selzer in Iowa always so accurate?
    2. How did USC/Dornsife pick a panel of 3000 voters that reflected the outcome so well?

    • MPP

      For #2… They didn’t.

      Trump didn’t win the popular vote by 3.2 pts, he lost it.

      They were no more accurate than the other surveys.

      IBD/TIPP seems to have gotten closest with Clinton +1.6

      You’re going to have to look at state level results to find a pollster that truly called it correctly.

  • George H

    Sam,

    Fantastic work. Thanks.

    The polls were wrong. Not necessarily the model is wrong. Hope you can keep this “hobby”. I find this a refuge.

    • YCW

      George, I agree. You can’t fix bad data. I think the next issue for the field is sampling, not modeling.
      Sam, I admire your work and appreciate your composure. I hope you can keep up this hobby too.

    • Mikey

      No, the model was definitely wrong. Complaining ‘The polls were wrong’ shows you do not understand the model and why it was wrong.

  • bks

    Hey, look on the bright side: It could have been Ted Cruz!

  • AA

    I looked at the individual battleground state stats on cnn.com. A few things that stand out are :

    [1] Republican voted consolidated behind Trump. Around ~90% of Republicans voted Trump when only ~87% Democrats voted for Clinton. This could point to the post Comey consolidation of Republican voters.

    [2] Independent voters went for Trump by > 10% margin.

    [3] Women vote didn’t fall overwhelmingly behind Clinton.

  • Chuck

    Just FYI, the NYT election site currently shows that Trump has not, so far, won the popular vote. http://www.nytimes.com/elections/results/president Neither he nor Clinton have a majority. I don’t expect either Trump of his supporters to acknowledge this fact since they have been entirely resistant to facts so far, but perhaps more rational members of his party will want to take that into account.

  • AwK

    Hi Sam,

    Thanks for all your efforts. I agree that is there is no obvious way to estimate the probability of systematic polling errors, although a different way to account for correlated fluctuations might have reduced the confidence levels.

    At this point what would be really interesting is to understand where the polling errors came from, not small fluctuations like NC, but large ones like in WI.

    Any chance you’d provide some analysis of this while we are licking our wounds?

  • nhm

    I almost don’t think that’s good enough…

    As you will well remember, I (and others) argued vociferously for you to allow for increased uncertainty in your model. And you had no cogent argument for a model that incorporated such low uncertainty (you can claim hindsight… the rest of us can claim foresight).

    Yes, the primary reason everybody missed this call was because the polls were flat-out wrong. But you exacerbated the problem by convincing yourself and virtually all of your readers (and radio listeners, and t.v. watchers) of your “correctness”. Hell, after being berated by commenters (and not allowed to respond because my responses were never posted), you had me convinced, too. “I must be wrong, because everybody’s piling up on me, and my responses must be so out-of-line that they’re not even publishable.”

    No… I think your readers deserve more than an “I was wrong. Now I’m gonna go back to the lab.”

    I think we deserve an “I’m sorry. Now I’m gonna go back to the lab.”

  • 538 Refugee

    My area had always been safe Democratic so I admit I slacked. That’s changed. All local races stayed solidly Dem but president and senate didn’t. I just sent an email to the county party. As far as I’m concerned the next cycle starts today.

  • BA

    I think now it’s time for the Wang/Silver feud to end. I think if the predictors come together and try to figure this out we can understand why the polls failed. There may be a value in both the Silver method of following trends and the method of Wang which focuses on the big picture. The answer probably somewhere in the middle. I hope that the American people will someday trust statistics, because it is helpful and works in most cases.

  • Michael Bol

    99% chance of HRC winning was a very big mistake. for some of course in hindsight. I understand though the reasoning behind it and the errors which you have explained. Perhaps you also put your personal feelings/views in the equation?

    • Sam Wang

      Maybe, but as I wrote, the history of polling errors would only reasonably give a probability that was still well above 90%. There’s no real way around that.

      As to feelings…not in the way that you are implying, I think. I set up the calculation back in early summer, and I had to commit to leaving it in place in the closing days. I did have a bias toward over-certainty, which other analysts did not have.

    • Matt Frank

      Sam- I think you are still in denial, which is understandable. Your _model_ failed, and has been invalidated by the data. The polls were a little off, but not as much as you seem to be blaming them for.

      The polls seemed pretty accurate: the national polls predicted a small popular-vote win for Clinton, and … Clinton seems to have thinly won the popular vote.

      I know your model was built only on state polls, but even here, I think (not sure, you can confirm/deny) that your models of systematic polling bias didn’t take into account the massive number of _undecided_ voters that (some, not all) the state-level polls were reporting.

      I think you will need to reject the Bayesian epistemology that led you to believing that you were incorporating valid “priors” into a statistical prediction, and, if you wish to fix your model, you will need to adopt a more “Popperian” view of what statistics can do, as described in Gelman and Shalizi, “Philosophy and the practice of Bayesian statistics”, British Journal of Mathemtatical and Statistical Psychology, 66:8-38, 2013. That is: statisticians build models and then use data to try to invalidate those models.

      You succeeded in invalidating an incorrect model. There’s nothing to be defensive about.

    • Some Body

      @Sam: I think that post of yours had its problems. It only looked on the last four US presidential elections, ignoring earlier races, non-presidential races, non-US races. The relevant universe for assessing the likelihood of systematic polling errors is much larger than n=4.

      That was the basic error. The other one follows from it: You also looked at the size of the error alone, without trying to discern conditions affecting it (reasonable for n=4, but n shouldn’t have been 4).

      Of course, there was one strong candidate for such a conditions: number of undecided and 3rd party voters left. This was raised often by commenters here, and, of course, cited by 538 in their assessments of uncertainty. It should have been a good idea to see how this particular factor correlates with systematic error. But your four examples would have probably given you an atypical sample for that purpose (you had an accurate MM for 2000, somehow, and a larger miss in 2012).

      Finally, even with the four examples you looked at, an error 0f 2.3 pts. (given a 2.6 MM) in one of the four should have alerted you to the possibility that the parameter should have been set much higher.

      Other things being equal, simplicity of design in statistical modeling is a virtue, but other things are not always equal. This time more complexity would have helped.

    • J.R. Mole

      A 90% figure would have been plausible even in the faec of these results. This was not a clearcut, crushing win (or a “mandate”, whatever that may be). Trump lost the popular vote but won enough states by narrow margins. Last I looked, PA+MI+WI together had a margin of about 110K. A good dump of snow in the upper midwest could have at least substantially narrowed those leads.

      For all that I find 538′s model stunningly baroque, it does at least explicitly allow for non-trivial systematic polling error. Accounting for it as described elsewhere on this site would have produced a plausible model that’s simpler and more transparent than 538′s.

      538, as I understand it, also explicitly accounts for undecided voters. From what I see from a quick look, PEC also takes this into account (e.g., http://election.princeton.edu/2004/10/20/more-on-allocating-undecided-voters/). All other things equal, more undecideds should drive both probabilities toward 50%.

      I don’t have time to figure out the magnitude of that effect in the two models, but as a general point I’d think the impact of undecided voters would be highly non-linear in how far the poll results are from an even split. If a race is tied in the polls, and we believe the polls, then everything depends on how the undecided voters split. If a candidate has a 4-point lead it should make a difference whether there are 2% or 10% undecided. With 2% the opponent needs the polls to be off as well a favorable split (or for them to be way off with an even split — not sure how you separate the two effects). With 10% an unusually favorable split would do the trick.

  • Cathy

    Dr. Wang,
    I know this didn’t turn out the way it was supposed to, but I want to thank you for your efforts, and particularly for the tenor of your website. It was calm, witty, and reasoned. You did all of us hangers-oners proud, not to mention yourself and your university. I am sad, but very appreciative.

  • Anje

    I’m sitting here listening to people talk about the fossil fuel ethanol and how we need to go back to oil. “Fossil fuels are ruining our cars!”

  • HomerEb

    I’m as stunned as anyone. I’m not seeing turnout discussed as much as I think it should. Trump looks like he’s going to end up with fewer popular votes than not just Hillary, but Romney12. If he does eke by Romney it won’t be by much. How did Hillary and the Democrats underperform by close to 5 million votes? That’s one of the biggest stories of this election. At the end of the day, people rejected both of these candidates and stayed home. And when turnout is depressed, Republicans have a field day.

  • Wild Irish

    Professor Wang, I am sorry you are bearing the brunt of the disappointment and fear that many people seem to be experiencing. One of the things I have always admired about your website is that your methodology is transparent, and there is interesting and intelligent debate about how to improve it without skewing the results. That has seemed to work for everyone, and the results have been amazingly accurate, until now. One would think that these factors would be taken into consideration, but that seems not to be the case. Please do not be discouraged or take the criticisms personally. Those who criticize you are unable to do this monumental task themselves.
    Now, more than ever, we need someone who is NOT trying to get a particular outcome, and who is not worrying about how many clicks you are getting.
    Sending good thoughts your way, and much appreciation for your diligence and hard work over all these years. Thank you for providing an oasis of sanity among the clamoring voices.

    • Paul Griner

      Yes. Exactly. Calmness. Wit. Reasonable discussion. I suspect these will be in short supply over the coming months and years. So, here’s one more long time reader hoping you will continue your work here

  • Perry

    So…. It seems most of the polling outfits were using 2012 data in order to estimate likely voter/turn out.

    Some qualitative observations: Both candidates were/are very weak candidates.

    Mrs. Clinton has the stronger resume/background, but, I always thought she was a poor public speaker/campaigner. And never seemed to sell any reason to vote for her; other than she’s not Donald Trump.

    Mr. Trump has zero government experience and a questionable business background (among other things). But, he has charisma. Mrs. Clinton seems relatively boring.

    The debates: The media (and some of my colleagues) seemed to dismiss Mr. Trump’s debate performance. It seemed they were quick to declare Mrs. Clinton the winner.

    That was not what I observed. I thought Mr. Trump did reasonably well at selling himself in those debates (and yes, he lacked substance/depth. But his target audience doesn’t care).

  • Scott

    I’m dying to know how the 9 day slump during the FBI letter debacle affected things – and also how the millions of insurance company letters that went out in the final days telling people their i premiums were going double and triple (I got one myself and it’s terrifying.

  • JPI

    I had this weird dream last night that I ate a bug.

  • Chuck

    I don’t know if Sam will still delete Lichtman’s 13 keys posts. But it appears his model was wrong the other way.

    The 13 keys predict the popular vote winner. She pulled ahead it appears most of votes left are in CA Wash Or with CA allowing mail in marked day of election receivedd within 3 days.

  • DW the Bayesian

    It could be that the polls had an implicit bias, not properly taking into account who was going to vote. This election had fewer votes for Trump than any other winner since 2004.

    If you look at percentages only when asking people who they are going to vote for, it has to be convolved with the question, “Are you actually going to vote?”

    Perhaps, the percentages did reflect “likely voters” and I am wrong. But, when pulling polls off the web, it is hard to take into account other priors that are important.

  • Paul

    Is it possible that some voters upset with the establishment are so upset that they won’t answer calls from pollsters, creating a bias in the polls? If Trump voters are difficult to get on the phone then the polls are fine in themselves, but they’re not a pure random sample.

  • Ken L

    Sam,

    Thank-you for all the interesting and thought provoking work.

    I have enjoyed the politics & polls podcasts and found them very informative.

    Your work on gerrymandering is extremely important.

    Please keep up the outstanding work.

    • Joeff

      Your work on gerrymandering would have been very useful to a Dem-dominated Court. Now, not so much.