Protected: Two ways to estimate primary outcomes without polls (transcript)
This content is password protected. To view it please enter your password below: Password:...
Election thread #2
11:44pm: The business about 65%, 91%, 93%, 99% probability is not the main point. The entire polling industry – public, campaign-associated, aggregators – ended up with data that missed tonight’s results by a very large margin. There is now the question of understanding how a mature industry could have gone so wrong. And of course, most of all, there is the shock of a likely Trump presidency. I apologize that I underestimated the possibility of such an event.
I agree w/@NateSilver538 that there was high uncertainty, much more than I assumed. Median polling error 4% Presidential, 6% Senate so far.
— Sam Wang (@SamWangPhD) November 9, 2016
The American people have rejected the person they thought was qualified in flavor of the one they thought was not.
— Greg Dworkin (@DemFromCT) November 9, 2016
11:12pm: Using the projections of the NY Times, Donald Trump is outperforming his pre-election polling margins by a median of 4.0 +/- 2.6 percentage points (the 8 states in the Geek’s Guide). In Senate races, Republicans are outperforming by 6.0 +/- 3.7 percentage points. A five-percentage-point polling miss would be a tremendous error by modern polling standards. Undecided or minor-party voters coming home to Trump? Shy Trump voters? I don’t know.
GOP polls weren’t predicting this night either. Senate, House, Gov sources I talked to all expected Clinton would win per their polls
— Jessica Taylor (@JessicaTaylor) November 9, 2016
10:38pm: At the Senate level, the polling error is looking pretty substantial at the moment, maybe 5 points toward Republicans. A polling error of this size would be the largest on record, at least in a Presidential year. I was wrong to downplay this possibility.
We still have to see what will happen at the top of the ticket. But obviously, with a Meta-Margin of only 2.2%, an equally large across-the-board polling error at the Presidential level would suggest a Trump win of the Electoral College.
Trump owns the party whether he wins or loses
— Gideon Resnick (@GideonResnick) November 9, 2016
9:31pm: NYT presidential tracker showing things very close. Looks like a late night. And perhaps bug cookery for me.
Topics:
Time to retire the concept of opinion polling. It had a good run, but it’s going the way of the telex machine and telephone party lines.
Could the USC poll be the biggest winner (outside of Trump)? Their model was criticized by nearly everyone, but they might just be right.
No. They missed the popular vote by about as much as everyone suspected. They didn’t try to predict the EC.
Dear Sam, I think your next analysis should be 1.) whether there was evidence the election was hacked, and 2.) a state-by-state examination of the effect of various types of voter suppression in the the early vote, and on Election Day.
Yes, I am serious. If nothing else, it would be good to eliminate these possibilities.
I have a really hard time believing the polls could have been this wrong – and just as hard a time believing there could have been a last-minute, 5-point swing to Trump. That would be unprecedented in modern times; at the outset I just don’t buy it.
I don’t either. Perhaps I’m just in shock. Has the nation’s taste and character so degenerated? Sam, I have followed your site since 2008 and value it greatly. I simply cannot believe the numbers I’m seeing on my tv screen. If you have the heart, I hope you take Olav’s suggestion. There is something very awry here.
Hacking a US federal election is hard – there are too many disparate bits. But lots of 15th Amendment violations and a few targeted hacks – say Miami, Raleigh, Philadelphia, Detroit, Milwaukee, maybe Cleveland just in case – could be worth 5% in a few key states while only being 1-2% nationally. Add some meddling from Putin and a violation of the Hatch Act or three and you might just end up with a constitutional crisis and a fascist dictatorship. I wish that didn’t seem so believable.
So, the thing you all were laughing at Trump about, “the election is rigged” is the very first thing you think of to say after an incredibly widespread but consistent surprise across huge geographic areas and varying types off voting technologies. You should be ashamed of your asinine paranoia.
Just sitting here with my head in my hands.
Hillary Clinton has just posted a semi-concession tweet…
https://twitter.com/HillaryClinton/status/796169187882369024
Women did not vote for Hillary in big number. Hm… seems like I have made the wrong assumption about women my whole life. I will have to start being super aggressive towards women because all those women want respect talk is clearly BS.
New Year Resolution 2017: I’m gonna go start grabbing all the pussies I want. Don’t even bother asking for permission.
In the end, women remain to be the unknown and unpredictable, but at least now we understand women a little better based on how they vote and reward men. They may tell you that they want men to respect them and all those BS… well, not really.
The point is: what women say (in this casem what they told the pollsters) vs what they actually do… turned out to be very different. It’s finally become clear. Wow. The Red Pill was right on.
Tapper: “If Trump persists, it’s going to put polling industry out of biz, it’s going to put voter projection industry out of biz.”
Me: “If Trump persists, it’s going to put voters – and voting – out of business.”
at the end, the Clinton coalition is weaker than the Trump devotees. White college educated women, independents under performed the Clinton. Black and millennials under performed for Clinton.
Just killed me to see a sense of Brexit again.
After the 2015 UK general election produced results significantly deviating from the pre-election polls, the British Polling Council and the Market Research Society jointly sponsored an independent inquiry into what might have gone wrong. (The report’s readily available online.)
Is there any chance of something similar happening in the US?
The NYT now projects a >95% chance of Donald Trump winning the presidency.
538 projects 58% chance of Trump winning, and 40% chance of Clinton winning.
She needs to flip back Pennsylvania, Winconsin and Michigan from Trump. She needs a miracle to flip back all three.
Awhile ago, I came across the “Primary Model” by Professor Helmut Norpoth. Everyone else was forecasting a Hillary victory, but Norpoth alone forecasted a Trump victory:
http://primarymodel.com/
Seems like he alone turns out to be correct. He will be a celebrity (and replace Nate Silver, Sam Wang, etc.,) after this election.
A lot of Democrats and pundits are having buyer’s remorse. They are rationalizing that Bernie Sanders, if he had been the nominee, would have won this election. However, based on the Norpoth’s Primary Model, Donald Trump would have defeated Bernie Sanders anyway.
I suspect that he was lucky. His predictions are very wonky for some years. In 2008 he had Clinton beating Obama in the primaries (wrong) and then Clinton beating McCain by half a percent, and Obama winning by only 0.1% (wrong).
The magnitude of the win he forecast for Trump is also wrong. Clinton won the popular vote, for instance.
He doesn’t have any documentation of his methods. Dude seems like a crank.
Wisconsin was +6.5 Clinton RCP average.
Trump +4.0 now.
At least here in NJ we will be seeing the back of Chris Christie as he leaves for DC.
I was wrong. This really is the end of the world.
As for projections… GIGO. If the polls are all wrong, aggregating the polls can’t possibly get it right.
Just eyeballing it here, but it appears Republicans that claimed they were going to vote for Johnson may have come back home to roost while Democrat leaning Johnson voters got complacent. This entire thing is feeling like 2000 on steroids.
Perhaps. Trump is outperforming Romney in Republican leaning counties. The differences cannot be just explained by Johnson Republicans coming back home.
if polls are anonymous, how can there be a silent Trump voter? How can polls be off by so much?
the people that said character mattered were lying
Dare I say it?: Allan Lichtman waa right, I guess.
Don’t say because you might not be right. Lichtman’s forecast is for the popular vote not the electoral college. It’s a technicality but it enabled him to preserve his perfect record in 2000. At least this is how I understand it
Sam – could the FBI letter and 9 days of hell it put HRCs campaign through account for this?
Clinton’s team felt they had a smaller lead than polls were showing the entire time apparently, but they had to show confidence publicly.
Something was clearly off with their internal data because Clinton never made a stop in Wisconsin. It’s clear from Clinton’s trips to Michigan and Pennsylvania after Comeygate that they sensed a problem.
Sam is right about this polling miss being more than just a simple modeling error.
There is something profound going on in the country that we obviously did not, and still do not understand.
Polls are only polls. Math about polls is itself.
People kept saying Trump _couldn’t_ win.
To which I retorted, of course he can win.
He is the GOP nominee.
Juliet– O shut the door! and when thou hast done so,
Come weep with me; Past hope, past cure, past help!
Friar Laurence– Ah, Juliet, I already know thy grief;
It strains me past the compass of my wits.
Sam Wang’s *wrong* diagnosis of “Could the polls be understating Trump’s chances?”
http://election.princeton.edu/2016/10/14/motivated-reasoning-strikes-again/
So, now you *cannot* say that it was unexpected. A good statistical estimate is supposed to take this into account. And Nate Silver did just that, which is why he was more circumspect throughout.
I see a path forward.
Marijuana wins big on election night .
https://www.washingtonpost.com/news/wonk/wp/2016/11/08/medical-marijuana-sails-to-victory-in-florida/
It’s not a sideshow. Different models made very different assumptions about how likely a global polling error of this sort would be, based on historical data and judgment calls. This was a source of controversy before the election. The models that predicted a virtually zero chance of large polling error now look like they were overconfident.
Not a good night for your probability computations vs. 538.
Just goes to show that even the most sophisticated model gives garbage predictions if the input data is garbage.
Brexit on ‘roids.
Like Trump promised…he finally turned out the missing white voters.
We never thought about a Trump wave because it seemed so impossible to us intellectual elites.
and there were shy Trumpers– as we see now.
character became a nonissue.
Trump actually won less votes than McCain won in 2008. There were no extra white voters. Democratic turnout was just pathetic.
This is the 2nd time in 16 years the popular vote doesn’t match the electoral college. There needs to be a serious conversation about this.
Where do you see the popular vote? Republicans are poised to retain the House and Senate. It looks like pretty comprehensive defeat for Democrats.
Twice in 16 years ain’t that bad, really.
This is a feature, not a bug. We are a Republic, calling America a democracy is a misnomer: we are a representative republic and things like the Senate and EVs were put in to keep one faction or section of the country from dominating the others: if purely popular vote were taken into account a candidate could rack up huge majorities in New York and CA and bulldoze the rest of the country. Also, people talking about not getting it changed while the GOP controls the House and Senate need to realize the Constitution itself would have to be changed, a very dim possibility. The GOP holds the vast majority of statehouses and state legislatures, which would be vital in a Constitutional Convention.
The alternative is Trump trying to shore up vast majorities in the South and Texas while hillary tries to max out urban areas and many states and sections of the country become irrelevant. Also, the potential for fraud is much greater. Right now you have to rig 50 different elections to rig the presidential: with popular vote you could vote-pack in just one or 2 states (or even cities!) and steal the election.
The Electoral College is a good buffer against these sorts of things and the 2 Senator per state and EV allotment protect the smaller states. It’s a fine system unless your side loses, then all the sudden you get people complaining and wanting to change it. It’s there for a reason, and the reasons for keeping it are more compelling than not.
A number of blue states have signed on to the national popular vote interstate compact, but red states aren’t (probably because conservatives tend to perceive the system benefits them). If 270 electoral votes worth of states sign on, the electoral college will become functionally meaningless.
So the angry white guys took their country back. Too bad they gave it to a sociopath.
I don’t think it’s just the “angry white guys”. Whatever unseen/unknown factors that have propelled Donald Trump to winning the presidency, they are beyond the scope of Sam Wang’s hobby in statistics. Personally, even though I prefer Clinton over Trump, I can see some benefits to having a Trump presidency. Maybe other people rate those benefits as being more important than I do.
BTW, we have had at least a few presidents in the past who were sociopaths.
Does this prove that Americans are far less educated than was assumed? To be angry is one thing, but to be so angry that you lose your ability to reason and elect a gorilla to throw the whole system out the window is stupid. That it’s for the position of most powerful person on earth is worse than stupid, it’s dangerous. I am honestly terrified.
I’m sick at heart, and very scared too. People are going to suffer.
I can not accept that an entire industry, a ‘mature industry’ as Dr Wang says, could be so wildly and across the board, wrong. Is there an alternative possible explanation? Was the election hacked?
I can not accept that a man who has a court date for racketeering, and another for child rape could be elected president.
yeap, you got this rught.
Isn’t the big problem, clearly, the fact that polling relies on phone calls and too many people don’t want to answer the phone and talk to people they don’t know? How can polling possibly be accurate when nearly everyone has CallerID?
Thanks for your efforts in explaining things for us.
Cheers,
Scott.
I strongly believe sexism has a whole lot to do with these results. Despite the gains women have made over the last few decades sexism remains deep and broad. It’s complex and insidious.
Working white women without any college education voted for Trump. That really isn’t surprising. There is a rather strong affinity for traditional values and the traditional role of women– especially married. Among African
Americans and Latinos with whom Clinton performed there is a strong sexist strain among these groups as well.
I base this mostly on non empirical observations and experiences I have had working with African American and working class and lower middle white women.
There are other factors involved of course. But blaming it on polling error is like blaming a loss in sporting event on statistical error
Sam: You’re a brilliant scientist and a straight shooter. But this election has torpedoed your and every aggregating pollster’s credibility — forever. Shut down the site and don’t torment yourself with it anymore. I won’t be paying attention to polls anymore for however long I live.
We expected LV screens would undercount Hispanic voters and other first-time voters. From commentary tonight, sounding like Republican voters (largely but not exclusively) in Rural districts who boycotted Romney in ’12 were discounted and turned-out hard for someone who didn’t sound over-educated and elite, and African-American Obama voters were over-weighted as if assuming all ’08,’12 voters remain LV.
The business about 65%, 91%, 93%, 99% probability is not at all a sideshow. One of the biggest mistakes a forecaster can make is saying something has a >99% chance of happening and it not happening.
Your method is systematically overconfident and this result is proof that it needs to be reworked.
The irony is we talked a possible Clinton wave. Now we talked a Trump wave.
The one good thing I can see coming out of this is that I fully expect the Trumplicans in the Senate to nuke the filibuster once and for all. The Democrats would never have the guts to do it, but the Trumpicans will. No way are they going to let a minority stand in their way!
People aren’t electrons….sorry folks. They can do whatever they want.
I am curious to what extent vote exchanges completely threw off the polls. If votes were being exchanged in large numbers between states there could have been a “smoothing effect” whereby extreme leads could have been narrowed. Also, since the vote exchanges were a bit like “the prisoner’s dilemma”, I wonder if one-sided vote exchanges (one voter reneges, while the other doesn’t) also may have skewed the polling. Are there reliable numbers for participation in vote exchanges?
Elsenet, someone mentioned the way Strom Thurmond consistently outperformed his polls; people just weren’t willing to say, not even in private to a pollster, that they would support a racist. I wonder if something similar was at work here?
Nothing you could have done, the error in polls was something we have never seen before and I feel like there was literally zero data that pointed toward this outcome. Don’t let this eat you way too much and get back to the drawing board and make it better next time! If you are looking for a good country to move to, Finland is not too bad despite the weather!
One point getting lost tonight is that Clinton will probably win the popular vote by at least 1 million votes. After 2000, there was talk about getting rid of the electoral college. Now it’s time to do more than talk.
Not while we have a Republican President, a Republican House, and a Republican Senate.
It is totally appropriate to focus on the % chance each forecaster picked. The most appropriate way to score probabilistic predictions is through Brier scores, and in that regard you were crushed by 538’s picks.
How about the logarithmic score instead? ln(1-p) where p > 99% Brier is useful, but not “most appropriate” for all purposes. For pure shaming, the logarithmic score is better. By the way, 538’s “pick” was Clinton, but they were more uncertain about it. They are not celebrating either.
This is not a reasonable result. It does not make sense to me that there could be such a large systematic error.
I suspect it’s the same factor that caused the systematic error in pre-Brexit polls. It’s political incorrect to tell pollsters an answer that would imply, “I want those migrants/immigrants/Mexicans/Muslims OUT of my country.” However, that wass what they were actually voting for when no one else was watching.
Watching the CNN trying to rationalize tonight’s outcome. They still do not get it. It is really NOT about the economics or his “honesty”.
Ultimately, it all comes down to one thing: the “Wall”. It is about illegal immigrants. It is the very same reason why British voted for Brexit: people want migrants/Mexicans/Muslims OUT.
Here is my observation: Donald Trump has flipped flopped and refocused his messages a few times. However, the “Wall” (which encompasses his Muslim ban) has been the ONLY message that has stayed consistent throughout his campaign.
Oh boy, am I glad I am not a Mexican or Muslim.
“The business about 65%, 91%, 93%, 99% probability is not the main point. The entire polling industry – public, campaign-associated, aggregators – ended up with data that missed tonight’s results by a very large margin.”
They missed big, but with respect to probabilistic models, a main point is most certainly the extent to which they appropriately factor in possibilities of polling errors. It’s unreasonable to expect probabilistic models to have factored in possible errors of this magnitude. However, it’s perfectly reasonable to say that 99% represent massive methodological flaws in the model’s treatment and use of polling data. This might not be “the main point” about this season of data collection and aggregating, but it’s certainly a main point with respect to methodological adequacy and performance. It’s not personal and there’s no need to apologize.
Could it be possible that the polls were rigged? i.e., the companies and organizations that did the polling inflated Hillary’s numbers as an attempt to drum up her support? After all, those companies and organizations do have ties and invested interested in Hillary winning. Maybe the same thing happened to Brexit polling.
I know that seems unfathomable… but, who knows. Maybe we were the ones who were too naive to think that could not possibly happen.
I am just stunned and speechless. Scared too,
I just had a huge payday.
2016 was not going to be 2012. This election was easily predictable. Pollsters across the board over sampled democrats by a wide margin and under sampled independents. If you stripped the polls of this bias, Trump lead the race the almost the entire time and at the end had a lead of about 4 points.
Dont believe me? Head over to huffpolster and look at the sub-populations. Open the tab and tick the individual likely voter categories. When averaged together, Trump was the clear favorite.
In addition, messing around with fivethirtyeight’s interactive map, lowering black voter turnout and turning up noncollege white turnout almost exactly matches the electoral map as it stands.
Then why did Clinton win the nationwide popular vote?
Clearly, many public polls were off.
But your suggestion that with a better estimate of the composition of the electorate Trump lead all along by 4% is also just as wrong.
Trump lost the national vote by 1.5%
If people are looking for recent examples of polling failures, then they should add to their lists for study the British Columbia 2013 Provincial elections which missed by double digits. http://www.ekospolitics.com/index.php/2013/05/an-unapologetic-analysis-of-the-bc-polling-debacle/