Princeton Election Consortium

A first draft of electoral history. Since 2004

All The Reasons You Doubt Polls: Motivated Reasoning Strikes Again

October 14th, 2016, 8:00am by Sam Wang


Every Presidential election, it happens. People on the side that is heading for a loss find ways to disbelieve what polls are telling them. This year is no different.

First, a tiny dose of cognitive science. Our brains are really good at letting in information that agrees with our prior views – and we look for reasons to reject information that is disagreeable. In a complex media environment, this tendency is deadly. It probably underlies our deep political divisions: getting the agreeable information is very easy. Witness the echo chambers in which dumps of fairly anodyne email from Hillary Clinton take on sinister significance.

People are the same way when they interpret polls. Two past cases come to mind:

  • In 2004, state polls were dead-on in giving a snapshot of the close race between Senator John Kerry and President George W. Bush. However, Kerry supporters suggested that undecided voters would break toward the challenger. This was a pretty small break, but it was enough because the race was so close that year. As longtime PEC readers know, I made this error.
  • In 2012, “poll un-skewers” on the Republican side took it upon themselves to correct polls that they felt were demographically unbalanced. The king of the un-skewers was Dean Chambers.

This year, I have heard multiple possible objections to interpreting polls at face value (for example, see this PEC comment thread). Here they are, with my answers.

1. Brexit. It is said, wrongly, that polls missed Brexit. However, that is not true. Pre-election polls indicated that Brexit was too close to call – and there were 9% undecideds. It was pundits and conventional wisdom that failed.

2. The 2014 midterms. This is fair – on average, there was a 5 percentage point error that year. However, as I have analyzed, it is generally the case that in midterm years, which represent a low-turnout condition, polls are terrible. However, general election polls did extremely well in 2000, 2004, 2008, and 2012. I wrote an article about it.

3. “We didn’t think Trump would win the primaries, either.” Erm…speak for yourself. Once again, pundits were wrong – but a poll-based approach was correct before the primaries began, and continued to be correct all the way through the primaries. Go through PEC’s archives and you’ll see.

4. Known unknowns. Will Sanders supporters show up to vote for Clinton? Will Trump bring new voters out of the woodwork, particularly people who don’t necessarily answer the phone for pollsters?

Most of these reasons are already captured in polling data, at least at the state level (national polls aren’t as good). For example, if Sanders supporters have come home to the Democratic nominee, they will be captured in polls. The same is likely to be true of Trump supporters; for example, primary-season polls did very well in predicting Trump support.

The most unmoored form of this argument is that somehow, tons of Trump supporters are simply not captured by polls. Evidence contradicts this speculation. This has not been apparent in voter registrations, so there is no clear place for these voters to come from.

Unpersuaded? Maybe you like that one poll from USC/Dornsife, which has serious weighting problems. If you want to imagine what would happen if your side got a few percentage points, the “Clinton +2%” and “Trump +2%” links at right will show you that.

5. Undecideds and Gary Johnson/Jill Stein supporters. In the past, nearly all of these voters end up supporting one of the two major candidates. For example, Johnson and Stein supporters combined got less than 2% of the vote in 2012. What will they do this year? Below, I estimate the impact.

Currently, undecideds and Johnson/Stein supporters constitute about 14% of voters. Drew Linzer has pointed out that undecideds/minor party supporters are unusually high this year, about 6% ahead of 2008 and 2012. This is a legitimate source of uncertainty about the eventual outcome. However, there is information about how these voters will eventually fall.

Undecided voters usually break somewhat evenly (sorry, 2004 Sam!). Data from SurveyMonkey suggests that Johnson supporters break about evenly between Clinton and Trump, while Stein supporters tilt strongly toward  Clinton. This is consistent with many state polls that show Clinton doing the same or slightly better when the matchup is Clinton/Trump compared with Clinton/Trump/Johnson/Stein. So the net expected effect is, on average, slightly toward Clinton*.

The topic of undecided/minor-party support requires more unpacking. I would rate it as the most legitimate concern about prediction. In my estimation it alters the probability by a tiny amount at most.

Anyway, if there’s some hidden reason why polls are all wet, it hasn’t come up yet. If you’re a Trump supporter, it would be more productive to focus on downticket Senate and House races, where your side has a better chance of surviving November.

>>>

*This can be quantified. Based on recent data, let us assume 6% undecideds, 6% Johnson supporters, and 2% Stein supporters. By November 8th, they might break as follows.

Undecideds: Let’s say these voters end up somewhere between 4.5%-1.5% favoring Clinton, to 4.5%-1.5% favoring Trump. Using the rule of thumb that the range is 4*standard deviation, this leads to a net change of 0.0 +/- 1.5% (mean +/- SD).

Johnson voters: Assuming 1% stay with Johnson, the same logic leads to a net change of 0.0 +/- 1.3%.

Stein voters: Assuming 0.5% stay with Stein and an average 3-to-1 split for Clinton, the effect is an increase in Clinton’s margin by 0.7 +/- 0.5%.

The overall combined change is a net increase of margin in Clinton’s favor by 0.7 +/- 2.3%. Given current conditions and based on the uncertainty of 2.3%, in 3 out of 100 cases enough voters would switch to Trump to close his current deficit.

Tags: 2004 Election · 2012 Election · 2016 Election · President

42 Comments so far ↓

  • Runner

    “Unskewed Polling” was debunked in 2012. It has no applicability in 2016.

    Will the 98% Bayesian win probability reach 100% prior to Nov 8?

    • Amitabh Lath

      There is no way to reach zero uncertainty using measurements (like opinion polls) that have non-zero uncertainties themselves. I suspect we will stop paying attention if it reaches 3 sigma (99.7%), but I doubt that will happen in the remaining three weeks.

    • Five Tool Player

      I also agree with AL above. A statistic of one hundred percent certainty of a Clinton victory is mathematically unattainable. There is always the possibility that tapes might emerge showing Hillary indiscriminately shooting puppies on the White House lawn, engaging in sacrificial satanic sex rituals on a cruise ship off the coast of Baffin Island, or flipping off a bunch of Kindergartners on their first day of school. As the Infinite Monkey Theorem suggests, a chimpanzee just might reproduce a Dostoyevsky classic, given enough time.

  • PatR

    Sam–can you talk about how likely voter screens are constructed, and whether they are subject to change over time? Is there any probability that the weights are off because of changes in voting behavior? Is this something you’ve already discussed somewhere else?

  • Howard Appel

    Bill, re: Romney internals:

    I also understood that the Romney internals were very bullish on his chances. IIRC, the reasoning behind that was that they didn’t believe the actual numbers they were seeing and took the position that their numbers overstated the turnout for African-Americans and the other components of the Obama coalition. They didn’t believe that they would turn out in the same numbers or the same effect as in 2008. President Romney was wrong.

    • coolchien

      I recall reading recently some of that internal polling data. It seems that it’s not internal polling alone–those data still had him losing Ohio, Pennsylvania, but might have him winning Florida. He counted on the ephemeral momentum, as he was within a few points in both OH and PA, but seems to be rising. I just can’t believe why that was reason enough not to write a concession speech–PE guys shouldn’t be big betting men.

  • BenD

    As Ebeneezer commented, voter turnout has some folks concerned (including me). Sam, do aggregated poll predictions lose much accuracy as turnout dips or rises? My guess is it’s a wash, but I am not so much worried about chance variation due weather and such, but active voter suppression. History probably can’t tell us much about that, but I would be interested in hearing your thoughts.

    • Sam Wang

      In midterm years, poll accuracy takes a giant dive. In Presidential years, polls hit the target pretty well. See my American Prospect article on the subject.

  • TheOmnivore

    You left out bogus “fake polling memos” which are (ime) obvious parodies but 1000′s of people believe them. See: http://www.dailykos.com/stories/2016/8/24/1563471/-Pathetic-Right-wing-misinformation-or-the-best-Left-wing-troll-ev

  • Scott J. Tepper

    Nate Silver has a new article up about watching New Hampshire for a Trump comeback.

    Has he defended his claim yet that this is a volatile election? The only place it seems to be volatile is on 538.com, based on click-bait headlines.

    Sad considering what Nate used to do. His site is now a mish mash of pseudo science, sports, pop culture and concern trolling over the election.

    • Matt McIrvin

      On the face of it, speaking literally, Silver is right; a Trump comeback would manifest there. It’s also a good early state to watch on Election Night to see whether projections like PEC’s are going to be on target.

      All the article is lacking is any reason whatsoever to believe that such a comeback is actually imminent.

    • Charles

      In all fairness, he couched 12 ways from Sunday. He didn’t say it was even plausible. He began with by saying if you are looking for any sign of a comeback, look at NH. Not that it was likely, probable, or even inside the tail. It was little more than a mind exercise.

  • Richard

    Sadly, a large fraction of the electorate believes in outlandish conspiracy theories and rejects many well-verified scientific results wholesale. That coupled with the “echo chamber” bias Sam describes leads to many people thinking the election outcome will be very different than what polls clearly indicate, and if not it’s because the election was rigged. It’s ironic since there are systematic biases in the voting system introduced by the electoral college and gerrymandering, which favor the side most inclined to think the election is rigged against them. Sigh …

  • Just Saying

    Why would any Trump supporter bother to vote in a “rigged” election? If it’s rigged, then Trump can’t win, but by voting, they are endorsing the election’s legitimacy. Bit of a conundrum.

  • bks

    Statistics have a liberal bias:
    But that’s not the depressing part of the poll’s findings. It’s this: 25 percent of Americans “completely distrust the economic data reported by the federal government, including statistics like the unemployment rate, the number of jobs added, and the amount of consumer spending.” Perhaps unsurprisingly, that number soars to 48 percent of Donald Trump supporters compared to just 5 percent of Hillary Clinton supporters.
    http://www.politico.com/tipsheets/morning-money/2016/10/depressing-poll-of-the-day-216855

  • William Ockham

    I am going to quibble over the analysis of undecided/third party voters which vastly overstates the potential impact of this group. Let’s start with the polling concepts of registered voters and likely voters. Registered voters is a term of art that is shorthand for poll respondents who self-identify as being registered to vote. We know that poll respondents over report voter registration.

    Likely voters are a subset of those “registered” voters and that subset is chosen by the pollster. For U.S. presidential elections, the accuracy of polls is clearly improved by a good likely voter screen. However, there is research that indicates that passing a likely voter screen doesn’t mean that those respondents are more likely to vote than other “registered” voters. Instead, “likely” voters’ preferences simply track the preferences of the people who do show up better than the larger group.

    The problem, from my perspective, is that nobody votes for “undecided” in a U.S. presidential election and likely voter screens consistently overestimate the performance of third-party candidates. The support for third-party candidates and the “undecideds” is consistently closer to the the registered voter results than to the actual election outcomes. To me, that suggests that those likely voter screens are consistently failing.

    Having said that, I realize that it still might be difficult to improve the likely voter screens. I believe that there is a mostly overlooked clue to how it should be done. The state-by-state polling in the last presidential election showed a similar issue in non-competitive states. The loser’s support was consistently overstated. After the election, if the same trend occurs, analysts should consider carefully how to discount the support of “likely” voters in a U.S. presidential election backing candidates (including “undecided”) with no chance to win.

  • Ebenezer Scrooge

    A lot of people in this thread are mentioning something that Sam didn’t address: voter turnout. Will Republican voter suppression be better this year? Will Republicans with some reservations about Trump stay home? Will Latino voters show up disproportionately? Will black voters stay home disproportionately? Young voters? The unprecedented difference in the parties’ GOTV efforts?
    These are all small effects, and they point in opposite directions. But I wouldn’t be surprised if Hillary does 1-3 points better than the best eve-of-election polls. I’d be shocked if things go the other way. As Sam says, there are few signs of a Bradley effect.

  • Don S

    Any thoughts about the possibility of Likely Voter screens being off?

    I am guessing an overestimate of the likelihood of Trump voters coming out based on reported intent and an underestimate on the HRC side. But LV screens do seem to be having to make some best guesses with somewhat unusual voter shifts.

    • Bernd

      My favorite nightmare scenario is nerdy milennials not showing up to vote because they were hanging around too much on sites like this one. After all, science told them that the election wasn’t going to be close …

    • Matt McIrvin

      Political junkies who read election prediction sites are, generally speaking, people who vote. I don’t think you have to worry about PEC-induced complacency.

      (Though Nate Silver makes a big deal out of not voting to preserve his neutrality. I wonder if he’s keeping with that this year–though he posts a lot of clickbait about Trump’s chances, he has explicit contempt for Trump.)

  • Ed Wittens Cat

    Again, this is asymmetrical.
    Republican base is far more anti-intellectual, anti-science and anti-mathematics as a rule.
    “Common-sense” leads them to believe in mythology and magical thinking.
    Like the mythical missing white voters.

  • Peter T

    Is there any systematic analysis of things like voter registration and early polling? I’ve seen snippets where the GOP seems to be in big trouble on both (eg new Dem registrations outrunning Republican ones by 10 to 1 in Florida, large fall in GOP early voters in North Carolina). These might give indications of coat-tail effects.

    • JamesE

      The latest right wing meme I’ve seen while perusing conservative blogs is that 100,000 Democrats in Pennsylvania have re-registered as Republicans.
      http://www.thegatewaypundit.com/2016/10/100000-pa-democrats-switch-gop-240000-new-voters-join-pa-republican-party/

    • David Fry

      Yes, we have definitely seen many registered Democrats who haven’t actually voted for a Dem in years switching their affiliations. This is documented in PA, NC, GA, FL, and a I presume elsewhere. But the stat that Peter T cited was for *new voter* registrations and the Democrats do seem to have an advantage there in FL at least.

    • Matthew Coons

      Michael McDonald from the United States Election Project does a good job of reporting on early voting statistics and comparing them to previous elections. He just posted an article to the Huffington Post about an hour ago summarizing what we know about early voting so far. Basically he said Democrats appear to be doing better in Florida, North Carolina and Virginia than they did in 2012 and Republicans appear to be doing better in Iowa and Ohio.

  • Chuck

    The only thing we really can’t figure in—because it actually is unprecedented, at least in my lifetime—is the possible influence of voter suppression by red states and voter intimidation by armed Trump supporters.

    This is the first Presidential election in my lifetime (I’m 68) in which the Republican candidate was openly encouraging his supporters to gather in force in “certain neighborhoods” to make sure the Wrong Kind of People aren’t voting. I have no idea whether that will have a measurable effect—or any effect at all, for that matter. But it does worry me simply because it’s new.

  • James

    I’m pretty sure theories that one candidate has a more motivated base and will therefore outperform the polls usually turn out to be wrong, but I’m still going to guess that this year, Clinton will outperform. While Trump has a small number of people that really love him, most people that support him are still not very enthusiastic. Clinton isn’t too popular, but her supporters are terrified of Trump.
    And then there’s the “ground game.” Clinton has strong local organization, and Trump has almost none.

    • Suvro

      I am in Los Angeles (Pasadena), and got two calls, and one email from the Clinton campaign – to go GOTV in Nevada.
      Unfortunately, I cannot do it on the dates they proposed because of other commitments.
      What I see is a much better organized effort from the Clinton campaign, than in previous election cycles.

    • Greg Gross

      Same here. I know that GOTV is not viewed as being reliable or impactful here, but nevertheless: I had a Clinton GOTV person armed with an iPad show up in my mostly Repub neighborhood a few days ago. First time in the almost 15 years I’ve lived here that that ever happened. The GOTV person also encouraged support in the PA senate race (McGinty).

    • Michael Hahn

      Early voting starts here in Georgia on Monday. Yesterday I received a call from the local Democratic Party asking if I was going to vote, if I was going to vote early, and, of course, if I supported HRC. No big deal …… except that this is the first time in my 30 year residency in Georgia that I have been called!!! Ground game ON!!

    • Tom Benjamin

      This is one of the questions that I think will be resolved this election. One of the only niggling doubts I have about Sam’s work is that GOTV efforts appear meaningless. The obvious advantage enjoyed by the Obama campaigns is entirely reflected in the polls. All Obama GOTV did was increase Democratic likely voters.

      Or perhaps all a GOTV effort can do is make a minor difference because both sides try hard to get out the vote. Even an obvious advantage produces a difference that is too small to show up.

      This year? We have the best ground game ever – according to reports the Clinton campaign has a profile on every voter in the battleground states – against no ground game at all.

      Clinton is spending hundreds of millions on data, identifying her voters, and getting them to the polls. Trump is spending next to nothing.

      This difference won’t move any needles this month? It is all baked in? What does it mean if all that money, that technology, that effort is worth nothing?

    • Sam Wang

      I would say we’re about to find out what unopposed GOTV can accomplish.

  • Scott J. Tepper

    Apparently Trump’s “internal” polling (adding some trend lines to the USC Dornsife poll) let him know he’s WINNING! http://americanlookout.com/trump-internal-polling-analyst-i-couldnt-be-happier-charts/

    You, Sam, and all the other pollsters who don’t agree with USC Dornsife, are making “smear attacks.” I knew it!

    Enjoy the link.

    • Chuck

      Did Dean Chambers finally find a job, then? (:-)

    • John H

      These lines-on-graphs look suspiciously like the ones used to justify the perenial predictions that the economy will collapse in the next few months that I’ve been seeing for at least ten years. All you have to do is cherry pick a few local highs/lows, draw a trend line into the future, and crown yourself king of the prognosticators.

    • Matt McIrvin

      I think those are the skylines of the cities that Richard Hoagland found in NASA photos of Mars.

    • Dexter Yanagisawa

      It’s like they got a stock investor who specializes in technical analysis to analyze polling data.

    • Bill

      If I remember correctly, the Romney campaign’s internal polling looked more favorable that most other polling in 2012 as well. The scary thing now is that while Romney was shocked at the final result, he graciously bowed out when it became apparent he was not going to win. Trump has been pushing the “it’s going to be rigged” line hard since the second debate, and even floated the idea weeks before that, starting more than a month before election day. Part of me wants to believe that he doesn’t really believe that and is trying to whip up support (side note: I don’t see how essentially telling your supporters that their vote won’t matter because it’s going to be rigged anyway is supposed to inspire them to go out and vote), but when I see things like this it makes me think that he might actually believe it.

      My biggest concern these days isn’t Trump winning, it’s how long he drags out conceding and how much muck raking he does about a rigged election.

    • Matt McIrvin

      My biggest concerns are

      (1) terrorist attacks on polling places at Election Day, perhaps successfully skewing the election by keeping people away;

      (2) Hillary wins but the Republicans retain the Senate, and decide to make the Clinton administration a dead letter by denying 100% of appointments. No Cabinet, no new leadership at the Cabinet agencies, no judges. Keep it up for four years and the resulting chaos makes Clinton look like a catastrophically ineffective leader; win in 2020 with a slightly less berserk authoritarian, maybe Mike Pence or Ben Sasse.

  • microtherion

    One question I haven’t seen discussed much is whether the “likely voter” screens in polls might be underweighting Latinos, which historically had a considerably lower turnout than other ethnicities, but this year might be extra motivated.