Princeton Election Consortium

Innovations in democracy since 2004

Outcome: Biden 306 EV (D+1.2% from toss-up), Senate 50 D (D+1.0%)
Nov 3 polls: Biden 342 EV (D+5.3%), Senate 50-55 D (D+3.9%), House control D+4.6%
Moneyball states: President AZ NE-2 NV, Senate MT ME AK, Legislatures KS TX NC

The exuberance of likelier voters

November 12th, 2008, 10:44pm by Sam Wang

In today’s news, the circus continues: Sarah Palin is still looking for the real killer. Meanwhile, in real news: universal health care? Alaska Senate update: As I predicted from polls, Begich is pulling ahead. Andrew Sullivan is still clinging to the idea that turnout is suspiciously down from 2004, which I have pointed out is probably not true. I’ll stick with my prediction of a Begich win by 2-7%, and normal turnout. In the face of hard polling data, a straightforward interpretation without conspiracies is most likely to be right.

Now, to my real topic today, an oddity of polling. Although average poll margins can predict the eventual winner of a race, they usually underreport the final margin for whoever wins. How can this be?

A few days ago I showed you a plot. Here it is, this time with a fit line.

If polls were accurate, the slope of the green line would be 1 with an intercept of 0. But it is not. Indeed, for every 1% of actual margin, only 0.84+/-0.03% is captured in polls*. This underperformance implies that there is a hidden bonus for whoever is ahead. The intercept essentially goes through the origin, so there’s no overall bias toward either candidate.

This result can be understood better by restating it in terms of pollsters’ methods. Pollsters identify the people in their sample that they consider more likely to vote. Then they re-weight according to likely voting as well as other factors (e.g. age and sex). In light of this, one or both of the following is probably true:

  1. Within a state, the leading candidate’s supporters are relatively more likely to vote than predicted by likely-voter screens.
  2. Within a state, the trailing candidate’s supporters are relatively less likely to vote than predicted by likely-voter screens.

Both scenarios are consistent with the fact that the direction of the effect depends on who’s leading locally.

So likely-voter screens are evidently missing some aspect of voter behavior on Election Day. What’s missing, and why does it show up differently on a state by state level?

One possibility is that many voters are aware of which way their state is likely to go, and modify their behavior slightly. For example, a small fraction might stay home (or at work) because it’s not fun to vote for a known loser. If the probability of voting differered by a net 8% between McCain and Obama “likely voters,” this could then enlarge the size of a win.

Another possibility is that pollsters’ estimates of the likelihood of voting is not sufficiently quantitative. It’s reasonable that they can figure out that one person is more likely to vote than another. But can they do it accurately? For instance, if I am more enthusiastic than you about voting for my candidate, am I 5% more likely to vote than you – or 15%? This can easily lead to bad weighting.

Why has this flaw been allowed to persist? Basically, there’s no penalty. Fixing the error would not lead to improved prediction of winners. A repair would only give a more accurate estimate of the winner’s eventual margin. Yet it is rare to hear anyone ask why a pollster’s projection was too conservative. So there’s no pressure to do better.

Note: Comments and some private correspondence indicate that a few of you have gotten the direction of this discrepancy backwards. To reiterate, polls underreport the eventual margins, which are about 19% larger than the last pre-election surveys would indicate. The effect is in the opposite direction as a hypothetical regression to the mean (itself not reliable, as I point out in comment #20).

*The fit was an unweighted regression of poll vs. actual. The plotted slope is the inverse, 1/0.84=1.19. A weighted fit gives a fit slope of 0.81+/-0.03 poll % per actual % margin.

Tags: 2008 Election

25 Comments so far ↓

  • JJtw

    A few questions about the regression above…

    Model I regression (which I assume you’re doing) depends on the x-axis variable being measured with much less error than the y-axis, and so the standard ordinary least squares method minimizes the square of the y-axis residuals.

    In this case, it is clearly the case that measurement error is greater in the polling data. I know you’re probably wanting to use the predictor/response analogy here, so I see where you’re going. Anyway, it seems as if extra variance in the x-axis would tend to push the regression line more toward a slope of 0, making your estimate conservative if anything.

    Have you considered flipping the axes and taking the reciprocal of the slope? Or perhaps going wild and doing orthogonal regression, or major axis or other model II regression methods that don’t require the y-axis to be the axis with error? I think you can even correct for different variances between the two axes using some methods of model II regression. ( and a pdf here )

  • Paul

    How do we know the slope of 1.13 is statistically significantly different from 1.0?

  • fdeblauwe

    I’ve been experimenting a bit with similar graphic representation of possible correlation. Reading about the Obama team’s relationship with the “net roots,” I decided to plot broadband penetration against Obama vote % and Obama % – McCain %. I did this with state data. Some correlation seems likely and the outliers make sense. See my Word face-Off blog. As I’m rather rusty on this type of statistics, I’d be interested to hear some constructive criticism ;-)

  • Sam Wang

    JJtw and Paul – Updated fit. Paul, it is possible to tell by eye if the points would be fitted by a line with slope significantly different from 1 if they are highly asymmetrically distributed around the black diagonal.

  • Matt McIrvin

    This is interesting for a couple of reasons.

    It argues against Nate Silver’s model adjustment for a theorized tightening effect in the final days of the campaign (I noticed he was still applying a small correction of this sort even right up to the end, as if he expected significant tightening in the last 24 hours, and in fact he underestimated Obama’s electoral win).

    It also suggests that a common worry of progressive political activists is unfounded: that if you act like you’re winning, your side will get complacent and won’t turn out for the vote (or work for you), so you’d better imply perpetual peril instead. Republicans used the opposite strategy for years: they always claimed the vast majority of Americans supported them and their ideals, and that they were natural winners. This might have helped them in close races.

  • blair alef

    Very good work. If one were a pollster interested in more accurate polling they might look at the effect of ‘% of strongly support’ as a possible indication or factor to adjust for.

  • Sam Wang

    The story with “corrections” is this: data of this quality already have a lot of uncertainty. If you are going to add more assumptions, it is essential to be sure that you are adding more signal than noise.

    In Silver’s model, post-data assumptions evidently induced an error of 4.5 to 15.5 EV. They also led to a continuous range of possible outcomes, where as only a small number of options was likely (e.g. 353 EV, 364 EV, 375 EV, and stragglers, not counting NE-2).

  • Frank

    Sam: Some of this (the underestimation of the absolute value of the margin) may be an artifact of the way you calculate the margin, namely by including the undecided respondents in its denominator, which assumes that they split evenly. If instead you divide the difference between Obama and McCain by the total of Obama and McCain (which however you may not have in your data), your slope will move toward unity.

  • Sam Wang

    Frank – That should not make a significant difference. In general, if the fraction of undecideds is U, then the margin itself will be off by a factor of about 1+U. For example, if the split is 50-45-5 (a 5% margin), assigning undecideds gives a 52.63%-47.37%, an increase of 0.3% in the margin. This is far smaller than the discrepancies shown above.

  • Rachel Findley

    What are your thoughts on the Georgia runoff?

  • Sam Wang

    Rachel Findley – That’s a favorable race for Chambliss. It’s unclear what factors work in Martin’s favor here: Chambliss was within a hair of 50% the first time, and Obama’s no longer at the top of the ticket to attract Democrats.

    The one chance Martin has would be for the Libertarian voters to break strongly for the Democrat. That would not be illogical to do but it would require a strong push such as a direct endorsement.

  • Frank

    Sam: Not FAR smaller. Your example explains 28% of the difference between your unadjusted pre-election margin and the final result, and this would be greater if more people were undecided.
    u=proportion polled who did not respond Obama or McCain=5%

  • Sam Wang

    I agree that your equation is correct, though it’s easier to read as u*(s-1)/(1-u), since u<<1. This isn't a sole explanation since it would require 16% of voters to be undecided. You've made a good point, but I think the bulk of the effect is still a puzzle.

  • Frank


  • gary

    I’ve been curious as to this question: If the president were picked by popular vote, how would the final tally differ? I hypothesize the following: 1) Turnout would decrease in the battleground states and increase in all others. 2) With a stronger GOTV effort by Obama, the increased turnout would have benefited him more. 3) With his advantage on the economy, Obama may have picked up more votes by campaigning in the South or other states that he didn’t go to. 4) More pressure on states to have early voting.
    Do you agree with this and is there any way to quantify it?

  • Steve Roth

    Sam, I think (?) this relates to a curious thing I noticed in this NYT Op-Art:

    The lagging candidate always seems to catch up some compared to the final polls.

    This doesn’t sample many polls, but still. Thoughts?

  • Steve Roth

    Let me put that more accurately: the lagging candidate’s vote percent always seems to come in higher than the final polls. (The leader may come in even further ahead.)

  • Jack Rems

    Sam- I got your book yesterday (2nd printing!). I’m looking forward to settling down with it once my election fever passes.

    But I hope you’ll give the Georgia runoff the full meta-analysis treatment. It will be fun to see how the pollsters do; there are a lot of weirdness factors.

    One key will be early voting, but that is only a 3-day week due to Thanksgiving, and I would suggest the GOTV people offer to take voters to the poll and then give them a ride to the mall, grocery store, foodbank—if that’s legal. Funding should go to finding people who need their hand held and taking them to vote early. Election day many polling stations will be scary places, especially if your “papers are not in order” or you think they might not be.

    I read this on fivethirtyeight, from “soozle” in GA, thought it was useful:
    “Having lived for many years in a community that routinely had runoffs for local office, it is our experience that they are won or lost in the 5 days after the general election. That’s the time that the candidates get their postcards in the mail to constituents for requesting absentee/mail ballots for the runoff. In the afterglow of the election, with uncertainty in the outcome, the voter gets his postcard, sends it in, and he received his ballots later. He is far more likely to vote with the ballot in his hand than if it requires a trip to the polling place. we would go so far as to color code the postcards, so a glance at the pile in the city clerk’s office would tell us what parts of the city were responding, and whether they were ours or theirs.
    “Runoffs are an art form unto themselves; most candidates/campaigns make the mistake of waiting to find out the final tally, etc., before they figure out how to position themselves to win the runoff. We found it better to have our runoff postcards ordered and a draft at the printer on election night awaiting only the final details so we would have them in the mail in a day or two. Granted these were local elections, but the scale shouldn’t make a difference.” [November 13, 2008 10:14 AM]

  • William Ockham

    I will posit an alternative explanation for this feature. In any non-close race, everyone with a megaphone in a presidential campaign (leading candidate, trailing candidate, and the press) has a vested interest in making the race look closer than it really is. The leader wants to turn out his voters, and so must ward off complacency. The losing side wants to avoid a rout and hope for lightening to strike. The media needs a contest to get viewers/sell papers/pump their web traffic.

    Pollsters respond to this demand by tweaking their results to shave a little bit (or in the case of a really big lead, a lot) off the leader’s margin. Nobody, except elections nerds like us, really cares if you miss a 20 point win by 5 points.

    If my explanation is correct, then the bigger the real margin, the more it will be understated. That looks right to me from that graph, but I’m not entirely sure I’m reading it correctly.

  • Bill

    I tend to agree with post 18 from William Ockham at 5:30 pm.

    Could it prove informative to gather up a bunch of polling data from lower profile races (house, state legislature, ballot propositions?) and compare it to the outcomes. Fewer polls over all, but lots of potential races to choose from. As a non-statistician, I’m not sure how those factors would effect everything.

  • Sam Wang

    Ockham – Your model doesn’t seem all that parsimonious to me. But I do agree that market forces might encourage pollsters to make races look closer. I don’t think it’s done on purpose, though.

    Steve Roth – If the lagging candidate catches up, (a) that’s at the national level, not state; and (b) it’s in the opposite direction as the effect I am describing.

    Also, see Gallup’s history of trial heats, 1936-2004. It shows the leader pulling away in 1996, 1988, and 1936. So although there is a tendency in the direction you indicate, these exceptions show that it’s not reliable by any means.

  • Frank

    All right, here’s one for you. On p 78 of Remnick’s article on Obama in the latest NYer, an assoc of Axelrod is quoted as saying “When Barack came back from Europe and he was using that line about how he didn’t look like all the other Presidents on American currency, his numbers went down.” Is this reflected in your numbers?

  • bks

    Did you try plotting just the states that McCain won versus just the states that Obama won, leaving out the close contests?
    I would predict that the effect was stronger in the Obama states.


  • Sam Wang

    bks – the post contains a graph that you can inspect to see if this is true.

  • Sam Wang

    Frank – Look at the history plots. I don’t see any such event there. All the horserace-moving events of the campaign that I can identify are labeled.

Leave a Comment