Princeton Election Consortium

A first draft of electoral history. Since 2004

Midterm National Senate Polling Error Is Five Times Larger Than In Presidential Years

October 17th, 2014, 12:49pm by Sam Wang



Yesterday, Nate Silver and I both examined Senate polling errors. He noticed no overall bias averaged across all elections; I pointed out that recent bias has been unusually large. Both statements are true. But neither of us pointed out that the biases follow a significant pattern: midterm-year polling is far less accurate than Presidential-year polling, and can be biased in the same direction across-the-board.

From a practical standpoint, this is good news for those of you who don’t like where things have headed lately: in midterms, Senate polling errors are five times larger than in Presidential years. There is bad news too: the error can go in either direction, and a GOP blowout is also possible.

I am interested in why midterm errors are so large. In midterm elections, voter attention is lower than in a Presidential year. In 2012, the Presidential campaigns of Barack Obama and Mitt Romney were in the news every day. This year, it’s…Ebola, which I am pretty sure none of you will ever get. Yet Ebola is in the media much more than any of the aspirant Senators, Representatives, and state officials who will affect our lives. I would guess that more of my neighbors know there is Ebola in Texas than know that Senator Cory Booker is up for re-election.

With lower voter attention comes lower turnout – and evidently, lower certainty about which voters will show up to vote. Other distractions take away from the important issue of Joni Ernst’s desire to eliminate the Environmental Protection Agency…hey, pay attention, I’m talking to you! Ebola Ebola Ebola! There, now you’re back. Thank you.

Here, let me show you how bad the error is.

First, let’s revisit the FiveThirtyEight and PEC analyses to see how big the “midterm error” can be. Here I have sorted out Silver’s list of polling errors according to midterm vs. Presidential election years. I have reversed the notation to show the “bonus” for each party on Election Day: for example, “R by 3.1%” means that Republican candidates outperformed the last 21 days of polling (the time window in his analysis).

The directional median indicates a bonus that favors one party over the other. Over the last six election cycles, there has been no net bonus.

However, the median of the absolute values (i.e. irrespective of direction) tells a different story. In Presidential years, the median bonus is only 0.6%. However, in midterm election years it is much larger, 2.9%, nearly five times as large.

Pollsters are good at identifying likely voters in Presidential years, when the airwaves are saturated with talk of the candidates at the top of the two party tickets. But with less information about the election easily available, voter attention can be captured by other topics like Ebola, which crowd out political coverage. What effect does this have on voting? I’m not sure, though I will say that reduced turnout generally favors Republicans.

Some of you have speculated about whether get-out-the-vote (GOTV) activity could make a difference. For example, the Democratic bonus in 2012 sticks out as being larger than other Presidential years. It could potentially have been caused by improved turnout efforts. However, that is only one data point. More generally, GOTV will only provide a bonus if it identifies people who were not identified in surveys as likely voters. And we don’t know how well pollsters have accounted for it.

As the historical record shows, the sum total of pollster error can be in either direction. For example, 2010 was a great year for Republicans, and Democrats outperformed the low expectations set by polls. But in 1994 and 2002, also Republican wave years, Republicans were the ones who overperformed. It is not time for either side to get excited or despondent – only to work harder.

In short, all I would be willing to say based on this historical pattern is that any race currently within 4 percentage points or less is still uncertain.

Polling errors persist, even in the home stretch

As I mentioned yesterday, even in the week before the election, the hidden bonus can be quite substantial. Here, again, is my analysis of races that were ultimately decided by 10 percentage points or less. This calculation is based on the last week of pre-election polling.

Even at the last minute, the bonus can be rather large. Candidates could still win if they trailed by a margin of less than 3 percentage points in the week before the election. Here are the details for 2010 and 2012:

Here, negative numbers (in red) indicate a GOP final win or polling lead, positive numbers (in blue) indicate a Democratic final win or polling lead (click to see data going back to 2004). In 2010, the error was enough to flip the result to the opposing candidate in Colorado and Nevada. In 2012, no such errors occurred.

With all this in mind, here is the bottom line: for a range of possible bonuses between 3% for Republicans and 3% for Democrats, based on current polling the range of outcomes would be from 49 to 54 Republicans. That range is so wide because of the unusually large number of close races this year – and of course, it straddles the 50-vote threshold for a change in control. Midterm polling, lots of close races, Senate on the edge – all in all, it’s a challenging problem!

The PEC probability calculation above assumes a slightly lower range of errors than is indicated by midterm data alone. Any model that combines midterm and Presidential historical data might be underestimating the uncertainty. Therefore, as I wrote yesterday, I suggest that a better number to follow is the Senate Meta-Margin, which is currently R+1.4%. As long as the margin remains below R+4.0%, a polling error favoring Democrats can still reasonably alter the picture. And, of course, a significant polling error favoring Republicans would lead to a major blowout. For now, nobody on either side gets to take their eye off the ball.

Tags: 2012 Election · 2014 Election · Senate

37 Comments so far ↓

  • SJWangsnesss

    I’m just keeping the TV off Tuesday, so I can look at the paper the next day and either see what I’m resigned to seeing, or receive unexpectedly joyous news.

  • whirlaway

    I think Obama made a big mistake – and that was to wax eloquent on how well the economy was doing. The reality is that is not going well for the vast majority of the people.

    The Republicans seized on it to say that the President was out of touch. That seemed like a correct diagnosis to most of the voters.

    So, rather than saying that the economy was doing badly because of the Republicans, Obama said the opposite – that it was (a) doing well and (b) in spite of the Republicans. The people felt that since he was wrong about (a), he was probably wrong about (b) too. And of course, the Republicans made sure that narrative stuck, many Democratic candidates didn’t want Obama to even campaign for them, and it was all downhill for them from there.

    My best guess for the Senate is D-48 and R-52.

  • Jeff Jones

    At least one exception – here in NC you cannot turn on the TV or head to any internet site without being bombarded by Tillis and Hagan ads. They are a pox.

  • Philip Diehl

    While the evidence in these charts is thin, it supports an impression I’ve had from telephone canvassing over the years.

    With voters abandoning landlines and screening calls from numbers they don’t recognize–rising since 2009, I suspect, among the 1 in 3 Americans in debt collection‎, it’s becoming more difficult every year to reach voters by phone. And those who do answer seem to be increasingly unrepresentative of a random sample in a way that biases results in favor of Rs.

    I know this is discussed in the business, but IMO it’s not been given its due in assessments of polling error.

    I have another theory (ok, speculation) regarding polling bias. In recent years, polls are increasingly being conducted and issued for strategic reasons, not simply as an attempt to fix the current state of a campaign. This is one reason why we see so many more partisan polls these days.

    A poll designed for strategic purposes can (and do) incorporate assumptions and modeling choices that tilt results to fit a strategic purpose.

    Another factor with partisan polls is that campaigns are not obligated to release results unfavorable to their sponsors.

    One might expect bias of this kind to wash out of national poll averages, but that wouldn’t be the case if one Party were consistently less constrained in such practices or had more money to conduct more polls of this type. If so, they could bias poll averages, including those based on medians, in a way that has nothing to do with RV/LV models, sampling error, and other “legitimate” methodological sources of error.

  • wendy fleet

    Sam, I wish you hadn’t closed the Comments on your UP appearance — I mean we oughta get to gush about what a good job you did and how much fun & informative it was to, like, see you for some minutes. I hope everyone goes and sees the video linked in that post.

    • Bill

      I third, shma and Olav. It only makes good sense to amend the numbers if you have a better understanding of the error. To NOT do so would be sacrificing your developing statistical understanding in the service of avoiding criticism and grumpy posts. Go for it!

  • shma

    I appreciate you trying to incorporate aggregate pollster error into your meta-margin error, but now the banner probabilities make no sense at all. A 1 sigma confidence interval of R +3.1 to D+1.9 does not correspond to a probability range of 30%±15%. The probability range should be something closer to 15% to 75%.

    • Sam Wang

      Well, tell me…now that I have a clearer idea of the last-minute error – previously assumed to be 1.3%, now 2.5% based on midterm-specific error, as per the Friday post — if I alter the probability calculation, who will stick up for me here?

      The actual effect is to bring the 30% toward 50%, which would make it around 40%, i.e. R+0.6% was 0.46 sigma from zero, but now it it will be 0.24 sigma.

    • shma

      I think most posters on this site will stick up for you as long as you explain the change clearly.

      New analysis leads to a better error estimate, which leads to an updated probability range. Seems pretty straightforward.

    • Olav Grinde

      I second that, shma.

  • Canadian fan

    Utterly delighted with the two new polls showing Udall up 3 in Colorado. The Huffpost average now shows Udall trailing by only 1.4 %. What makes Colorado so historically difficult to poll is that the significant Latino population is hard for pollsters to fully gauge or even access. One recent poll had Gardner carrying the Latino vote and another had Udall carrying the white vote ! As a result, Democratic performance easily outperforms in Colorado. But possibly the most decisive factor – rarely discussed – is that Colorado – for the first time in its history – is conducting a 100 % mail-in only election.

  • atothec

    Two new polls from Mellman and Berenson Group released both with Udall +3 in CO.

    No doubt the GOP will now say Dems are weighting the polls…

    • Poll interrupted

      There are few, if any, truly non partisan polls this cycle, and, even when they are, they seem modeled after 2010, which I remain unconvinced with states like. SD KY GA and KS in play. are in accurate read of the electorate. The conditions are unfavorable to both parties. There will be a bias this cycle too. Likely voter models are assuming a GOP wave. The more likely model is voters don’t much like either party, but since there are only two choices on average on the ballot, that makes for a volatile out come.

    • JayBoy2k

      I am as comfortable with these 2 new polls as you are with Quinnipiac (Gardner +6) and CNN (Gardner +4) this week. It seems like they can not all be correct, unless there was some huge black swan event Thursday night. It may signal a significant change but we will have to see a few more polls to confirm.

      I am getting more comfortable with the PEC process of median only, no fundamentals or house bias required. It seems to match other poll aggregators without all the complexities.

  • Art Brown

    Katrina was a big political story; given the stumbles to date, Ebola could be too. The White House focus is fully justified.

    There was a brilliant piece of Republican advocacy in NYT’s John Harwood’s Political Memo: “Republican Pollster Takes Lessons Learned in 2012 to Senate Races”.

    “We got a wake-up call after 2012″.

    “A chastened cautiousness has made Republicans more trusting in favorable trends seen during the homestretch. A renewed confidence extends from campaign operatives to the donors, who have lately increased financing for their efforts.”

    So because they’ve learned their lessons and are more cautious, they believe the recent poll swing in their direction. I love that logic (although it’s hard to dispute their confidence at this point).

    Question: Do disgraced Republican pollsters go to France for a month for recuperation or atonement?

  • Tlaloc

    FYI using mean we get a tiny dem +.06 in the midterm years but a dem+.7 in presidential years for directional.

    For the absolute values the midterm mean is 2.8 and for presidential years it’s 1.2

  • Tlaloc

    I’m unclear why you’d choose to use a median instead of a mean average for the first chart. Since we have equal numbers of +rep and +dem results all the median is telling us is that the spacing between the smallest of each.

    But there’s a huge difference between
    rep+1, rep +.1, dem +.1, dem + 10
    (median 0, mean dem+2.5)
    and
    rep +1, rep +.1, dem + .1, dem + 1
    (median and mean 0)

    • Chris

      I think that first example is the reason Sam uses the median. In the case where 3 polls are within ~1 point of each other, and one is 9-10 points off, it usually means that last is a junk poll, or way off, and so unreliable.

    • Sam Wang

      Exactly. Using the average would give the wrong impression. What I wanted to show is how likely an error was in either direction. The answer: overall error can go in either direction, and could be large!

    • Tlaloc

      But we aren’t talking about a single junk poll, we’re talking about how the election day result compares with the polling *average* before the election. In that situation you can’t say the polling average was an outlier, Rather you had the case that the overall polling was significantly skewed from the final result, i.e. exactly what we are looking for.

  • JayBoy2k

    I think that Ebola is a problem for Ds. This is not based on reality of the situation or risk to Americans.
    The CDC (and by extension the Administration) are being portrayed as close to incompetent and unable to generate trust. Looking this morning at the NY Times, Braley ripping into the CDC chief at House hearings, Kay Hagan calling for a travel ban, and left leading pundits like Eugene Robinson questioning the CDC and the administration http://www.realclearpolitics.com/articles/2014/10/17/on_ebola_candor_needed_124330.html

    This can not be positive for Ds in close races, this late in the cycle. It may not have a significant impact, but Braley and Hagan must believe it could.

    • atothec

      Do you really the think the average American associates the CDC with the administration? Do you think the average American even knows what CDC stands for? I don’t.

      The only way Ebola would be negative for Dems is if the GOP is allowed to continue pushing their Obolisisghazi narrative unchecked.

    • Philip Diehl

      I agree. This is not good for the Ds.

      The CDC/Ebola story fits into a larger narrative about the alleged incompetence of the Obama administration that has been gaining traction for months. Of course, it’s bogus, but the administration’s inability to anticipate and get ahead of issues and its long record of poor strategic communications has fueled this narrative.

      This is rattling some vulnerable D candidates who are no doubt reading reactions of voters.

      BTW, why aren’t Rs talking about putting restrictions on travelers entering the country from Texas? I live in Austin so I’m not advocating it. Just curious. :-)

  • Steve Scarborough

    Hi Sam. Excellent article! In my opinion, it is likely that 1 or more Dem Senate candidates will win when polls show otherwise. What I cannot discern at this time is the question of how many. If I had to go out on a limb and bet, it would be one, and a candidate would be Colorado.

    On the Rep side, my guess is that zero would beat the polls. But, there could be one. Just which one, I am not sure. Maybe in NC?

    Again, job well done on the article!

    Steve

    • Dean

      I agree with the kudos for the article. You have to wonder if the error is because of the number of Spanish speakers at homes and maybe the voter is uncomfortable in answering questions. That could be the reason for the Dem plus at the polls.

  • Canadian fan

    Wonderful article, Sam. In a just released Washington Post poll Democrats said they were either very confident or somewhat confident in the government’s response to Ebola by 76 %. Republicans ? 54 % ! So the Republican mantra of fear appears not to be working. Sam is right – Ernst wants to do away with the EPA. And if a recently released tape of Ernst from last year is any indication, she wants to do away with pretty much everything since the New Deal. So it’s clear Republicans are strenuously trying to deflect focus on things like that – the very issues Sam correctly points out is what will drive a new Senate.

    • J

      It isn’t helped that the likes of Joni Ernst, Thom Tillis, Tom Cotton, etc are paraded in the media as “establishment” candidates when they’re even more right-wing than some the Tea Party candidates that were running in 2010. The only difference between a Tea Party Republican and an Establishment Republican is that one is more prone to telling the truth about what they stand for than the other. Or as the media calls them “gaffes”. (At least Christine O’Donnell and her “I’m not a witch” ad provided a bit of comedy, though it helped knowing she was guaranteed to lose.)

      At least that Washington Post poll confirms that for all the media and Republican fearmongering, the people are more concerned about the positions the candidates are running on instead of succumbing to fear. Electing candidates who advocate depriving the working poor of government assistance to teach them a lesson, support personhood bills, or wanting to bomb countries like Iran is not a solution to containing the Ebola outbreak by any stretch of the imagination.

    • Bert

      As is typical of the GOP, if they’d nominated a somewhat moderate Republican in Iowa they’d be running away with it. Instead they nominated a far right candidate like Ernst. I’m reminded of the last Governor’s race in Virginia in which an unpopular Democrat narrowly won because the Republican candidate was too far right for the state as a whole.

    • Olav Grinde

      What bothers me even more, J, is that the likes of Joni Ernst, Thom Tillis, Tom Cotton and Ted Cruz are referred to as conservatives.

      They are nothing of the sort! These politicians are radicals – the are destructive extremists who would gleefully tear down what has been built through generations of political consensus and compromise.

      I find it disheartening to observe how the very meaning of “conservative” has been hijacked in this millennium.

      Where is the press?

  • Davey

    Hooray! My favorite article all season. Not because of my political leanings, I just like when math tells us we don’t know whether math telling us what we don’t know is accurate. It has a certain Zen quality.

  • Alex

    The coverage of Ebola again supports the point of media sensationalization.

    The flu and especially non-communicable diseases (i.e. cardiovascular disease, stroke, cancer) killed far more people in the same time-frame.

    The fact that our health infrastructure is capable of effectively quarantining a virus that doesn’t spread easily makes it a non-story. That it’s being turned partisan and into a big issue makes me cynical towards society.

    • Joseph

      Turning the Ebola crisis into a political football is potentially a huge mistake by the Republicans. Sure, it might help energize the “crazies”, but sane people know it’s a real stretch to lay this at the President’s doorstep, and the “crazies” can hardly get any more wound up than they already are. Still, if we’re going to play the blame game, the who’s responsible for the failure to approve the President’s choice for Surgeon-General? Surely if we’re in the midsts of a major medical crisis that would be at the top of any truly non-partisan political agenda…..

    • Phoenix Woman

      I agree with Joseph that the GOP’s politicizing of Ebola is a big mistake. Once there are no more new cases in the US — and the fact that there haven’t been any new ones in nearly a week, and that all three of them are connected with the same Texas hospital — the scaremongering will be less potent.

  • Albanius

    How are “likely/unlikely” voters weighted by pollsters?
    Are they counted fully if deemed likely, omitted if deemed unlikely, or weighted by estimated likelihood of voting, eg the fraction of last few elections of the same type (presidential/off-year) in which the pollee voted?
    Do all pollster use the same method, on/off or weighted?
    IMHO, the latter method would be more accurate, since intermittent voters do contribute to the total vote.

  • Jason Marcel (@moviejay)

    Ebola is not a driving factor to get anyone to the polls.

    Republican voters show up and vote at EVERY opportunity for a government filled with people who thumb their noses at government.

    Democratic voters only show up when the bathroom is so hideously disgusting that they can’t possibly go on using it without cleaning it up first.

    Republicans disrespect democracy by purposefully wanting to block government from working while Democrats disrespect it with their apathy when it comes to exercising their civic duty.

    • Cal State Disneyland

      I am wondering if we are going to get to the point where there is a consistent majority popular vote for the Dems but, in seats, a substantial majority for the Republicans – due to both the natural distribution of Democrats into tight areas and cutthroat gerrymandering.

      At least if the majority popular vote went Republican, the sting would be less.

  • Joseph

    The key questions in my mind are: the state of mind of undecideds who intend to vote; and the impact of third party candidates/independents on the main candidates. I don’t think that has completely gelled yet. I am concerned about a loss of heart on the part of progressive and liberal voters, as well as a chilling effect on the liberal vote by what they see as a pro-war stance on the part of the President and the impact of that on the President’s job approval.

Leave a Comment