Yesterday, Nate Silver and I both examined Senate polling errors. He noticed no overall bias averaged across all elections; I pointed out that recent bias has been unusually large. Both statements are true. But neither of us pointed out that the biases follow a significant pattern: midterm-year polling is far less accurate than Presidential-year polling, and can be biased in the same direction across-the-board.
From a practical standpoint, this is good news for those of you who don’t like where things have headed lately: in midterms, Senate polling errors are five times larger than in Presidential years. There is bad news too: the error can go in either direction, and a GOP blowout is also possible.
I am interested in why midterm errors are so large. In midterm elections, voter attention is lower than in a Presidential year. In 2012, the Presidential campaigns of Barack Obama and Mitt Romney were in the news every day. This year, it’s…Ebola, which I am pretty sure none of you will ever get. Yet Ebola is in the media much more than any of the aspirant Senators, Representatives, and state officials who will affect our lives. I would guess that more of my neighbors know there is Ebola in Texas than know that Senator Cory Booker is up for re-election.
With lower voter attention comes lower turnout – and evidently, lower certainty about which voters will show up to vote. Other distractions take away from the important issue of Joni Ernst’s desire to eliminate the Environmental Protection Agency…hey, pay attention, I’m talking to you! Ebola Ebola Ebola! There, now you’re back. Thank you.
Here, let me show you how bad the error is.
First, let’s revisit the FiveThirtyEight and PEC analyses to see how big the “midterm error” can be. Here I have sorted out Silver’s list of polling errors according to midterm vs. Presidential election years. I have reversed the notation to show the “bonus” for each party on Election Day: for example, “R by 3.1%” means that Republican candidates outperformed the last 21 days of polling (the time window in his analysis).
However, the median of the absolute values (i.e. irrespective of direction) tells a different story. In Presidential years, the median bonus is only 0.6%. However, in midterm election years it is much larger, 2.9%, nearly five times as large.
Pollsters are good at identifying likely voters in Presidential years, when the airwaves are saturated with talk of the candidates at the top of the two party tickets. But with less information about the election easily available, voter attention can be captured by other topics like Ebola, which crowd out political coverage. What effect does this have on voting? I’m not sure, though I will say that reduced turnout generally favors Republicans.
Some of you have speculated about whether get-out-the-vote (GOTV) activity could make a difference. For example, the Democratic bonus in 2012 sticks out as being larger than other Presidential years. It could potentially have been caused by improved turnout efforts. However, that is only one data point. More generally, GOTV will only provide a bonus if it identifies people who were not identified in surveys as likely voters. And we don’t know how well pollsters have accounted for it.
As the historical record shows, the sum total of pollster error can be in either direction. For example, 2010 was a great year for Republicans, and Democrats outperformed the low expectations set by polls. But in 1994 and 2002, also Republican wave years, Republicans were the ones who overperformed. It is not time for either side to get excited or despondent – only to work harder.
In short, all I would be willing to say based on this historical pattern is that any race currently within 4 percentage points or less is still uncertain.
Polling errors persist, even in the home stretch
As I mentioned yesterday, even in the week before the election, the hidden bonus can be quite substantial. Here, again, is my analysis of races that were ultimately decided by 10 percentage points or less. This calculation is based on the last week of pre-election polling.
Even at the last minute, the bonus can be rather large. Candidates could still win if they trailed by a margin of less than 3 percentage points in the week before the election. Here are the details for 2010 and 2012:
Here, negative numbers (in red) indicate a GOP final win or polling lead, positive numbers (in blue) indicate a Democratic final win or polling lead (click to see data going back to 2004). In 2010, the error was enough to flip the result to the opposing candidate in Colorado and Nevada. In 2012, no such errors occurred.
With all this in mind, here is the bottom line: for a range of possible bonuses between 3% for Republicans and 3% for Democrats, based on current polling the range of outcomes would be from 49 to 54 Republicans. That range is so wide because of the unusually large number of close races this year – and of course, it straddles the 50-vote threshold for a change in control. Midterm polling, lots of close races, Senate on the edge – all in all, it’s a challenging problem!
The PEC probability calculation above assumes a slightly lower range of errors than is indicated by midterm data alone. Any model that combines midterm and Presidential historical data might be underestimating the uncertainty. Therefore, as I wrote yesterday, I suggest that a better number to follow is the Senate Meta-Margin, which is currently R+1.4%. As long as the margin remains below R+4.0%, a polling error favoring Democrats can still reasonably alter the picture. And, of course, a significant polling error favoring Republicans would lead to a major blowout. For now, nobody on either side gets to take their eye off the ball.