The Midterm Polling Curse (Morning-after edition)
Pre-election PEC Senate aggregate: 52 Republican seats. Outcome: 52 or more Republican seats (Alaska is not called, and Louisiana goes to a runof...
Exceptionally Low Turnout Can Account For Polling Errors
At The American Prospect: Tweet
In the home stretch, I wrote that midterm polling is far less accurate than in Presidential years. Today, in The American Prospect, I detail how this year’s polling errors are correlated with voter turnout, which was the lowest since 1942, as based on Michael McDonald’s tabulation so far. In 2014, Democrats underperformed expectations by over 5 percentage points on average, the largest such error in over 20 years. State by state, underperformance was correlated with low turnout. This suggests that voter apathy on the Democratic side was a significant factor in the 2014 election. Here’s the article.
From a polling standpoint, estimating turnout is likely to be a major source of systematic error. Here are some details of this year’s GOP “bonus.”
In the Senate, the GOP overperformed by 5 percentage points
Republican performance above and beyond pre-election Senate polls can be estimated in two ways.
The first is pretty simple: just calculate the difference between outcomes and polls for states that were in contention. This gives a “bonus” for Republicans of 5.2 ± 1.1% (mean ± SEM; SD=4.2%). Comparing with past Senate polling errors, this is the largest error in the FiveThirtyEight database going back to 1990. The next-largest error was a 4.9% bonus for Democrats in 1998.
The second way is to use the Meta-Margin, the amount of swing needed to create a tie for control. This can be calculated by arranging the races in order of winning margin in order, and calculating what the “tipping-point” margin is between the 50th and 51st GOP win. That’s a little ambiguous because the Louisiana runoff has not occurred yet. But if we assume that the tipping-point margin is halfway between Georgia and Louisiana, it would be R+5.8%. Compared with the pre-election Meta-Margin of R+1.0%, that’s a Republican bonus of 4.8%.
I note that one state where turnout cannot account for the polling error is Iowa. There, turnout was quite robust – and Joni Ernst (R) outperformed by about six percentage points. This is worth examining. Nate Cohn at the NYT suggests it’s a sign that Iowa is newly competitive. That is certainly possible. One possible reason is early voting in that state, an area where Republicans did much better than in 2010.
Similar overperformance in the House
In pre-election polls, the generic Congressional ballot showed a Republican lead of 1.5 percentage points. The House vote isn’t done being counted yet, but David Wasserman and his team at the Cook Political Report currently have Republicans leading the national popular vote by 6.8%. (They also confirm that turnout was exceptionally low, down by about 42% from the 2012 election.) The difference between polls and the actual House vote is a bonus for Republicans of 5.3%, similar to the Senate estimates.
Republicans gained at least 14 seats, with 6 more races outstanding (in these six, 3 Democrats lead and 3 Republicans lead). A gain of 17 seats is larger than expected (0 to 12 seats gained). However, it is smaller than 26 seats, which is the median post-World War II midterm gain by the President’s opposition party.
Finally, as I wrote last Wednesday, gubernatorial races showed a smaller bonus for Republicans, about 2 percentage points. I don’t know yet why this is smaller. One possibility is that voters are more likely to cross party lines when voting for governor. Many states have a history of electing governors from the non-dominant party. Another possibility is that when pollsters are focused on a particular state’s race, they are better at gauging likely voters than when they have to build a model that applies to many states at once.
It occurred to me to ask whether the direction of the correlation between turnout and polling bias was always the same. The answer to that is no. In 2006 and 2010, Democrats did better than expected in low-turnout states. The absolute magnitude of pollster error was similar – but the direction was different. In those years, it was Republicans who tended to show up less often than expected. I’ll post a graph of that later.
Currently, I think the general principle is that pollsters have a tough time identifying people who are marginally committed to voting. In 2010, Republicans seemed angry with the President and Congressional Democrats – but evidently not as mad as pollsters thought.
One interesting note: as David Wasserman points out, Republicans won the popular vote in 2010 by 6.6%, nearly the same margin as in 2014. Yet Republicans led the generic Congressional ballot in 2010 by over 10 percentage points. This year is a particularly vivid example of the difference that voter turnout can make.
>>>
P.S. Later I will do postmortem examination of long-term forecasting. It’s not far from my mind. More later.
Topics:
Great effort. Thanks very much.
Another problem with estimating voter turnout is the credence given to the answer to the pollster’s question “will you vote?” What demographic is more likely to say “yes” and for one reason or another fails to show up to vote? One might guess, for example, that a retiree may be more likely to follow through than, say, a busy college student, particularly if the voter is lukewarm on the choices. But, as Sam notes, it’s not always the Republicans who are more reliable voters than the Democrats, but apparently they were this time.
I just read an article on HuffPost Pollster which said the exact opposite, that low turnout doesn’t explain the election. For example almost 300K more people voted in Colorado this year than in 2010. Also in Maryland turnout can’t be the culprit either according to an article in the Washington Post. And as you point out more people voted in Iowa’s Senate race this year than they did in 2010. No turnout doesn’t explain Dem’s loss, their terrible policies do.
I think it does not need to be either/or. The question I am addressing is that of massive underperformance. Examine the data; it points pretty clearly toward turnout as a large factor. If there are anomalies that need to be explained, the prime example is Iowa.
You are addressing another point: why are polls where they are in the first place? I do not have an answer to that question. I think your answer is speculative, which is fine.
I’m with Pollster, and a bit surprised at the unusual shallowness in Sam’s analysis here. National-level turnout tell us little, compared to the specifics of high-profile-race turnout, which in nearly every case exceeded ’10, just by varying degrees. With the exception of a state like VA (and for obvious reasons there) the R ‘bonus’ charted above came much more often from R ‘overproduction’ than some failure in D production (sure they failed to achieve ’12 levels in most cases but nobody expected that–what was obviously unexpected in public polling was the R wave). The basic premise of this post’s title is incorrect.
No turnout doesn’t explain Dem’s loss, their terrible policies do.
In that case, why do they sometimes win elections? An explanation that implies that Democrats should lose every election forever won’t wash, because they don’t.
I think we’d have to delve into the data a little deeper, other Matthew, before we get too critical. 38 states had lower 2014 turnout than 2010, and the states with higher turnout weren’t spectacularly higher. Looking at Colorado, turnout bumped up a meager 0.7% over 2010. North Carolina and Florida inched up 0.9%. That’s kind of sad…considering the $4 billion spent to generate votes.
My hypothesis – and we would need quite a bit of data to test it – these meager bumps are the sum total of a small rise in GOP interest, with a slightly smaller decline in Democratic interest, which results in no significant increase of overall turnout but points to how powerful two small shifts in opposite directions can be in a closely divided electorate.
This theory would also substantiate something Professor Wang pointed to throughout – the immense power of a single vote in these states. A relatively tiny number of votes from one group and a simultaneously tiny number of non-voters from another shifted the races, and tipped the Senate.
Is there any sign what role different factors played in turnout? Has anyone done a regression to see if crosscheck or curtailing number of early voting days explains much of the turnout gap or if it is mostly ‘lack of enthusiasm’. I’d imagine all of these having some impact, but I’ve not seen an analysis of the relative likely effect.
Was there a disproportionate drop in city or lower income voters in states reducing voting days for example?
Thanks for the analysis, Sam You are helping me scratch an itch. A couple of questions:
You wrote: (They also confirm that turnout was exceptionally low, down by about 42% from the 2012 election.) . How do I parse a comparison of turnout between 2014 and 2012? Is not the more insightful comparison 2014 to 2010?
Since we have differences in the amplitudes of turnout state by state, should not the state level analysis inform our judgements? I am tremendously interested in those states where turnout (GOTV) was supposed to be a difference maker (CO, AK, Iowa, GA) and in the states that were close.
I am not so interested in non-close states where GOTV was a minor factor, since there may be different factors at play with nothing on the line.
I would like to discuss whether the D GOTV was lacking or the R GOTV was great or some combination of the 2.
Finally, a link to turnout data you are using to base the discussion. Your link is likely much better than one I find.
Does the voter turnout data include information about how many people flipped and voted for the opposite party this year and/or voters that voted opposite to their registered party? (the two sets of voters are not necessarily the same)
I suspect that in previous years, same-sex marriage was a wedge issue that caused some number of otherwise conservative republican voters to vote for democrats. With the recent gains and momentum in marriage equality across the US, I don’t believe that voters see it as a wedge issue and are now free to vote more in line with their chosen party.
Similarly, I believe that the recent uptick in gun regulation has made it a wedge issue that causes otherwise liberal democrats to vote for republican candidates based solely on their stance on gun rights. I know quite a few liberal people in their 30s who own guns and believe that the democrats have gone too far with nanny-state “protect the children” rhetoric when it comes to guns.
I have been searching and found this article:
http://www.pbs.org/newshour/updates/2014-midterm-election-turnout-lowest-in-70-years/
It indicates that turnout was up over 2010 in every state that the Rs won and switched a D senate seat. Lower turnout in WV, Mont., SD where it did not matter. From the article: Higher turnout 2014 over 2010.
1. Louisiana: +12.9% (38.9%-43.9%)
2. Nebraska: +10.1% (37.5%-41.3%)
3. Arkansas: +9.9% (37.5%-41.2%)
4. Wisconsin: +9.4% (52.0%-56.9%)
5. Maine: +7.4% (55.2%-59.3%)
6. New Hampshire: +6.8% (45.7%-48.8%)
7. Alaska: +6.6% (51.9%-55.3%)
8. Washington, D.C.: +4.8% (28.9%-30.3%)
9. Colorado: +4.7% (50.6%-53.0%)
10. Kentucky: +4.2% (42.4%-44.2%)
11. North Carolina: +3.8% (39.2%-40.7%)
12. Florida: +3.4% (41.7%-43.1%)
13. Kansas: +2.6% (41.7%-42.8%)
14. Iowa: +1.4% (49.9%-50.6%)
15. Oregon: +0.2% (52.6%-52.7%)
Since any individual state could have a high-profile race one year, then a low-profile race in another year, to gauge overall turnout it might be better to use national turnout as a comparison, for which the “VEP Highest Office” (see this for a discussion) is a good index. In 2010, the VEP-Highest-Office turnout was 40.9% nationally, and for 2014 it is 36.3% so far.
In regard to specific-state turnout, you raise an interesting point. Let me restate what I am trying to say: pollsters survey who they can reach. Then they apply stratification, and furthermore apply methods to correct for likely turnout. If deviations from their estimated result are strongly correlated with turnout, that implies that candidate preference and propensity to vote are correlated in some way. To me this seems inarguable.
This year, low-propensity voters appeared to tilt Democratic. That’s not the case every year – as I alluded, it was different in 2010 and 2006. In those years, turnout and polling bias are also correlated…but in a different direction. Then, Democratic candidates did relatively better than expected in low-turnout states.
Finally, in regard to Iowa, Colorado, and so on…I’m not saying anything about whether a candidate performed strongly or weakly relative to expectations, which is what I think is what Matthew is getting at. Udall or Braley may have been weak candidates, but presumably that would show up in polling in the first place. What I’m really getting at here is that variations in turnout can reveal something about low-propensity voters. I am not entirely sure what is shallow about that!
Polls do not have voter ID requirements, fewer absentee voter days and the rest of the new voter restrictions.
I have not seen a check of provisional votes (usually not counted) compared to other years or turned away voters or voters struck from lists before the vote without their knowing (which I saw at my precinct).
Voter suppression of 1-3% may have made a difference in many of the states with a “poll bias” comparing to 2010 and other non-prez years.
So maybe the polls were not that far off and the turnout was lower for Democrats because of voter restrictions.
I do not buy the Nate Silver “poll bias so that’s the way it is” argument without a numerical check on voter restrictions and voting problems like Florida’s failed voter registration “E-books”.
In 2008 I made about 3000 phone calls on behalf of Hillary between Jan & June. I was stunned, appalled, & frustrated to discover how many older women in Caucus States deeply wanted to vote for Hillary but do NOT go out in the winter or at night — “Oh, honey, I don’t dare go out — I might slip and break my hip.” “Getting from my car across the parking lot to the school gym is just too dangerous — I might fall.” I was wild about this info which I wildly tried to share with reporters. I kept saying to my callees, “But couldn’t you vote Absentee?” until I found out that there is zero absentee voting in caucuses. Everyone had theories about why Hillary lost but I knew something that no one else knew — I knew about these hidden votes that Could Not Be Cast.
Then phoning this year in NC CO IA & NH, I discovered that people 70+ just were clueless about Absentee Voting — it simply isn’t in their Voting Experience/Vocabulary. (Some states also have draconian requirements for Vote-By-Mail.) EVEN when people like in CO are supplied with Absent Ballots, very old people feel that “voting” *is* going to the polls.
I was so desperate & enraged about the idiot Caucus thing that I started a blog called Caucus Debacle. That Iowa still has a caucus is vilely anti-democratic. The state makes so much money with the onslaught of media renting motel rooms & tippling at midnights that I don’t see actual democracy mattering any time soon. Old women and single working women are basically disproportionately cut out of caucuses for stark practical reasons. This hugely skews who gets selected and people’s hypotheses about *why* are garbage. May be, Sam, you could at least get some enterprising pollster to poll those who Cannot Vote in Caucus states rather than them being labeled as Apathetic who aren’t? Sigh.
The irony behind the strong public acceptance in recent years of marriage equality, is that in 2004 Karl Rove used the opposition to it as a wedge issue in certain state ballot initiatives in order to drive up Republican turnout. Nothing much has changed since, outside of the fact that the American people ( and Supreme Court ) have forged ahead with this issue, while Republicans have been left far behind. The official policy platform of the GOP ( notwithstanding some elected officials ) remains as it always has been – against equality – and consequently outside of American majority opinion. In terms of gun-control, national polls have consistently shown that a majority of Americans support gun-control, including a majority of Republicans. This is another issue where Americans have left Republicans far behind. The GOP has allowed itself to be directed from their far-right Tea Party ranks, meaning that the current strategy will be even more obstructionist than it was. Distressingly, the party has now taken its mandate and decided that its number one focus will be the dismantling of health care for millions of Americans. What has changed since the election is that it will now be impossible for the GOP to be able to disguise their true intentions. They are in hiding no longer.
It is clear in the aftermath that the pollsters got the turnout wrong, both in magnitude and in composition.
The sad truth is that the “turnout model” or “likely voter filter” or whatever other science-y name they give it, is just a gut feeling. There is no actual science there. Most of the time they get away with it because any given year looks like previous years. Nov 2014 didn’t.
Until there is a robust way to estimate turnout among various subgroups, polls will contain a large component of this entrail-reading. Sure, some will guess well and be lauded, some will be off and be the goats of that election cycle, but all will be throwing the dice.
“It is clear in the aftermath that the pollsters got the turnout wrong, both in magnitude and in composition. ”
Since the election, I having been rolling in my mind polls and results from Iowa, Alaska, Kansas, Colorado, Georgia, and even Florida. I read an article about Florida indicating that total turnout, R turnout, and even D turnout were all up. All the National polls were using the same/similar voter screens (I guess polling minds tend to converge) but if you merely moved the R turnout advantage from +2% to +4%, the polls average perfectly predicted the outcomes.
I believe that you are spot on. The pollsters got the voter screens wrong. There were hints like R enthusiasm and Ds avoiding the President’s policies that were ignored.
How to explain J. Ann Selzer’s poll — here is one of her statements:
Selzer: >>>>I have not seen others’ poll in detail, but we had Ernst lead[ing] in Braley’s home district and we had her winning independents. Not sure other polls showed that. We looked specifically at the claim that Braley’s organization was identifying erratic mid-term voters. Ernst won with respondents who say they voted in 2010 and those who did not. I have no idea how they are modeling their data. <<<< It will be interesting to see what changes , if any, pollsters make for 2016.
Life has prevented me from following too closely. Was the turnout in Colorado where every registered voter was mailed a ballot low as well.
I copied some data from key 2014 Senate races into a post above. The line for Colorado was:
9. Colorado: +4.7% (50.6%-53.0%)
Translation: Colorado had the 9th largest turnout by %. In 2010, the turnout was 50.6% and in 2014, it was 53.0%. So, Colorado turnout was up from the previous midterm but not from 2012.
Sam, the numbers *I* want are *not* percentages, but actual body numbers in IA NH NC CO so I can figure the differences between 2010 turnout and 2014 turnout? *Where* does one find *these* numbers? Also, man, I cannot come across how many precincts/wards there are in X state?
I need these figures to be able to say, so-&-so won or lost by 23 votes (or whatever) a precinct — the number that shocks or surprises the Phoner & moi as a citizen.
*I* can do the math — I just need the underlying numbers not in percentages which, like, to me don’t really tell me the story of turnout or win magnitude. If you or anyone knows where the body-count numbers & precincts per state numbers are, gratitude atcha . .
Wendy –
Is this part of what you’re looking for?
http://www.electproject.org/2014g
http://www.electproject.org/2010g
As for precincts, best source I’ve seen is here:
http://projects.iq.harvard.edu/eda/data
That first document has the precinct level data state by state from 2002-2012. If all you want to do is know the number of precincts per state, you could probably get it from each individual 2012 document here and put it in a spreadsheet. The number of precincts won’t have changed enough from 2012-2014 to make a big impact on the number you’re talking about.
The individual policies of the Democratic party when grouped into headings – do you like pre-existing coverage, do you like your children on your plan, do you like Social Security, Medicare, Medicade, Medicade expansion ( even in southern Republican states that don’t have it but want it ), equal pay, equal marriage rights, minimum wage increase, student loans decreased, environmental protections – all of these things the national public supports. The Republicans want to do away with or curb the strength of any of these. So to say that Democrats stayed away from the policies in 2014 would be the same as saying they stayed away from the policies in 2010, but completely changed their minds two years later. The very real problem of midterm turnout is an historic one for the Democrats. Having said that, to be sure, all the policies I’ve listed will be under sustained attack from the new Republican senate. By ignoring a bipartisan bill that would actually help the GOP, the GOP are now poised to conduct primaries on a platform of deportation. By going after Obama’s remedial actions, and by going after these individual policies, they corner themselves into an electorally unsustainable narrative.
Here is an article from HuffPollster discussing 3 likely reasons for the bias/skew. http://www.huffingtonpost.com/2014/11/14/polls-republican_n_6158158.html
Summarizing:
1) Incumbents limited by President’s approval
2) Undecideds broke late predominately for Rs
3) Non-voters in poll results because of increased difficulty in finding poll respondents.
I can understand why all these factors may be in play, but not why late polling would miss these factors. For example, the president’s approval was low, but was low for all 6 months before the election. It should have been baked in to the voter screens.
Similar discussions for late breaking undecideds and non-voters in late polls. This stuff should be baked into voter screens poll by poll.
Long-term forecasting is only as good as the voter screens of the individual polls somehow matching the actual turnout some 5-6 months later.
Nate Silver sees evidence of pollster-herding:
http://fivethirtyeight.com/features/heres-proof-some-pollsters-are-putting-a-thumb-on-the-scale/
–bks
It seems to me that gerrymandering can also lower turnout. This then affects the popular vote. If Democrats are more likely to be in packed districts, they will be a bit more likely not to turn out to vote because they feel their vote doesn’t matter.
So I was wondering if there’s some way to correct for this effect to determine what the popular vote would be in the absence of differential turnout across districts. It seems to me that polls on the House generic ballot might actually be a more accurate reflection of the true popular sentiment than the actual House vote due to issues like that.
The pollsters didn’t get it wrong.
The Senate fell because of the Crosscheck program used to take the vote away from a nontrivial part of the electorate:
http://www.gregpalast.com/the-secret-lists-that-swiped-the-senate/
Palast frustrates me, because I value his journalistic efforts and he’s usually reporting on legitimate problems, but he always feels the need to make the most expansive claims possible so he can insist that any election Republicans win has been stolen. But the problem in claiming that is estimating the counterfactual. (This was also the issue with his claims about the 2004 presidential election being stolen.)
Here, he takes the number of registrations invalidated by Crosscheck and implicitly assumes that every one of those people would have voted Democratic if it hadn’t happened. But he has no evidence of that. Given turnout among registered voters, it could well be that most of them wouldn’t have voted anyway.
I think it was obviously slimy and wrong to invalidate registrations that way, and there’s no proof that it didn’t turn the Senate vote, but he has a lot more statistical work to do to demonstrate that it did.
The idea that voter suppression flipped the Senate is just cracked. Maybe North Carolina, that’s about it.
Note that turnout in Alaska and Colorado were very good. In Colorado, executive action on immigration might have helped.
That idea is totally wrong.
Greg Palast is now claiming the Senate election was swung by fraudulent disenfranchisement under the Crosscheck program:
http://www.gregpalast.com/the-secret-lists-that-swiped-the-senate/
Palast’s claims are interesting, but it does seem to me that his insistence that Crosscheck was actually decisive depends on the assumption that everyone on the list would have voted, which is unlikely. Still, the practice of just purging common names without any other evidence (if that is indeed what they’re doing) stinks.
NY Times article in UPSHOT column today, Nov 19 – claiming voter ID laws little to no effect on results. Do you have any comment or analysis on this claim?
Here is the link to Nate Cohn’s article http://www.nytimes.com/2014/11/20/upshot/why-voter-id-laws-dont-swing-many-elections.html?abt=0002&abg=1
It seems highly doubtful that Palast’s analysis, even if correct, could have kept the Rs from taking the Senate.. They won the R states by 8% or more and needed 6 more. They own 4 D states by 17% or more and Iowa with a minimum diversity population by 8.5%.
Palast would have to demonstrate serious suppression in Alaska, Colorado, NC, and Louisana — in ALL 4 to imply that Rs would not have taken the senate.
All year we knew that midterms favor the outparty, low presidential approval, lower enthuiasm for Ds, and Red/Purple states as battleground.
As many DNC spokespersons were asking in the last 6 months: Why were Rs not doing better in such a favorable environment?
Okay, enough about 2014. Jim Webb vs. Ted Cruz? Let’s run the numbers!
–bks
Oh geez. Cruz is “not that conservative,” he says. Combine that with our ongoing breakdown in how the federal government works…a real problem.
Anyway, Paul v. Clinton seems more likely. Or maybe Paul v. Warren.
Not Christie?
Christie does not surround himself with top tier personnel. You need types like Axelrod and Plouffe etc. if you want to run for president.
Think of all the truly idiotic ideas that team Christie has come up with. The one I was truly worried about was the new research university in south jersey. As if Rutgers isn’t already neglected enough.
“… pollsters have a tough time identifying people who are marginally committed to voting …” I think that one must also look at the sampling techniques used. If sample size stays the same but undecided voters or swing voters change then sampling may have to be readjusted. As far as I can see, early polls are often conducted via phone, however, determining the demographics of phone users has become next to impossible as the data are simply not there unlike with geographically fixed land lines. And it is also highly likely that those few that actually go voting in a low turnout scenario might be staunch supporters of a certain party line “no matter what” while these die-hards might well be under-represented in normal samples.