Exceptionally Low Turnout Can Account For Polling Errors
At The American Prospect: Tweet In the home stretch, I wrote that midterm polling is far less accurate than in Presidential years. Today, in The...
The Midterm Polling Curse (Morning-after edition)
Pre-election PEC Senate aggregate: 52 Republican seats.
Outcome: 52 or more Republican seats (Alaska is not called, and Louisiana goes to a runoff).
As I wrote in The New Republic, last night’s performance by the GOP was remarkable. In close Senate races, Republicans outperformed polls by an average of 5.3 percentage points. Prime examples of that effect could be seen with Republican wins in Kansas and North Carolina, two races that went against pre-election polls.
In gubernatorial races, Republicans outperformed polls nearly 2 percentage points on average. This was enough to put Paul LePage of Maine (tied), Rick Scott of Florida (tied), and Bruce Rauner of Illinois (Quinn +2.0%) over the top. All in all, Republicans had an excellent night.
Historically, midterm polling is much more prone to large biases than in Presidential years. In 2010, Democrats benefited; in 2014, it was Republicans. In six Senate races that were polling within less than three percentage points, two were won by the lagging candidate. That is entirely in line with past results. Added to the median poll-based snapshot of 52 Republicans, 48 Democrats+Independents, the result could be as large as a convincing 54-46 majority.
Before the election, I pointed out the possibility that polling bias could go in either direction. It is likely that pollsters face a tough challenge in identifying likely voters in an off-year.
With control the Senate so closely fought, even a small bias put into question who would control the chamber. And, as I wrote, it also opened the possibility of a GOP blowout. I said we didn’t know what would happen. Maybe we can call that my Peggy Noonan moment.
Brier scores
Over the weekend I suggested Brier scores as a way to compare predictions. Aggregators and analysts did worse than in 2012, when polls did not miss any races (PEC Brier score, 0.01; scores close to zero are considered good).
I used final probabilities as listed at The Upshot to calculate Brier scores. The lowest (and therefore best) score came from Drew Linzer (DailyKos Elections), who took a Bayesian polls-only approach and ended up with a Brier score of 0.10. Coming in second was The Washington Post with a mostly-polls approach, at 0.12. Next came HuffPost, FiveThirtyEight, and Betfair got 0.14, followed by The Upshot at 0.15. And finally we have PEC, with 0.18. Although the number of “misses” (i.e. being on the wrong side of 50% probability) was no worse than the other sites, we were done in by an across-the-board lack of certainty, which we predicated on the unreliability of midterm polls. Congratulations to Drew Linzer!
Postscript: as pointed out by commenter Paul, Drew Linzer shines even more if his calculation’s performance in the several months prior to the election is included.
P.P.S.: Doug Rivers at YouGov has evaluated his own organization’s miss of actual-voter behavior, as well as that of other polling organizations. The findings seem consistent with what I’ve reported here.
Topics:
Perhaps one reason the Republicans out-performed is because they changed the rules. Many states had changes to early voting, or voter ID requirements, etc. Perhaps those changes were large enough to make the results last night “incomparable” to results from past elections…and to justify some amount of “house bias” being added back into the aggregate numbers to make more accurate predictions.
The problem with that theory is that the bonus applied in almost every close race. I haven’t looked at the numbers to see how the other races compared to polls. Although several states had late changes to voting, the effect showed up in many that didn’t.
Additionally, voter ID studies generally show the effect is reduced turnout of about 2%. It’s not nearly enough to explain the polls.
The problem with that theory is that the Dems passed an election bill in Colorado that had all their favorite bells and whistles, yet lost about as badly as elsewhere.
Hats of to Drew Linzer, eh? His prediction was impressive not just for its day-before accuracy, but for its long-term stability; a scoring system like the one you imagined in a previous post that measures the accuracy of one’s prediction over the preceding 6 months would no doubt have Linzer sweep the field. In 2012, same thing: he was as close as anyone else, but much further in advance, with far less wobble.
It makes me wonder whether most or all of the jitter we see in the PEC metric has to do with quirks of which polls are most recently released when, and little to do with changing dynamics in the race.
An interesting exercise to that question: take all the polls you used, and for each poll, interpolate all the missing days. Then recalculate the meta-margin, but instead of using the n most recent polls, use the interpolated value for _all_ polls for every day. (Because of the interpolation, it’s a calculation that can only be done in hindsight.) Whether that smooths out the meta-margin’s twitchiness tells us something about how sensitive the MM is to which polls have published most recently.
Yes, that would be interesting. I have to say, though…Drew did so well that I am considering hanging up my hat!
I would hate to see that!
Through election after election, PEC has provided data-based predictions and a refreshing sense of sobriety. Please continue!
Please don’t exit the field because of a comparison of accuracy. That is just one way to assay your contribution. Your approach helps clarify the issues, at least for me. It is true that after watching aggregators for a few years I am nearly sold on the Bayesian approach. And it is also true that after this election I have a bit more understanding of and appreciation for the role of “fundamentals” à la 538. But that merely points to the value of several different perspectives. Your day job is surely more important than political prognostication (and presumably more rewarding) but it is great to have such cross-disciplinary analysis. I hope to still be reading your posts regularly in 2016. (Having said that this is the last time I will mention the 2016 campaign for at least 18 months…)
Thanks for the roundup.
Sam – I think comparing your analysis to the Noonan piece is a bit unfair to your objectivity. Yes, you both start with the premise that an unexpected blowout is a distinct slice of the probability pie. Then she goes on to pontificate about little kids eating hot dogs wishing for a Romney America and how archbishops wonder if President Obama was too fidgety to lead. She’s got flags and NASA and wrecked cityscapes and secret dreams.
Predicting is a tough game because a predictor puts into writing things that could turn out wrong, making them look foolish when they are. However, having read your blog for a couple of years, I can safely say that you have NEVER written anything as open to becoming hilarious kitsch as Peggy Noonan’s now-comical love poem to Mitt Romney, and ode to the America that was so clearly about-to-be, but never was.
But if you want to evolve into a Noonan-esque writer, we would still read you. “Bias can occur in either direction, like the wafting scent of a fresh-baked pie just out of the oven, the kids clutching the edge of the counter in anticipation, their tension one of the more delightful manifestations of childhood joy; the smell of butter and pastry thick upon the air, they don’t know if the pie is apple or cherry, but there is a hope, and a dream, and for the moment they know all is right in the world.”
On second thought, please don’t write like Noonan.
It is at times like this that I wish I could Like a PEC comment.
Sam always took care to remind us that fat-tailed distributions have two tails. As the climate scientists always say when people try to squeeze optimistic calls out of the error bars, uncertainty is not your friend.
The whistling-past-the-graveyard was mostly in his comment threads.
Dr. Wang, thanks for all the work you’ve done and (with luck) will continue to do.
One note is that Mass Governor should be added to the above chart. Baker won by 1.9%. I seem to recall the polls had him around +3 but I could be wrong on that.
Thanks. Numbers weren’t final when I constructed it.
Re: that postscript:
I’ve long wanted to see a fever graph of all the analysts’ prediction error over time. eg “On Aug. 1, PEC’s prediction was off by XX%.”
Getting it right the night before is all very well and good, but real moxie is calling it correctly, months back.
Might as well toss in fever-graph lines for the prediction markets as well, as long as we’re about it…
Linzer won the 2012 cycle walking away as well (pretty much the same results as Sam but he settled on them much, much earlier), though his methods were slightly different then.
Perhaps we should keep paying attention to this guy.
I’ll echo others Sam in hoping that you continue to do this: having more models out there approaching the same issue from different ways is always a good thing.
I said last week that I’d admit if I was wrong about CO, and I was, sort of. The Democrats actually got close to the R+5 electorate in the end and it appears if the youth turnout hadn’t been so apparently abysmal, they might have made it on the schedule I had hoped for, by yesterday morning, which probably would have saved Udall.
But the thing I am most interested in, which I hope someone delves into, are the Hickenlooper/Gardner voters. It looks like the final result be will somewhere in the neighborhood of Gardner +4, Hickenlooper +2, or around a six point lag for Udall.
Looking at the big metro Denver counties right now (Adams and Arapahoe are still tallying and given the movement in them, are probably adding D votes): Denver: H+52, U+46 (six points); Jefferson: H+5.4, G+0.5 (six points); Arapahoe: H+6.2, G+0.5 (nearly seven points); Adams: H+3, U+0.1 (three points); Boulder, H+40, U+38 (two points); Douglas: B+22, G+30 (eight points).
In this day and age that’s a big ticket split, 6% of 2,000,000 (the current estimate for total Colorado voters) is 120,000 people. It’s also possible at this point that the Democrats maintained control of the Colorado legislature (both houses). So a big chunk of voters decided they didn’t want a Democratic Senator anymore, but still wanted Democrats as their local representatives.
Now I know the raw exit poll data is a bit unreliable, but I looked at it in case it could present any theories for further exploration. It appears from the exit polling that Udall did just as well among Democrats as Hickenlooper did, but Hickenlooper won independents 49-42, those who somewhat disapprove of Obama 54-43, voters who indicated the economy as their main issue 48-45, independent men 47-43, independent women 54-41, non-married women 71-28, and lost married women by 2, men overall by 8, winning women by 12.
Udall lost men by 17 and won women by 8, lost independents by 8 and independent men by 14, only won independent women by 1, lost married women by 14, won non-married women 66-30, lost the somewhat disapprove of Obama group by 18, and the economy issue group by 8.
It’d also note that 29% said they voted to express their opposition to Obama (fun with polling errors, 3% of this group voted for Udall).
So the exit polls point to independent men and women (particularly married women) who are unhappy with Obama (but not virulently so) and worried about the economy being the main difference. Hickenlooper won this group and Udall lost it (and it’s also possible that this group is helping CO’s state legislature stay blue).
So at least in Colorado, the exit polls support the mainstream media’s theory of this being an anti-Obama election–there was a block of voters who wanted to express their displeasure with the president but when it came to their local government, wanted to keep Democrats in control.
I’d love to see someone smarter than me with better access to all the data take a deep dive and see if this is indeed what happened.
Thanks, great effort.
Because of that opposition-to-the-executive mentality that many independents have, maybe what’s needed to turn both houses of Congress blue, is a Republican president. Urgh.
Before the fact, people knew that the polls could be off in one direction or the other.
Had the polls been off in the direction of the Democrats by a couple points, instead you would’ve looked the smartest, having been the most conservative about the accuracy of the polls.
Everyone today is asking how the polls could be so wrong, but yours was the prediction that was most open to that possibility. But you nor anybody else thought it was likely they’d be off so much in favor of the Republicans, or offered anything much more than that the polls could be off in either direction. In that sense, given how far they were off, your uncertainty seems justified. The others were not betting on bias against Republicans in the polls, they were betting on accuracy. In particular, WaPo missed on NC, and had NH gone the other way as all the other races did, they would’ve missed big. The others didn’t predict a wave.
But in any game of chance, you’ll win some, you’ll lose some. If you took the same approach next time, and the polls missed in the opposite direction (flipping towards the underdog, that is, not the Democrats per se) you’d come out ahead.
The real question you should be asking, then, is whether the magnitude or direction of bias in the polls was foreseeable. If it was not, don’t beat yourself up too much.
The other thing you might want to do is some kind of measure based on the margins the people predicted. If you can incorporate the uncertainty they gave, that helps too. Predicting an Orman win by a narrow margin isn’t much worse than a Roberts win by a narrow margin, given the final margin, and nobody predicted a Roberts blowout.
That is, if someone predicted Roberts by 2% +/-3%, and he won by 11 pts, they were off by 3 sigma, and someone else predicted Orman by 1% +/-5%, they were off by 2.4 sigma.
Isn’t the latter a better prediction in some way?
In a binary sense, your predictions were worse, but it seems like if you based it on margins you might beat some of the other models…
I think the Brier score gives too much credit for certainty and… well… any score is obviously going to be subject to luck. I hope you continue with your analyses.
Your pre-election analyses of past races and systemic errors lead me to a final pre-election prediction that results possibilities are bimodal. Either Republicans win 53+ Senate seats or Democrats win 51+ Senate seats. All the prognosticators have been saying all along that it is equally likely the results could be biased in the opposite direction. So let’s consider if results had been biased by exactly the same degree in the opposite direction, moving each result 10.6% in the Democratic direction. The Democrats would have won in Alaska, Colorado, Georgia (marginal runoff), Iowa, and North Carolina, resulting in 51 Democratic Senate seats (counting Georgia) and your predictions looking the strongest.
But should we have a bi-modal prediction of off-year election results where we assume the modal result will either be about +2.5 D or +2.5 R versus the polls? I think that would more accurately have captured tonight’s election and most others.
(In Gubernatorial races, it would only have shifted Florida and Michigan into the D column).
Dr. Wang,
Given the lack of complexity in your model, it did surprisingly well. Everyone was blown away last night. Who would’ve thought VA was the state that may have a recount? Or, that R would win the MD governorship? No one,except those wearing foil hats and partisan hacks.
I believe, after the post-election autopsy, we’ll discover the R GOTV was at least as effective as the D GOTV. This doesn’t mean the D GOTV efforts were in vain. To the contrary, in my opinion, they saved many candidates, including senators in NH and VA, as well as governors in CO and AK. It’s just that the Rs did a very good job.
BTW, don’t beat yourself up. You’ve done a wonderful service and the model actually didn’t perform badly, even by your own measure.
Thanks and God bless.
looking at the Senate numbers one more thought occurred to me regarding Alaska. In 2008, there was over a 3000 vote shifted towards Begich over the following weeks after 100% of the precincts were in. It’s entirely possible that when the dust settles this year, Sullivan will win by less than 2%.
Alaska is notoriously hard to poll. But it seems that Alaska may turn out to be the single most accurately polled Senate race as far as the final snapshot goes.
Thanks Sam for all the data. Is there any analysis on the turnout, county by county, comparing them to the previous elections available somewhere and if it had any impact to the outcome? I think turnout may not have had a significant impact to the results since the spread between the winners and losers are quite large (except close races like Florida Gov)…. any thoughts?
Ditto. This is my 1st year at PEC. Is there an analysis of turnout by state, new voters by state, value of D or R GOTV, etc — simply Is there anything after the election.
In 2010 , 90 million votes, 2014 83 million. Sounds like no GOTV was significant. There was lots of discussions about the value of GOTV, Ballots by mail in CO, native Americans in Alaska, Blacks in Georgia, and many other activities that would invalidate polls and change the base dynamic of a midterm election.
Why were the polls off by so much? Did they over-sample cellphones ? Incorrect voter screens?
The Demographics of results should clarify some of this.
Paul Begala on CNN tonight stated that the country wanted to send a message of disapproval to President Obama. If true, this may have made Rs and Is more motivated and kept Ds at home. This is a huge difference of opinion between Democrat pundits — Would more focus on Obama policies have turned this election around? I would like to see the exit polls that relate to National policies.
Sam, when observations deviate significantly from our expectations, that’s when things are getting exciting and we learn something new.
There will be many analyses of polling performance in the coming weeks and months. As you have pointed out before, the original problem of polling is that pollsters have to create a model of the likely voter, and there is no a-priori way to do that.
I believe we are seeing the basic limitation of this method of polling: sampling a bunch of random people and then weight the subsets to the match your model of voting population. Going forward we will need to have better ways to estimate the turnout demographics.
There was a massive right wing wave that simply wasn’t detected in polls. That wave manifested itself in a spirited vote from the right, and probably apathy from the left.
Dr. Wang, you do an excellent job with polls, and with stimulating us to discuss public opinion. Your prediction of 52-48 Republican control is pretty good. That’s where it stands now. It could go 53-47 if Begich loses in Alaska, no? Landrieu is in a run-off that she could win.
I didn’t know at first what to think about Democratic candidates who distanced themselves from Obama. Was it a smart move, I wondered? It was against my gut, and now it’s proven. I think running away from Obama and his record is terrible and sends a strong signal of weakness. Fight and die with your boots on; people respond to those who fight for them.
I saw exit polls, and though they may not be very accurate, I believe they do say something. Republicans are strongly against policies that Democrats–and I–strongly favor: healthcare for all, marriage equality, immigration reform, marijuana legalization, etc. Some of these policies are popular with people on the left and in the middle, but not with people on the right.
I think the wave was one big hissy fit from the older white (no offense, I’m white also) on the right. If we want progressive change in America, we have to force out the extremism. Now that Republicans have more power, if they remain extreme, it will hurt them more I believe when demographics will favor them less in the future.
Yes I want Obama to do an Executive Order on immigration reform, first to help people and second to play some hardball politics. Democrats need to keep the Latino vote. Republicans engage in politics 24-7 with OCare repeal and Benghazi-like stuff. Play politics also and play hardball.
Sam, don’t even think of throwing in the towel on the PEC effort.
Keep this from Amitabh in mind: “Sam, when observations deviate significantly from our expectations, that’s when things are getting exciting and we learn something new.”
He is one of the — perhaps the — most scientific-minded of the commenters here. And he makes the right point. When your experiments don’t confirm your hypotheses, you devise more hypotheses and do new experiments. In other words: Man up! And learn from this tough year.
You have actually been doing that, in a very public way. You noted (what political scientists have known for eons) that mid-terms don’t perform like presidential years. You identified some interesting patterns in the deviations from expectations from polling in mid-terms. Now is not the time to quit — it’s the time to continue the fundamental approach of the scientific method. I.e., it’s not about publishing papers or making a public splash (though those may be required to thrive); it’s about a self-learning process that gets you to closer to the truth.
This year the election truth proved elusive — in terms of precision if not direction — but you just need to continue your process of learning how it really works, and what it takes to get the most accurate knowledge of where election contests stand.
You’ve been elevating the standard of discussion of these issues. (Silver does deserve some credit for that too, some snarkiness notwithstanding.) That was what you started out to do in 2004. It has been perhaps not as easy as you thought it would be. But that is not at all the same as saying it’s not worth the effort.
Below is what I posted on my LinkedIn account at about 7PM on election day:
“Wanted to get my election thinking on record before the event:
20% probability GOP wins big
40% probability narrow GOP win
20% probability Dems + indep. narrowly hold Senate
20% probability Dems do much better than expected.
I say this because in about half of Senate midterms in recent decades one or the other party has significantly outperformed polls. So there could be a skew in either direction.”
Once it became clear what had happened, I posted:
“I was hoping it would be “Orman decides” because then he would have been in a position to force some bipartisan compromises.”
Sam,
Your analysis has for several weeks correctly predicted the general result of the election. There is no way good way to correct for measurement error when you are relying on polls done by others, and that combined with sampling error probably accounts for missed calls.
Your approach is intuitively appealing because it relies on Occam’s razor. Your analytics are the result of years of experimentation, and seem very sensible to me. Don’t forget that your work generated handsome dividends in 2012.
So something is a bit different about midterms. A much smaller portion of the electorate actually turns out to vote. Perhaps this smaller sample amplifies the uncertainty that is inherent when trying to predict the behavior of often unpredictable human beings. Perhaps there are fewer good polls during off-year elections because polling is so expensive. And perhaps a decade of analysis – just six elections – is not enough to fully understand all of the dynamics of polls and elections.
You were the first I know of to try this kind of analysis, and you have a very strong record. You have always been careful to explain your method, your errors, and your corrections. You deserve credit for creating what is now a small election year industry. I suspect many other sites doing predictions got started by following your lead.
Don’t stop now. No one has done this enough to make a convincing claim they have found a formula for reliably calling the closest races every time. And I’m willing to bet that PEC’s polls only approach will be key to finding that formula, if it even exists.
What I find most peculiar about the midterm poll errors is that they seem to be overwhelmingly large. Admittedly it’s a small sample, and I haven’t done the statistics, but it seems like we get large errors (>2.5%) disproportionately more than small errors, which is the opposite of what I’d expect if we were dealing with some typical symmetric random distribution of errors.
I’d suggest a tentative theory that undecided voters tend to break in favor of whomever they perceive to be winning (nationally). It would make a lot of sense with human psychology – after all, if you’re fairly indifferent between the candidates, wouldn’t you like to feel like “your side” won after the election? – and it would explain why the errors are smaller in presidential election years, since when the president is being voted for, senate victory feels much less important.
That said, this theory seems, at best, a plausible fit for the data, and I am no expert on these matters, so I would appreciate any feedback.
In the last month before the election Sam’s Senate control topline moved .28 to the GOP, NYTimes .09, 538 .18, 270towin .15, and WaPo .11. I was not surprised by the results: the polls in aggregate were screaming that the sky was dark with bad news for the Democratic party. –bks
I’d be interested to see how the margin numbers look when we really have all the votes in; Colorado is still counting ballots in Democratic-leaning counties (enough that a large fraction of the state legislature is still uncertain), and both the Senate and Governor’s races have shifted by about 1% toward Democrats today.
i’ve really enjoyed reading your analysis this (and past) election(s). if nothing else it has helped me more viscerally (though that seems like a bad description) appreciate basic concepts like average vs median. i have an aesthetic desire for simpler models to prevail, but long ago work on climate modeling has also made me appreciate the benefits of informed elaboration of models. i wonder there are ways to improve the polls only model in ways that might be more “complex” but also avoid the “special sauce” of fundamentals. in this vein, i’d be curious if some sort of openly defined empirical weighting of historical pollster performance might be useful. in a similar vein, i would like to see you engage in an explicit reassessment of the (few) variables that went into your model. both to see if changes in those variables might have improved the (early and final) PEC model performance and also to better communicate what went in to choosing the values used for those variables (which, having read most all of the comment threads this cycle, was never entirely clear to me in particular detail for each of the variables… i’d wanted to keep a list of the variables whose underpinnings i’d failed to grok but did not.) thanks again for your outstanding work (on an unrelated tangent, i also appreciated your perspective on academic “redshirting” on your neuroscience blog… which i urge others to check out in this liminal election period. 🙂
The highest turnout for any U.S. presidential election was 81.8 % in 1876. Of course, I don’t think we should get too excited about that, as women and African-Americans weren’t allowed to vote. But that was the high-water mark nonetheless. In 2012, it was 59.3 % – the highest in recent years. But voter turnout in midterm elections has dwindled progressively since the nineteenth century, so for those who say that Democrats didn’t turn out this week because they were specifically disappointed in Obama will have to come up with a more tangible explanation, unless they wish to apply it to every president since Martin van Buren. The turnout for the 2010 midterms was 37.8 %. This year – although naturally all states varied – the overall voter turnout for the country was 33.3 %. As Democrats have historically done better in presidential elections than midterms, it is safe to say that Democrats stay home during midterms. Just about everyone agrees with that. But when two-thirds of the country stay at home, and especially since there has been a such a precipitous drop in voter turnout since 2010, it is impossible to rule out the fact that over thirty states had in place new voter ID laws. The saddest comment on this thread focused on the fact that studies ( thus far ) have shown only a 2 % drop as a result of that. Think of that word in that argument – only. What we unquestionably know about these elections is that Republicans sweep the board when only a third of the country votes. When the vote is closer to 50 %, Democrats have a distinct advantage. A law to make voting mandatory would be fitting for a country that has prided itself historically on the right to vote. Do not expect Republicans to get enthused about that, however, any time soon.
I don’t think the voter ID laws made a huge difference, as you point out Democratic turnout ALWAYS falls in the mid terms. Overall though I entirely buy your basic argument. With an electorate as polarized as this one it’s ALL about turnout. How else does a Republican become governor of MD?
@Canadian Fan, thanks for the info. 33.3% voting percent is abysmal to say the least. Pathetic. Only 1/3 of the country voted and 1/2 of those voted elected their rep. This isn’t the representation of the country! 3rd World countries do better. I think its time to make the voting mandatory in this country (and in other democracies as well)..
Your graph: http://election.princeton.edu/wp-content/uploads/2014/11/Senate_uncertainty_20141.jpg shows a remarkable bias: always to the extreme of the band and on the side where the center of gravity is located.
It doesn’t matter “how far in” the center of gravity seems to lie, once it’s clearly on one side, the bias will be to the extreme of that side.
That made me “guess’ before the elections that the GOP would win 53/54 seats which they now do, thus confirming that exact same pattern.
In other words: this pattern now holds for ALL examples you’ve given in this graph, regardless of the party involved. I think that’s not a coincidence anymore but shows an underlying behavioral pattern of some sorts. A “mood index” of sorts?
Fascinating idea. I looked back at data to 1990 and it does seem like the party that seems to be likely to get more votes based on polls tends to outperform their polling. Maybe that is something models can adjust for.
What have we learned from 2014 polling other than it is damn hard to measure micron-sized gaps with a meter stick?
1) Most of the firms got the turnout demographics wrong. The young:old ratio (for example) was smaller than expected. Even the ones who got it right were basically guessing. We need better ways to gauge the level of commitment to voting of various subgroups.
2) Something happened in the 3rd week of September. PEC picked it up, but we do not know what it was and how widespread. When observables do something that drastic we need a credible proximate mechanism.
3) Polls are still the best way to get at public sentiment. They are instruments of direct observation, and they can be made better and more robust. The data set from this election, with multiple close races, should be helpful. The wrong approach would be to just throw up our hands and say polls will always be off so we need to add “fundamentals” according to some secret recepie. That is not science, it is alchemy.
I am softening my stance on fundamentals…but otherwise I agree with your points.
Take a look at this graph of GOP overperformance in polls as a function of voter turnout, state by state. I’ll write about it later. Today is a science day.
Sam, the link in your post is broken.
At this point, the lesson I’m taking from this race is that the fewer the people that vote, the harder it is to figure out who those voters are.
Sam you can’t quit! I’m learning way too much about statistics, polling, and prediction from reading your site for you to do that. Please keep up the great work!
This has long been my go to site for election predictions but I’m bound to say I’ve thought for a while that the predictions were understating Republican strength in the senate races. Of the seats Democrats were defending six of them were in states won by Romney most of which haven’t voted for a Democratic presidential candidate for years and the retiring incumbents had legacy support. The notion that GA, a state I know fairly well, was ever going to be close was preposterous. Outside this their pick ups were in a couple of purple states. Add in the unpredictability of turnout, but you know it’s going to fall substantially, and I’m very skeptical of polling in mid terms. With the US electorate as polarized as it is, it’s all about turnout. No science involved but I can with reasonable confidence predict reasonably accurately what the total turnout will be in 2016 and what the maximum Republican vote for their candidate will be. No such confidence is possible in relation to these state races.
This may be simplifying too much but here’s one post-mortem question to ask the pollsters: Did you get the D/R voting behavior of the various subgroups right, and make a mistake in adding them up (ie, weights assigned to the subgroups)? Or was the voting behavior of the subgroups itself mismeasured?
Of course, the subgroups can be defined by age, gender, race, income… and different pollsters probably have different ways to slice up the population.
But if the problem is the latter one, and the voting behavior of, say, the young, or women, or African Americans went more R than measured, then there are deeper systematic uncertainties than simple demographic weighting.
So an alternative model evaluation method mentioned by 538 is RMSE so I collected final projected margins for the 15 Senate Races and 13 Governor’s Races listed above from PEC, 538, RCP, EV (Senate only), and HuffPost Pollster.
The first thing I noticed was the there was little variance in the projections. The average standard deviation of the projections was less than 1 percentage point and the greatest deviation was in races where all the model agreed on the winner to a large degree (PA-Gov, MS-Sen, WV-Sen). The deviation between the average projection and the actual result was 3 times as large, just under 3%. The bias of the models for Senate ranges from +4.4 D to +5.7 D with 538, the only model to use fundamentals having the largest bias. For Governor, it ranges from +2 D to +2.5 D with 538 and PEC tied as most biased. Overall bias ranges from +3.6 D in the RCP average to +4.1 D in the 538 average.
The RMSE in the Senate races was:
1. Electoral-Vote.com: 5.966
2. HuffPost Pollster: 6.590
3. Sam Wang’s PEC: 6.610
4. FiveThirtyEight-Polling Average: 6.617
5. RealClearPolitics: 6.826
6. FiveThirtyEight: 7.207
In the Governor’s races:
1. RealClearPolitics: 3.333
2. HuffPost Pollster: 3.623
3. FiveThirtyEight: 3.724
4. Sam Wang’s PEC: 4.158
Overall:
1. HuffPost Pollster: 5.366
2. RealClearPolitics: 5.428
3. Sam Wang’s PEC: 5.563
4. FiveThirtyEight: 5.793
The differences between the forecasters are tiny.
Check my work: https://dl.dropboxusercontent.com/u/31251440/2014%20Elections%20RMSE%20calculations.xls
FiveThirtyEight’s fancy state fundamentals and polling adjustments seem to have driven down their accuracy. In fact, looking at their raw polling average, their adjustments only improved their forecast in 4 out of 15 Senate races.
This all seems to support polls-only modelling. But polls do seem to consistently miss a national systemic error that is perhaps in favor of the leader so a polls-only model that takes that into account may be superior.
The Brier score seems unhelpful to me because if you take every race that every model on Upshot agreed upon the likely winner and assign it to that party 100% and then take Kansas which was split and assign it a 50% then you would get 0.14, exactly the same score as HuffPo, 538, and BetFair. It seems to reward certainty too much.
I agree with most of Dean’s comment, 11/5, 8:40, (well said) and with many commenters on the polling uncertainty.
I especially agree with Dean that the Dems need to grow a pair and play political hardball as he suggested in his last paragraph.
I do wonder about this statement, “I think the wave was one big hissy fit from the older white (no offense, I’m white also) on the right. ” I wonder if the stats will bear this out.
Some voter disaffection because of Democratic candidates refusal to dance with the one who brung them possibly led to lower voter turnout in districts important to Dems. Who voted and why is important, and how could the Dems lose the GOTV game — run away from the President reenforcing the weak leader moniker hung on him by the GOP.
I’d also add that the electorate seems confused and that the two thirds who did not vote will get the government they deserve. Confused because many appreciate and benefit from Ocare and other policies, mortgage refinancing comes to mind, yet buy the GOP bs about Obama’s “bad” policies.
The GOP paints Obama as a weak leader and his fellow Dems run away from him: throw enough mud at the wall and some will stick. Yet weakness breeds weakness.
Obama could be a stronger leader, and now he must be a stronger leader. Too much bumbling from the White House and the Democratic Party lead to this minor debacle. It’s not the end of the world, but it is a wakeup call.
At least this should be where things get interesting instead of the same ol, same ol.
And the polling will be wild and woolly going into 2016. Lots of work to do between then and now, Sam, better get busy
I’ve said ever since the spring of 2009 that the Obama admin is abysmal at messaging. Suggested to them (like anybody reads comments at Whitehouse.gov) that they talk to a real expert in messaging like George Lakoff of Berkeley. (Lakoff has made the point publicly himself, of letting the GOP “frame” almost every issue.)
There is of course no sign that the admin has ever given enough thought to this topic with precision. Over time, IMO, they’ve just got worse at messaging. Many commenters here have pointed out that the Dems did a crummy job of painting the very positive picture of their real accomplishments in these six years. Instead, of course, they have let the GOP paint a dominant picture of what has not been accomplished. (The GOP very much being in the position of the defendant offering as his defense to the crime of murdering his parents that he is a poor orphan boy.)
First, I hope you will stay with this work. I have always enjoyed reading tis site, and looked forward to reading it at different times of the day. Please keep up the good work! Also, I am wondering if instead of focusing on point estimates the emphasis should be on estimating confidence intervals. This has been somewhat of an ongoing debate within my own social and behavioral science discipline for a while. I think people like the point estimate, perhaps because it gives an illusion of knowledge, which I do not think it does. The knowledge is better captured in the confidence interval. It is too easy to misinterpret a point estimate as giving certainty; it is easy to over interpret. And instead of a series of point estimates for individual races, instead estimate some form of multivariate confidence interval. More difficult to interpret, but more accurate in that the uncertainty IS the estimate.
When two-thirds of the country stay at home, Republicans will win, and they’ll win big. A massive GOTV effort on the part of Democrats can’t make a dent in such a depleted voting environment. Therefore the Democrats are now coming to the realization that they erred by deferring and falling in step to a Republican narrative again and again, and subsequently became defensive about policies that they really should have been proud of. Who could possibly deny that the economy and jobs have massively improved since 2008 ? Or that millions of Americans now have health insurance that didn’t before ? Democrats made the same mistake they made in 2002 and 2004. They became defensive instead of proud. The current candidates simply pushed Obama away. That was their decision, not his. They bought the Republican narrative. By doing so, they made the same mistake Al Gore made in 2000 by not taking advantage of Clinton’s undeniable draw with the base. Obama would have rallied the base. He could have made a difference in North Carolina, possibly Iowa and Colorado, and would have certainly protected the North East. He would have spoken passionately about the Affordable Care Act, that has made such a positive difference to millions of Americans. I agree with Zeke. It’s time for Obama to come out fighting for what he believes in. If he does, he’ll find many rallying to his side.
Some in the polling and aggregator community are taking insights from this election. There were some historical trends that would have pointed to the 2014 results:
-Midterm elections have low turnout and likely will continue to have low turnout.
-Voters in Midterm elections have tended to favor Rs; Voter screens were updated based on both 2010 and 2012.
– The 2014 playing field of mostly Red and purple states favor Rs
– The Rs managed to field decent candidates, not the crazies from 2010 and 2012.
– Historically the President’s party pays the penalty for voter concerns in midterms.
– The President’s approval was in the low 40s and even lower in these red/purple states.
– The idea that the economy is doing great, unemployment is down, ACA got healthcare to the neediest, the stock market is up and that message can be sold to a middle class that can not find full-time work in Red and Purple states in just not facing reality. The country may be getting better but unless people perceive that in their lives, they will vote against the group in control. Telling people that, even though their lives are worse, the country is doing better, is not going to win votes.
All this was true, the voter screens were incorrect, GOTV could not possibly overcome the mood of the country at the last moment.
The Ds have a number of advantages in 2016. We should not ignore those advantages like we ignored the Rs advantages in 2014.
While all of your assertion are correct, the salient issue was always the president. Rightly or wrongly there is a widespread and substantial perception that Obama and his White House are thoroughly incompetent. This may help Hillary if she can keep her distance as long as Obama does no further damage to the Democratic brand. The lay of the 2016 senate land favors Dems yes, but it is no longer a near certainty they will regain.
Sam, will there be a postmortem discussion of the model and the data?
I would just like to take this opportunity to gloat a bit. Yep – I predicted R-54 on this very forum when we were given the opportunity. It was based nearly entirely on what I observed of the undecided vote in the 2-3 days prior. I didn’t see the scope of the R win coming, but I got the number correct and the states.
In Colorado, Gardner’s margin over Udall as more counts came in ( 98.5% reported now) is down to 48.5-46.0% today, or a 2.5% margin, much closer to the poll average than suggested by the data presented in the early returns cited in this article (the chart shows an earlier Gardner margin of 4.2% based on a less complete vote count), it still has a R bonus of 0.5 over the polls, but CO clearly having as much if a hidden republican bounce as other states or as might have looked at first… being Colorado centric I haven’t checked to see if other states came back towards poll numbers as much as later precincts reported…
Unemployment down to 5.8 %. Over 200,000 jobs per month created, as opposed to 800,000 jobs a month lost. Americans have waited a long time for that kind of competence. An Affordable Care Act that has affected the largest expansion in health care in the country in fifty years, something that presidents for nearly a hundred years wanted to achieve but could not. Millions of previously uninsured now have health care. If that isn’t the very definition of competence and public service, I don’t know what is. The exit polls showed – yet again – that most Americans do not want the law repealed. The Republicans achieved an extraordinary victory based on the participation of one-third of the voting public. Two-thirds of the country didn’t vote – a record by far. It’s time for mandatory voting. Australia has had it since the 1920s. It remains as popular there as it ever was. Their freedom doesn’t appear to have been adversely impacted whatsoever. In the meantime, we will be able to see first-hand what Republican competence looks like. If it is to take health care away from millions of Americans, undermine social programs, or get rid of environmental controls – it will be quite an odd definition of competence to I think most Americans.
What wasn’t remarkable in the least was Democrats campaigning by apologizing for being different from Republicans. Sorry to pull rank by virtue of age; but, Dems have been subservient, opportunist cowards since FDR died and took away their backbone.
The Republican rightwing walked in – led by a populist, hatefilled band of thugs and Joe McCarthy – and anyone who stood up as individuals or in small radical bands in opposition was crushed. Democrats spent decades saying “Yes, boss. How high, boss?”
Does anyone think this inspires turnout by minorities ranging from those made visible by color and ethnicity to social ideology? Have any Hispanic friends who were galvanized into voting by a president who said – standing up for Dreamers can wait until after I see how the election turns out?
Democrats got what they deserved. The rest of us got screwed, once again.
Ask John Mc Cain whether or not an unpopular President hurt him. Mrs. Clinton is in deep trouble if Obama is still as unpopular in 2 years.
That’s why she will run against the Rep. party and Obama. She had policy differences with him while SofS that were papered over. They won’t be this time..
In fairness, Senator McCain ran in a collapsing economy where the sitting President had just sent Congress a budget with a $1.4 trillion deficit.