Today, the NYT’s Nate Cohn speculates about the problem of low-quality polls in Senate races. It’s an interesting piece with lots for poll junkies. However, I am compelled to offer several gentle corrections. My bottom line: polls are better than he implies, especially when they are aggregated properly. And Senator Pryor (D-AR) is probably a little underwater at the moment. Oh, and Democrats aren’t as hosed as you might think.
First, how have Senate polls done in the last two cycles?
Cohn says that “polls have missed the result in three close Senate races in the last two cycles.” That statement is incorrect. The following plot comes from my Election Eve snapshots for close races in 2010 and in 2012, which show poll-based medians. They are plotted against the election results for 2010 and 2012.
Anything in the gray zones is a correct prediction, i.e. the polls and the results favored the same party.
As you can see, polls did well. Here is one metric: we can estimate how many we should have gotten correct by adding up the nominal win probabilities for the 15 leading canddiates. This sum is 13.0. And 13 out of 15 poll-leaders ended up winning. So by that measure, the polls were dead-on.
Looking at it another way, only one race was radically off: Nevada 2010, between Senator Harry Reid and Sharron Angle. Nevada has two factors that made polling difficult: a lot of Hispanic voters and an unusually mobile population. There’s the Colorado race, but that 2-point error is not exactly something to jump up and down about. Overall, I would characterize the result as 13 out of 14 correct, with one push.
At the same time, there is an interesting anomaly: in 14 out of 15 cases, Democrats outperformed opinion polls. I do not know why this is…though in 2008 I pointed out that pollsters tend to underreport margins – in both directions. Anyway, I will not be using this particular fact in my 2014 analysis (which is coming soon). As most readers know, I advocate the use of clean, uncorrected polling data, which has worked so well since 2004. However, I will give you simple tools so you can explore what the effects of a polling bias would be, same as I provided for past Presidential races. But in all the topline estimates I report, I will personally keep off the Unskewing Sauce.
This brings me to the example of Arkansas, which is offered as an example of a problem.
Cohn points out that all the polls showing a Cotton lead are Republican outfits. Counting each outfit once, the median is Cotton leading by 4.0±2.7% (n=7). However, there’s a problem: the Republicans (plus Rasmussen) were the only ones releasing survey data since the beginning of May. All the other data points are at least 9 weeks old.
It is completely plausible that Senator Pryor is behind. In April, his approval/disapproval numbers were 38/46, an abysmal position for an incumbent. Political scientists take the view that a major purpose of campaigns is to drive voters to where they “ought” to be in terms of fundamental factors. The Pryor and Cotton campaigns have certainly worked hard to do that – it’s been lively and nasty. Take a look at this, and this, and this.
Finally, I have a rather basic question. If Pryor is not behind, then where are the neutral and Democratic pollsters? I find it curious that they have not released any data. I think if they had anything to offer, it would have come out by now. As Sherlock Holmes would put it, that’s the dog that didn’t bark. Anyway, this will all get resolved by a few more data points, which I am sure will come soon. Bottom line, Pryor had better get the lead out.
This brings me to back to one of my favorite points. Poll junkies love to sniff over individual polls, offer corrections, and so on. I think it is a waste of time. Historically, poll medians do a great job. At some level, Cohn seems to agree, since he ends up concluding that aggregation is still OK. But hey, it’s July, and he’s got a column to write.
All this aside…if you’re not from Arkansas, what you really want to know is who’s going to take over the Senate. Today’s snapshot is a Democratic retention probability of 55%, median outcome 50 D, 50 R. Boy, that’s close.
Graphs are coming soon – we’re still setting up the rollout. Stay tuned!