Via Andrew Sullivan, Dylan Matthews asserts that my House outlook (a Democratic takeover probabilty of 74%) is weakened by a model described over at the Monkey Cage. However, a careful reading of that model reveals problems in using it as a prediction – problems of the type I have been warning about for several months.
Here is a graph that puts their model into proper context.
I will explain – and as a bonus, address an issue raised by Kevin Drum at Mother Jones.
First, let me say explicitly that I do not find a problem per se with the Monkey Cage’s research efforts. They have considered many details which I will read with interest. However, their calculations contain huge uncertainties of the kind that political scientists – and reporters – often choose to downplay.
First, to quote Matthews’s claim:
John Sides and the team at the Monkey Cage have a model that uses GDP, the president’s party and approval rating, incumbency, and district-level presidential vote, rather than House polling. Their model gets the seat margin wrong by 2.61 seats, on average, much lower than Wang’s error. It gives Republicans a three out of four chance of keeping the House.
This is seemingly a convincing criticism. However, at its heart is some misdirection. The “2.61 seats” statement is revealing because it is too small to be realistic. It is the same weakness I detected in the FiveThirtyEight “four-factor” model yesterday: Overfitting of small residuals is basically chasing noise, and leads to massive uncertainties.
Now, the Monkey Cage crew is aware of this issue. To quote them:
The standard error for the vote share estimate is 5.6%; for seat share, 8.7%. That’s a lot of uncertainty. It means there is at least a little probability of some pretty crazy outcomes. It explains why there is still a 1 in 4 chance that the Democrats will get the 25 seats they need to retake the House, when our own median prediction is only one seat.
In other words, the “median gain of one seat” sounds precise…but is meaningless.
Let me make the point graphically. Here are our two national-popular-vote predictions plotted side by side…but with uncertainties included:
For those of you unfamiliar with this kind of plot, the data points are the values that get reported in the popular press. The horizontal lines are error bars. They indicate the confidence with which we know the median. A large error bar indicates high uncertainty.
As you can see, our two ranges are perfectly consistent – but the PEC estimate gives much more certainty – and information. In contrast, their range, from R+13% to D+9%, contains many possibilities that we can be confident will not happen in November. If the Republicans win by 10 points, I will personally wash Dylan Matthews’s car with a toothbrush.
What about seat count, the ultimate measure of House control? Same story:
In short, their model indicates a three in four chance of GOP control because their uncertainty is massive. Do you think the Republicans will attain a 278-157 majority?
In some sense, our two calculations are consistent. However, what I presented is not a complex model in the same sense, but a precise short-term projection of likely outcomes.
My general take is that the Monkey Cage model has the potential to identify the broad picture of what influences House elections, especially if they start culling unnecessary parameters in the way that I recommended yesterday. The end product will be a hypothesis that can then be tested by current polls, which give us ground truth of true conditions. Then, in 2014, they can refine their model with the 2012 outcome in hand.
As I wrote this summer, models based on “fundamentals” (GDP growth, previous seat count, and so on) are research tools that set a range for what might happen before an election season starts. To make my favorite analogy to weather forecasting, they are like what climatologists do when they warn that there may be a lot of hurricanes next year.
However, “next year” has already started. And climatologists are not of use when one is trying to identify a hurricane strike zone. At this point the best indicator of opinion is…measurements of opinion. Polls are like a thermometer that tells us what is happening now. As I have pointed out, this is why econometric models for the Presidential race have been all over the place, yet our Meta-Analysis has been tightly clustered around a probable Obama victory since July.
OK, now that we have addressed the Monkey Cage…in a second concern, Kevin Drum expresses skepticism as to whether the generic Congressional ballot is really predictive of national popular vote. Here are some comparisons of final-week polls, courtesy of RealClearPolitics:
2010 Polling average, R+9.4%. Outcome: R+6.6%.
2008 Polling average, D+9.0%. Outcome: D+10.9%.
2006 Polling average, D+11.5%. Outcome: D+7.9%.
2004 Polling average, tie. Outcome: R+2.6%.
2002 Polling average, R+1.7%. Outcome: R+4.6%.
The differences between polls and outcome range from 2.8% toward the Democrats to 3.6% toward the Republicans. This is a larger discrepancy than Presidential polls, which get within 1% when treated the same way. But it’s not too bad – and it is an error that is contained within the error bars above.
Update: Kevin Drum drills a little deeper and points out the possibility of pro-Republican drift over the coming six weeks. Hmmm…
I will certainly entertain further criticisms. There were some good ones in the comments section yesterday.