General overview 2012: the state of play
Welcome to all the new readers. Traffic is booming. First, a brief update. Several weeks ago I showed that convention bounces (and other fluct...
Cognitive biases of the pundit class
Matt McIrvin writes in comments: “What really drives me nuts is the multi-paragraph news story about what’s supposedly driving a shift in a single poll result.” I concur. Most horserace commentary is drivel. Market pressures reward pundits who give the impression of uncertainty. And individual polls vary enough that there will always be some change up or down. Right now there are claims that Obama is up by 9-10%. This is unlikely to be true.
Also, paradoxically, adding data to an argument makes matters worse because it creates the cognitive illusion of sound reasoning. For example, here is an attempt to argue that the Presidential race has a high chance of flipping.
With poll aggregation more popular than ever, two major points still remain underappreciated:
1) Individual polls should be largely ignored.
2) In the aggregate, polls are extremely reliable – especially if they are kept free of data that add noise.
Based on these ideas, here is an accurate picture of the trajectory, past and future, of the 2012 Presidential race. It is based on two assumptions, which I’ll detail.
The black trace from May to August is based on the following assumptions.
Assumption #1. State polls in the 2012 race perform as well as they have in past elections: When analyzed by neutral methods, they provide an index that by Election Day lands on the final outcome, within a few electoral votes and <0.5% popular vote.
The “neutral methods” in question are explained in the FAQ. Neutral means that all pollsters are included, no corrections are made, and robust methods are used to reduce the influence of outliers. In fact, you could do it manually in an hour with a calculator and a bit of patience (which is about what I did in 2004).
The prediction shown by the yellow-and-red bar on November 6th is based on the following:
Assumption #2. The likely range of variation in this year’s race will be similar to previous races of a similar type.
Since President Obama is an incumbent, we use for comparison the following defending incumbents: Bush II (2004), Clinton (1996), Bush I (1992), Reagan (1984), and Carter (1980). A caveat is that before 2004, only sparse national data are available, necessitating some indirect estimation of the range of variation.
The above two assumptions are enough to generate the following assessment: As of August 10th, Obama is up by an equivalent of 4.7%. In November, he will be re-elected with 87% probability.
I make a further assertion that goes against nearly every model out there (a number of them are listed by FiveThirtyEight):
Assertion: Almost any indirect predictor of opinion, for example GDP growth, unemployment, or campaign finance, adds more uncertainty than it adds information.
Here is one example. Recently it was reported that GDP growth helps Obama’s re-election prospects in a poll-based model. I believe that GDP information is of only limited usefulness for sharpening a prediction. GDP growth and re-election are known to be positively correlated. But we do not know if this growth is already baked into public opinion! Furthermore, the uncertainty of the estimated effect is rather large.
When a modeler runs a regression to find out the influence of variable X, he/she must also calculate the uncertainty of that influence. In addition, if polls are to be included in the estimate, I would re-pose the question as: how much does variable X move conditions from where they are known to be now?
Here is another example, which I like better, in which it was asked what effect a VP choice might have. It is based on the average gain in the VP nominee’s home state. It lacks a clear consideration of the uncertainty involved, but is otherwise an interesting point. As a very rough comparable, one recommendation would be to pick a VP from a state where The Power Of Your Vote (see the right-hand column) is high:
| NV | Obama +5% | 100.0 |
| FL | Obama +5% | 43.5 |
| WI | Obama +5% | 32.7 |
| NH | Obama +3% | 26.3 |
| PA | Obama +6% | 26.0 |
| OH | Obama +6% | 25.0 |
Paging Senators Rubio and Portman? Or…Scott Walker. Genius! Paul Ryan!
Models based purely on econometric variables still have value before a campaign season starts, to make a prediction before solid polling data are available; or after the election is over, as a research tool within the field of political science. However, during the campaign, political-science-based models are not of interest to a layperson who wants to know: who’s gonna win?
Topics:
Keep up the great work, Sam. Best polling analysis in the universe!
Yes, I saw FiveThirtyEight’s collection of prediction models a few days ago. There was, however, one striking omission: Princeton Election Consortium. Didn’t see Nate Silver refer to that anywhere. Now why would that be…
Thank you, Xtalographer. Flatterrer.
Olav: No, I think he linked here since I saw elevated traffic. I am trying to moderate my own tendencies, with only partial success. I’d be interested in feedback (private, probably) on that front.
I can’t help thinking that the more complicated models just fuzz up a relatively clear picture. To my own taste, GDP-based models are ok but ought to be kept separate from a poll-based “electoral thermometer.”
well….since my interest is the new domain of red/blue genetics or “neuropolitics”, my guess is that humans with libertarian and/or conservative organic tendency are more susceptible to asymmetrical political behavior bias.
Thus liberals make better mathematical statisticians.
/bows to the master
He does include PEC.com, but given his habit of referring to poll aggregators by the name of the individual running the site, he refers to it as “Sam Wang”.
His numbers seem to be drifting toward PEC’s over time, and his median, which you have to get by eyeballing his histogram (and certainly his mode) has always looked like it agrees almost exactly with PEC’s.
One thought I have had about 538, which I don’t think applies as much to other sites, is that Nate’s model, while built primarily to predict the election’s outcome, was also built to serve other purposes
(Such as testing out sensitivity to various hypotheticals (e.g., VP choices), enabling him to make and present certain state by state analyses, and, in general, take an approach something more along the lines of what avid watchers of sports and sports statistics like to do.)
Well, we do use a lot of the same data, and eventually polls win, so one would imagine the calculations ought to converge.
I think the hypotheticals can be gamed out more easily in a simpler model. I prefer transparency and calculations I can do in my head or in MATLAB, and without numerical simulation. That is my own shtick. He is very good at retaining many details, which is consistent with his roots as an über-baseball nut.
Keep up the awesome job Sam. I’m an avid follower of your site. I find your method to be simple enough yet realistic and robust — which I highly prefer compared to Nate’s method. I remember clicking your website daily and asking “is Sam coming back???”, and here you are!
I’m wondering if you would do an analysis before the GOP convention to go over some of things you’ve noticed in past convention bounce and whether they tend to stay or it’s a temporary one.
Well, it will be interesting to watch what happens now that it looks like Ryan will be the VP pick. It will be interesting to see what happens to the poll results in Wisconsin as well as overall. This pick might energize the conservative base to a higher level than it already is. If this is a “game changer” as the press is portraying it, how quickly would you expect to see this appear in the EV projection?
I don’t think Ryan is the magic bullet. Mainly aficionados will care, and he is energizing on both sides. Anyway, I estimate that like Bain, as well as the addition of Edwards in 2004, it should take two weeks to become clear.
I read PEC and 538 both everyday. Appreciate the differences and also the fact that they seem to be converging toward the same happy outcome.