Princeton Election Consortium

A first draft of electoral history. Since 2004

Why not to want a “prediction”

August 24th, 2008, 6:30am by Sam Wang


Many of you have asked for the snapshot to be extended to make a prediction of what will happen on Election Day. A close look at my Meta-Analysis from 2004 shows the problems in wanting such a thing, and how to gauge the limitations of even the most careful of predictions.

For those of you who want predictions, there are econometric models and InTrade. But predictions are a bad application of polling data, at least until mid-October. Here is why.

A rule of thumb in estimation is that adding uncertain information adds to overall uncertainty. How uncertain is it to estimate November outcomes based on polls? Let’s start with the one example of a high-data campaign, the 2004 race.

To allow easy analysis let’s treat all changes as jumps. Jumps are hard to identify: they are variable in size, do not always follow events, and do not have clearly defined start and end times. But the point is not to be certain about any one event, but rather to get a sense of how things change over time. So let’s try. We have seven possible events:

Kerry meta-analytic EV estimate history in 2004

Meta-analytic EV estimator for 2004 Kerry-Bush race

6/23 – Fahrenheit 9/11 opening day (Kerry +10 EV)
7/6 – Edwards VP nomination (Kerry +30 EV)
7/26 – Democratic convention (Kerry +35 EV)
8/5, 20, 26, 31 – Swift Boat ads (Bush +60 EV)
8/30 – Republican convention (Bush +50 EV)
9/30 – Debate #1 (Kerry +40 EV)
10/8 – Debate #2 (Bush +20 EV)
(10/13 – Debate #3 – no measurable effect)
Events ranged from 10 to 60 EV (median size, 35 EV). The average inter-event interval was about 20 days.

If a similar density of events occurred this year, there would be about four events between now and Election Day.To spare you the math (sorry, fellow geeks), four random-amplitude events of 10-60 EV give a total net change of 0 +/- 76 EV (mean +/- SD). That’s a big change. There is a rule of thumb that two-thirds of the time, a number taken from a distribution will be within 1 SD of the mean. Therefore, in this case, one-third of the time, the change will be more than 76 EV in one direction or the other. Unless the current snapshot were to be above 269+76=345 EV for either candidate, a prediction done today would not be worth much.

As Election Day approaches, major events will occur and/or time will run out them to happen. As future possibilities are replaced by actual changes, the uncertainty will decrease. Once this uncertainty is much smaller than the Obama-McCain EV margin – perhaps in October – we can think about making a prediction. Until then I end with an excellent adage ascribed to a physicist (Niels Bohr) and a baseball player (Yogi Berra): “Prediction is hard, especially of the future.”

Tags: 2006 Elections · 2008 Election

4 Comments so far ↓

  • Sam Wang

    David, the right reason for reading this site is that its calculation is the most precise. I’m not aware of anyone else who could identify a post-houses gaffe bounce within 1-2 days.

    Regarding Silver – I like most of his commentary, and his reader base is of course enviable. There is no point in replicating the other data. That would be silly. Personally, I prefer Pollster.com, which leaves out the projection mumbo-jumbo.

    In regard to thinking of innovations, I’ll come up with more over time. The post above was one example. It gives you an understanding of the dynamics in 2004. The 2008 analysis will do the same. In fact, it already has.

  • David

    I admit, since I found this site I’ve been checking here every day because the outlook for Obama is a lot more positive than the other sites. :D (bad reason, I know)

    You’re probably right about it being better to estimate the here and now rather than pointlessly project. Nevertheless, you have to admit you sound a little jealous of fivethirtyeight since you harp on it so much in posts.

    Nate’s site also projects senate races, gets into politics more, and is updated more frequently. If you want some publicity, add something innovative to the site instead of constantly just saying all the other guys are wrong.

  • Sam Wang

    Too little data to tell. Attacks happened in August 2004 and this August this year as well.

  • Independent

    Do you see a seasonality in the EV estimates? Between March and July/August the Democratic candidate has gained about 30% EVs in both 2004 and 2008. In 2004, Kerry then proceeded to lose all of that advantage. This year, Obama has lost about 10% so far. In very broad terms, there seems to be a hint of a pattern there.