Princeton Election Consortium

A first draft of electoral history. Since 2004

Meta-Margins for control: House D+1.0% Senate R+4.2% Find key elections near you!

Obama’s continuing rise, and a criticism from Silver

October 1st, 2008, 12:38pm by Sam Wang


The EV estimator is continuing its sharp rise (now Obama 322 EV, McCain 216 EV). It is catching up with the increase in support for Obama, which has been maintained after the first debate. National polls are now at a median of Obama +5.5 +/- 0.8% (n=12, Sept. 25-30; updated 12:53am). The EV estimator will provide a more sensitive indicator of any post-debate bounce. Keep watching it.

In a recent online chat at the Washington Times (yikes!), Nate Silver of FiveThirtyEight wrote that the Meta-Analysis presented here answers a “largely meaningless question.” Coming from him, I find the comment surprising. One should be careful not to throw stones in a glass house. My reply:

The Meta-Analysis is a high-precision snapshot of where the race stands today. The history shows you where swings occurred – and where they did not. The “Celebrity” ad campaign? Effective. Obama’s Europe/Asia tour? Not effective. McCain’s houses gaffe? Bad. If you like the national and state trendlines at Pollster.com, the Meta-Analysis is the next logical step. If you like seeing the state of play in the game, and not just an informed guess at the ending, tell Silver in another online chat he and Charles Franklin are doing at the Washington Post.

And now, my own stone…

His calculation starts from the same data, then adds adjustments that include a major guess about future movement and an asymmetric assignment of undecided voters. This approach raises questions.

The latest change to Silver’s model is an adjustment to assign undecided black voters to Obama. He calls this a “reverse Bradley effect.” As many readers of this site know, I am not a fan of “adjustments” unless they are extremely well grounded. His rationale is that Obama overperformed his polls in the primaries. This is thin ice. In recent work on the disappearing Bradley effect, Dan Hopkins pointed out that front-runners often do not do as well as expected from polls. If we think of Hillary Clinton as the front-runner, then this could explain the anomaly. Another possibility is that the discrepancy came from first-time voters. In that case people who voted in that primary have, by doing so, made the transition from registered voter to likely voter.

In other ways, Silver’s methods are generally acceptable, though some of the details add uncertainty more than they improve accuracy. He is a fiend for numerical information and has a good feel for what’s interesting in polling data. For example, today he noticed that even though the debate did not lead to a jump in Obama’s popular lead, it did strengthen enthusiasm for him among his supporters.

Tags: 2008 Election · Site News

32 Comments so far ↓

  • Fran

    I am troubled by Silver’s assumption that undecided black voters will vote for Obama. Surely if voters were going to decide whom to vote for on the basis of race alone, they would have done it long ago. Those who are still undecided, regardless of race, seem more likely to be moderates who are still trying to determine which candidate is closest to their thinking and values. I think this assumption is likely to bias the results.

  • Eddie

    I’m no expert, but I find it tough to believe that McCain has a less than 2% chance for a win (if held today), given Obama would have to pick up a couple out of VA, OH, NH, CO, NC, FL, MN, which there has to be wide uncertainty about (given for one that the poll sample is different than the election day sample). I love checking out this site, but I feel that it somehow underestimates uncertainty– am I interpreting right?

    One more thing: Is it true that you expect the outcome to be beyond a 50% band half the time, and a 90% band 1/10 times, etc?– I think it would be good to directly remind readers of how often, and how far you expect to be wrong. I bring that up because you were SO close in 2004.

    GO OBAMA!!!

  • John W

    Sam — I agree with your approach, which allows you to avoid the trap of building your wishes into the projected results. Be glad that Silver has become the flavor of the moment in the press. You are better off spending time with your students and family instead of Dan Rather.

  • Sam Wang

    Eddie, the Meta-Analysis only speaks to an election held today, and not to any swings that may occur between now and November 4th. Based on the history so far this year (and in 2004), any race that is within a few points could well flip.

    This year’s race is less close than 2004. However, as the witching hour approaches I will make sure to remind people of the proper uses of the information presented here. I have purposely avoided writing of probabilities in order to avoid some types of misinterpretation.

    John W, I am in fact pleased to see Silver’s analysis become prominent. However, now that he is the big meta-analyzer it seems like it would be polite if he acknowledged those who came before him in a more positive way. Off to the family!

  • Alex Caro

    I know you’ve made this argument before, but I think it is clear that your model is more precise. You take polls, you combine them to create much lower margins of errors. Period.

    Nate isn’t even arguing that he lowers the margin of error, mostly because I doubt it is calculable given all the manipulations he performs.

    Given that precision, if I want to know what has been going on the last week, or voter reactions to political events, your model is incredibly useful. Your chart labels, which I am sure you are very deliberate in adding, give invaluable markers for key political events in campaigns. Historians are going to use your charts as primary sources for the history of presidential elections, and they give citizens a key tool to discard all the noise in media coverage of the campaign.

    Nate doesn’t even post a chart of his EV results over time. The national vote tracker is there, yes, but it is purposefully smoothed out so as to emphasize its lack of precision.

    If I could only have one statistical model, no doubt Prof Wang’s meta-analysis and chart is more enlightening. That said the 538.com site does a lot more than just his model, so I would hate to lose Nate’s contributions.

    I would say Nate gives me about a 3-4 day notice of where things are likely going in the EV count. And Prof Wang gives me the best picture of what the polls really tell us about the campaign and political events. Precise data vs good prediction.

  • Alex Caro

    Of course, with increased state polling, this model is no longer 3-4 days behind. That was earlier in the summer when state polling was more infrequent.

  • Vijay

    Sam,

    I haven’t read Silver’s comments in the live chat session, but … it’s weird for him to say that the calculation of expected electoral votes based on polls conducted thus far “answers a largely meaningless question.” I suspect that it has something to do with his competitive instinct. He’s doing this for a living, and for you this isn’t your day job.

    I see the business of going from state polls to projecting who will be the next President (and by how much) as a three step process:
    1. Converting the state polls to probabilities of a candidate carrying that state
    2. Using the probabilities to calculate the expected number of EVs for each candidate
    3. Subjecting the state probabilities to a transformation to predict where they will be come election day, and then repeating step 2.

    I realize that Silver doesn’t arrive at his EVs by this process, but it’s useful to cast his approach in this way because you can see clearly that he needs more untested assumptions in order to support prediction. To answer the question he purports to answer (and that he considers to be meaningful), one could quite reasonably stop after step 2 and say: “Hey, this is what would happen if the elections were done today. If you think matters won’t change very much from now till Nov 4, take this as an estimate of the EVs; if you want more, I will have to give you a model of prediction for events from now till election day and tell you how I do the transformation, but you may not agree with my model.”

    I think the estimate of EVs after Step 2 is the most useful one for Obama’s and McCain’s campaign at this point; they can accept it as being completely accurate (subject to your code being bug-free) and transparent. The calculation of expected EVs from the state probabilities is a straightforward exercise in dynamic programming; the calculation of state probabilities from polling data is also straightforward–my point is that these calculations don’t require a lot of untested and unknown assumptions, unlike Step 3 ; the only assumptions in Steps 1 and 2 have to do with whether the polls did the sampling well.

    Your presentation of what you have done is quite clear (and transparent too, for anyone wanting to look at the code). Good job! Don’t worry about Silver’s sniping.

  • William

    I have a very minor note about your graphs–on the first graph it is very hard to see what the current margin of error is after a large shift in the graph, due to the way the graph works. It might be nice to be able to see the 95% confidence interval on the histogram itself.

  • Fred

    I love both your sites, and read both compulsively.
    Please, stick to what you’re doing, letting the data do the talking, giving us the best possible prediction of what would happen if the election were held today. Nate’s site tries to predict the future, a completely different mission.
    I didn’t hear the context of his remark, and often things are misstated or misunderstood.
    I find BOTH approaches useful in better understanding the race.

  • Bill S

    Your website looks better and more importantly your graphs are far, far better. Much closer to Edward Tufte’s prinicpals then 538 or Pollster. There sites/graphs suffer from lots of chart junk.

    I check your site for data. I look at the other site primarily for links to articles.

  • Frank

    You now have McCain below Kerry’s lowest point, and with less time remaining.

  • Bill B

    Vijay:

    How odd that you start by saying that you didn’t read Silver’s posts and then say that ‘… it’s weird for him to say that the calculation of expected electoral votes based on polls conducted thus far “answers a largely meaningless question.”’ I usually don’t opine on stuff I haven’t read.

    That’s not what Sam’s post asserted. He suggested that 538-man believes that the Meta-Analysis “answers a largely meaningless question.”

    And that is _still_ different from what Silver said. He said that who would win the election were it held today “answers a largely meaningless question.”

    538: prediction (among other things)
    Here: state of the race and events affecting it so far

    Not the same thing.

    And really, Bill S. “lots of chart junk”? Do we really need to devolve into fanboy posting? I value this site a great deal and monitor it almost obsessively. 538, too.

    Do we need attack ads on academic political analysis sites, too?

  • Sam Wang

    Vijay and Bill S., thanks for the comments. Considering that I basically do this as a hobby, I am glad to know that the details (even chart quality) are noticed. It’s also a pleasure to have a few readers who understand the details.

    William, we’ll improve the 95% CI display soon.

    Alex Caro, I agree that there’s lots of good stuff over at FiveThirtyEight. It’s ironic that the EV projection is not that site’s greatest strength.

    Bill B, reasonable point…but I sense this thread turning into the kind of food fight one sees over at the competition. No food fights, please!

  • James

    I think that there is some merit to the assertion that the question, “Who would be the winner if the election were held today?” is at least somewhat weak (maybe not “meaningless”). I can imagine that many people view the answer to
    that question as a prediction, regardless of your careful attention to “Obama wins today” and “McCain wins today” graph labels. The construct of “wins today” is not really possible in the first place. I can see how people viewing the site, knowing that it is not possible (subconsciously or explicitly), automatically transform it into a prediction about something that could actually happen (i.e., a win for either candidate in November).

    The state-specific information on your site would seem to help with resource allocation. In addition, the history of EV vote estimates does provide information about how events helped to shape the state of the race. However, I do not believe that the “wins today” concept, the EV vote estimate for TODAY (fundamentally separate from the historical graph), nor the red line indicating a win for either candidate helps with either informational goal.

    What IS the meaning of the question (Which candidate would win if the election were held today?) whose answer represents an impossible scenario, when the other goals of your site, as you stated in this post, can be answered (by looking at relative support between the states and by examining the relative movement of EV vote distribution alongside historical events) without ever addressing it? Perhaps I’m missing something fundamental…if so I would love to hear thoughts on the matter!

    I appreciate that there is a difference between methodologies (even if I don’t understand the polynomial method of counting on your Q&A page), and it is fun to see things moving in the same general direction. Thank you very much!

    ps…I hope that this doesn’t count as food fight material! :)

  • Marvin Danielson

    It’s worth remembering that the election *is* being held today. Early voting has commenced in a few important states, including Ohio.

    Though these voters cannot, by definition, come from the undecided category, they also can’t be re-decided or re-undecided by October surprises.

    If there is a significant shift in public opinion between now and Nov. 4, these daily snapshots will be essential in estimating the influence of early and absentee voters on the final tally. Doesn’t sound “largely meaningless” to me.

    If there’s no significant shift, I don’t need 538 or Prof. Wang to know which way the wind blows.

  • Sam Wang

    James, I like the term “snapshot” better.

    What’s it good for? I don’t know of another resource on the ‘tubes that does as good a job of revealing which campaign events may have worked and which didn’t. Think of it as a very good automated play-by-play commentator.

    Other good resources are the Meta-Margin and the voting power calculator. At the moment, voting power is not so helpful, but that’s a recent development as Obama’s lead has widened.

    In regard to making a prediction, it’s not hard. All I would have to do is take a stab at guessing the range of possibilities for where the results will drift, turn that into an error bar, then calculate a win probability. However, in the process a major piece of information is lost: exactly how things are going now.

    To use a sports analogy, the other guy is good at telling you who he thinks will win. I tell you the current score.

  • Bill S

    Chartjunk, for those who may not be familiar with it, is a term coined by Tufte, that refers to the presence of unecessary ink in a graph. The idea is if the ink isn’t displaying data or enhancing interpretation of the data then it’s getting in the way and should be removed. The borders around the “Median EV Estimator” graph and the “All posible outcomes” graph qualify as “chartjunk” (but are not egregious examples). Other then that the graphs are very clean. The design of the graphs on the other websites, in comparison, are much messier.

    From my perspective the difference in the quality of the graphs reflects the focus of the different web sites, i.e. using data to probalistically estimate the outcome if the vote was today vs using data and a whole bunch of assumptions in an effort to predict the future. I suppose in some sense the answer to the question “What would the result be today?” is “meaningles” for today and every other day except election day. Nevertheless, the history in the EV estimator chart does give a good sense of where things are trending and correlates shifts with major events. On the other hand, I find the notion of predicting the future rather squishy.

    PS I monitor this site quite frequnetly too and I don’t think it is improved by directly attacking individuals for their comments. Perhaps the rule for speaking in a New England Town Meeting (the real kind, not the campaign kind) should apply.

    “Don’t speak to individuals, speak to the ideas and address your comments to the moderator .”

  • Observer

    I am in strong agreement with the ‘no food fights’ principle. That is one of the worst features of the comments sections over at 538.
    .
    But to be consistent with that, we should try to avoid food fights between Sam Wang and Nate Silver as well. In that regard, Silver said — in that online chat — that he respects Sam Wang and his methodology. For myself, I would have to add ‘one can also admire Nate Silver’s methodology,’ a model aimed at predicting the eventual outcome. Of course, such a model is immensely more difficult than drilling down to where the race stands right now. Silver may get it wrong in November; he may not even be going about it right. (His baseball work is also predictive in intent, and apparently does pretty well though.)
    .
    Sam Wang, on the other hand, seems (to me anyway) to have deliberately eschewed the question who will win in November. Too much can happen between now and then, etc., etc.
    .
    If I were a campaign manager, I would want to have models for both questions. I’d want to know as accurately as possible where we stand right now. And I’d want to know what the trends are toward election day. Ultimately, being ahead on September 15 is only useful if it is somehow correlated with being ahead on Nov. 4. And knowing who will on Nov. 4 on Nov. 3 is good; some pollsters don’t get that right; but I really do want an analysis that can get me closer to the Nov. 4 result in mid-September. A good forecast for tomorrow’s weather is nice; a good forecast of whether Nov. will be a wet or dry month can be fundamental.
    .
    Both Silver and Wang use undisclosed algorithms for consolidating state polls to come up with the likely winner in that state. Silver does sort of tackle that question, in the ‘snapshot’ lines of his state summaries. Wang on the other hand has a patent on at least part of his algorithm. Silver goes beyond this, to give reasoned analyses (with only partially transparent statistical operations) for projecting out to Nov 4. I have found this very interesting, and often useful. I think he has got a few things wrong from time to time (‘reverse Bradley’ may well be one of them). But he has helped me refine my sense of who will win and why (quite apart from who I want to win).
    .
    Hyper-ideally, I’d like a combination of these two sites: the most accurate ‘who’s ahead today’ and a projection who will be ahead at the end of the day on Nov. 4 [and preferably not six weeks later as in 2000!]. Perhaps that’s not realistic.

  • Kiran

    I like both your sites. They both give different information to the reader, and serve different purposes. You can both peacefully coexist, as both of you do an unbelievably good job of “breaking down” the data. I am on both sites religiously! I studied math in college, am a baseball junkie (like Nate) and also am an amateur poker player (like Nate). That’s why I like both of your sites. Very insightful (perhaps inciteful too!).

    Keep up the great work!

    Cheers,
    Kiran

  • gvc

    I find it troubling that Silver purports to compute the probability of a particular outcome at all.

    Frequentist statistics declines to do that at all. A confidence interval is the likelihood of the observation given a particular outcome, not the probability of the outcome given the observation. All polls that I’m aware of are based on frequentist statistics.

    Built in to Silver’s assumptions are all sorts of prior probabilities; i.e. wild-assed guesses.

  • gprimos1

    Rather than compare the methodologies I find it more interesting to compare the results of two sites and try to explain the differences.

    First, let’s look at their measure of central tendency, the number of electoral votes for Obama. The two sites come up with identical results, with 335 from Dr Wang and 336 from Nate. It is interesting to me that all the additional assumptions that Nate adds to his model do not change this measure. One could almost ask at this point what benefit do all those tweaks give?

    We may get an answer to that question when we look at the measures of dispersion as provided by their electoral vote histograms. Dr Wang’s current “All Possible Outcomes” histogram shows no(!) outcome where McCain wins. It is a resembles a very light tailed symmetric probability distribution. I am too lazy to calculate the kurtosis but it looks normal-ish.

    Nate’s “Electoral Vote Distribution” histogram is also symmetric but has much heavier tails (or perhaps simply wider variance, I can’t tell). It shows McCain winning 15% of the time. So we might conclude that all that Nate’s tweaks to the basic model buy him are longer tails on is EV distribution. This could easily be accomplished with minimal additional assumptions in Dr. Wang’s model simply be adding a little more variance to each of the distributions of the state polls.

    With these observations in mind, let’s recall that Dr Wang presents his results as a prediction of what would happen if the election were held today. I would add that there is also an implicit assumption that the state polls are correctly reporting their margin of error. People have complained about the assumptions Nate makes, but there are an also a number of sometimes incorrect assumptions pollsters make to weight their results and I don’t think the variance generated by the statistical models encapsulate that additional uncertainty. I think Nate is probably more likely to be correct (even if the election was held today) only because his model acknowledges this additional uncertainty. On the other hand, I admire the simplicity and transparency of the Dr Wang’s model, and I don’t think you need to jump through all the hoops Nate does to get to a similar result. Just make an estimate of the additional variance and slap it on there! BAM! Instant predictions of election day.

    (btw i like the :) on the bottom)

  • Eddie

    I don’t understand– was my question so misinformed that it couldn’t be shared for review? (I know you don’t have time to teach me statistics)

    I think it does appear that the site is reporting that, if the election were somehow held today, there would be a virtually 0% chance of Obama gaining less than 280 ev’s. Could you just email me if this is correct?

  • Sam Wang

    Eddie – sorry, I tend to get trigger-happy with intuitive statements of incredulity, since they are answered in the FAQ and in past posts. I will clean that stuff up soon.

    Basically, what the distribution tells you is what would happen if the population sampled by pollsters were to vote. In that context, the likelihood of a McCain win right now is nearly zero.

    The exception is the possibility of systematic error, such as a bias in polls. To affect the above conclusion the bias would have to be at least 2% to have any noticeable effect. (Note that the Meta-Margin is what it would take to bring things to 269 EV-269 EV). There isn’t a known hiding place for a large bias to help McCain. The only candidate I know of is the Bradley effect, which seems to have disappeared starting in the mid-1990s.

    gprimos1 – Your point of view is reasonable, but I want to correct one thing. I don’t use estimated sampling error, except to place a lower bound on the SD of median. This only comes up if the polls happen to cluster tightly. More usually, the algorithm uses the poll sample itself to get an estimated SD. This implicitly takes care of variability of accuracy among pollsters.

    The extreme similarity between my estimate and Silver’s is partly a coincidence, but is helped by the fact that the election is not far off. His random drift model for future changes is presumably smaller now. All it does is add uncertainty, which I have more or less addressed here. Think of it as a hurricane that is nearing landfall: the accuracy of a prediction goes up when the thing is near enough to see.

    He has a 15% win probability for McCain, about 1/6 of the area under the histogram. This suggests that his additional assumptions are equivalent to an assumption that 1/6 of the time, voter sentiment may swing toward McCain by 4% (the Meta-Margin) or more. I’m pretty sure that’s an overestimate.

    Whatever a correct model would be, I agree that a comparison with the Meta-Analysis shows you the cumulative effect of all of FiveThirtyEight’s accumulated assumptions. And yes, the best way to see it is to compare the width and spikiness of the distributions.

  • gprimos1

    Dr Wang,

    Thanks for the link it was very helpful.

    The way I read it you are basically saying that while yes covariation in the polls vrs election day performance is not being modeled here, that the difference between the two is so small as to be in consequential. For instance, while your model currently shows McCain with no chance of winning if the election were held today, if covariation where taken into account the tails would shift to where there might be some non-zero chance of winning. However, since that chance is less than, let’s say, a 2.5% threshold (corresponding to your 95% CI) we are safe to assume that the two models are equivalent.

  • Michael K

    I check both this site and 538 avidly and find both very useful.

    What I find most useful about 538 is the trend and demographic adjustment applied to states which have few recent polls out. For example, there are currently few if any very recent polls for Wisconsin, New Hampshire, Nevada, Montana, and Missouri. On 538, those states have more favorably odds for Obama based on his most recent polling gains nationally and in demographically similar or neighboring states.

    On this site, I can fiddle with the meta-margin to account for nation-wide polling shifts, but I can’t very easily account for discrepancies in the “freshness” of the polling data from state to state.

  • Michael K

    I suspect the point Nate was not so tactfully trying to make is that the question of who would win today is in some sense an academic one. What I think most of us are mainly interested in is the likely outcome in November.

    I totally understand and largely agree with Sam’s counter point that modeling future polling shifts, even on the basis of very educated assumptions, is a shaky exercise of questionable value.

    That said, I think it’s still somewhat interesting and certainly entertaining to see what happens if we apply some assumptions. Ideally I’d like to be able to fiddle with those assumptions and see how much difference they make.

  • Sam Wang

    Michael K, I agree that the trend adjustment is a valuable part of the FiveThirtyEight model. Until recently I regarded it as the one part I’d want to implement. However, the phenomenon of McCain’s post-RNC swing being concentrated in rural states spooked me. This was a demographic trend that was not predicted in advance. So I took the conservative route.

    We are thinking of what functions to add to the interactive map. Your comment makes me think that adding N% to one state only might be of interest.

  • Observer

    Sam, on 538 I’d like to see a clearer (aggregated) statement of Nate’s snapshot of state-by-state polling aggregations. He gives the data; I’m just too lazy to add them up.
    .

    On here, I too would like the ability to apply a %variance on a state by state basis, on the electroal map.

    Cheers.

  • Michael K

    A suggestion: how about a function that gives the user the option to trend-adjust the entire interactive map (as well as perhaps the snapshot graphs for that matter)?

    An appropriate disclaimer could warn of the possible flaws with the adjusted information (i.e. as you point out, that trends may not be distributed uniformly across the nation).

  • Mike LaFave

    Sam,
    I greatly appreciate all the time you put forth to create your daily snapshot (as I did in 2004).

    My question regarding your Median EV Estimator graph is how can one validly designate a single factor to have precipitated a movement in the median EV?

    Most problematic would seem to be the latest significant movement: McCain’s floundering during the Congressional funding negotiations drama and the (pseudo?) suspension of his campaign in order to claim pseudo-credit for Congressional action might have been important negative factors along with the Presidential Debate which you designated.

    If you are correct that Obama’s first major downturn was primarily related to the damage from “celebrity” attack advertisements, then the current purported decision of McCain’s campaign to go 100% negative in ads and go the mongrel in debates would appear to be a sound, if less than admirable, strategy.

  • Sam Wang

    Mr. LaFave, I list events such as debates because they have been movers of opinion in the past. Debate #1 in the 2004 history provides an example. Debates #2 and #3 that year provide counterexamples.

    I agree that the economic crisis was a major force that potentially moved opinion at the same time. National polls provide better time resolution and could resolve this question better. However, it’s still correlative. Detailed surveys of voters’ reasons and so on are even better.

    I just gave a talk yesterday to some political scientists, one of whom reminded me of the power of models based on economic variables to predict poll movements during a campaign. It was fascinating and I am going to read about that.

    In regard to negative campaigning, the fact is that it works. However, I think the particular approach has been costly to McCain’s reputation.

  • Mike LaFave

    Prof. Wang,

    Many thanks for your cogent responses. It is extremely sad for me to see McCain willing to permanently degrade his reputation in what appears now to be the winter of both his discontent and career.

    Regarding the power of models based on economic variables, they seems to work in small economies like Australia. Prior to their Federal election a year ago, a model based on leading economic indicators which could accurately “predict” election results going back 40 years was published on:

    http://blogs.crikey.com.au/pollbludger/ and http://blogs.crikey.com.au/pollytics/nsw-polling/ (both run by statisticians).

    This economics based model did generate an accurate predicton for the 2007 Aussie election (won by the Opposition who then withdrew Australia’s combat troops from the Iraq War).

Leave a Comment