Princeton Election Consortium

A first draft of electoral history. Since 2004

Nerds under attack!

October 29th, 2012, 10:00am by Sam Wang

Paul Krugman is calling out National Review Online for their attempted takedown of Nate Silver for biased methods and somehow cooking the books. Krugman writes:

This is, of course, reminiscent of the attack on the Bureau of Labor Statistics — not to mention the attacks on climate science and much more. On the right, apparently, there is no such thing as an objective calculation. Everything must have a political motive.

Now more commentators on the right, including Jay Cost (The Weekly Standard) and Jennifer Rubin (Washington Post), are getting in on the act. Wow, dogpile on the rabbit!

A popular approach to undermining technical knowledge is to throw mud, assert expertise, make picky points, and sow doubts among the less savvy. In this case, what’s the argument? The NRO writer, Josh Jordan, makes this core criticism:

When you weight a poll based on what you think of the pollster and the results and not based on what is actually inside the poll (party sampling, changes in favorability, job approval, etc), it can make for forecasts that mirror what you hope will happen rather than what’s most likely to happen.

Jordan sounds like many partisan polling enthusiasts – on both sides. However, his style of poll-dissection can very easily lead a person astray. The human mind has a large capacity for finding reasons to reject a piece of disagreeable evidence. I’ve written about this in the context of how people form false beliefs in politics (“Your Brain Lies To You,” NYT, June 27, 2008). Polling internals lend themselves very well to such “motivated reasoning.” It is always possible to find something not to like in a poll. This is why I discourage all of you from chewing over single polls.

Silver’s evaluations of pollster reliability are quantitative parameters. However, there isn’t full transparency about how he arrives at them and what he does with them. This leaves him open to attack.

Partly because of this risk, I have stayed with simpler rules such as

Combined with a probabilistic calculation, these rules guided our Meta-Analysis to the exact EV outcome in 2004. It missed by only 1 EV in 2008. Such simple methods are easy to make transparent. You (or Jay Cost, I guess) could download my code in an instant.

I have my own technical beefs with FiveThirtyEight (for example, see here, here, and here). I believe Silver doesn’t extract all the information and tends to add unnecessary factors, which leads to blurry probabilities and poor time resolution. However, his intuitions about the data are excellent and he is very concerned with getting things right. For purposes of popular consumption, he is a fine and honest nerd.

Jordan’s capacity for wishful thinking is apparent when he writes:

While it’s impossible to know how the late deciders will break, the historical trend has been for them to break for the challenger.

I sympathize with this, since I thought the same in 2004, and added a “turnout/undecideds” parameter. For this I received a well-deserved drubbing afterward. In fact, undecideds split about equally, as amply documented by Charles Franklin. I don’t add such parameters any more.

(However, if Jordan wants to implement his idea, he can do so easily by clicking the “With +2% for Romney” link, over in the right sidebar.)

Finally, I will state something obvious. None of this storm of criticism would be happening if “Ro-mentum” (Oct. 23) were real. In fact, Mitt Romney’s fortunes peaked around October 4-9. Since then, the race has moved back toward Obama by about 2.5 points. National polls* give the graph at left. (See update, below.)

And the Popular Vote Meta-Margin, which describes how much state polls would have to swing to generate a tie in Electoral College mechanisms, looks like this:

History of Popular Meta-Margin for Obama

The Meta-Margin may still be catching up with national polls. If so, it has a few tenths of a point to go before it stabilizes. Alternately, something different is happening in swing states. In either case, the overall picture is the same: a narrow Obama lead that is static – or perhaps widening. There is no evidence for Ro-mentum.


*Analyzed as previously described. To generate the graph above, Gallup/Rasmussen were excluded. I am not at all averse to using them, but they have large house effects and so would need to be analyzed using median-based statistics, which I did not apply above. Anyway, adding them back gives the same relative picture of rapid decline and bounceback, with the same shape, except that the entire graph is slid upward toward Romney by 1.0%.

Median of all national polls, day by dayUpdate: OK, here is the poll-median graph, including Gallup/Rasmussen, day by day. It is not a weighted median, but it’s close enough (adverse blogging conditions). Note that the drop on October 3rd is due to a Rasmussen poll, which demonstrates their house effect, about two points relative to other pollsters. The conclusion from this plot, as well as the plot above, is that the race has been at a standstill for the last two weeks.

Tags: 2012 Election · President

297 Comments so far ↓

  • A Skeptical Reader

    Sorry to dent the self-glorification of the nerds, but what PEC and 538 do is not “science.” It bears a closer relationship to fantasy football than to a systemized study of general laws through the scientific method.

    Although that sounds denigrating, it’s not. Both PEC and 538 are making general assumptions, applying statistical models based on those assumptions, and reporting the resulting odds. Are these odds correct? Who knows. You can’t freeze that moment of time and conduct a conclusive dissection to confirm that an assessment of a 9:1, 7:3 or 1:1 probability call was in fact correct.

    Yes, you can certainly replicate the results of the PEC model, but it’s based upon underlying assumptions of input data (state vs. national polls). Even the less transparent 538 model can be mostly replicated with a Monte Carlo simulation of aggregated state polls within their margin of error ranges. But their claimed mathematical precision of any of these handicapping systems is a myth because (1) they are built on certain assumptions that are correct except when they’re not and (2) they are applying a level of implied certainty on a series of observations with inherent uncertainty and that interact with each other in a cascade system that compounds such uncertainties.

    Again, this is not anti-”science”. In fact, such predictive models are extremely helpful. Bookies in Vegas can only put shoes on the feet of their children because such models operate far better than base intuition (or the older heuristic models used to replicate such models in a pre-computer era). However, at the end of the day, any bookie recognizes that his success is not based upon infallibility of his statistical model but by his ability to bake in the house advantage to reap his profits and to account for the inherent variability of results vs modeled performance.

    What the bookie also knows is that models fail. Sometimes quite badly. It is somewhat ironic to read the comments on PEC extolling the absolute scientific certainty of Silver and Wang et al. This was the same attitude of the financial community in predicting the credit performance of compounded derivatives to extremely high confidence levels of statistical precision. I guess the 99% and the 1% have more in common in their confidence on achieving their preferred outcomes despite inherently large and unknown uncertainty ranges.

    Last comment. Before we decide to canonize Dr. Wang and Mr. Silver as paragons of infallible predictions, let’s review the tape: 2010 mid-term elections, both models performed badly. I believe Dr. Wang noted that there was clearly a systemic error, but I’m less sure if he ever identified exactly what that was. I also find it instructive that Mr. Silver was a professional online poker player. Now, poker’s a game with a finite set of outcomes and wholly absolute set of inputs in which to calculate statistical outcomes. No polling errors here. Mr. Silver did quite well … until he didn’t and his losses of $130k (or as I like to call it, Mitt Romney ante money) prompted him to seek new territory in which to run his Monte Carlo simulations. Far better to call the odds on somethings that so much less … provable.

    • orchidmantis

      No one is attempting to canonize them as paragons of infallible predictions. We are reading with interest their aggregates of polls, with a fair level of confidence in light of how they have done at predicting past elections based on polling data.

    • orchidmantis

      But if I had to canonize someone, I’d choose the nerds, and math, and science, over the Republicans spamming the threads to explain that the model does not account for their special feelings about how Romney is poised to take PA. Over, say, Mourdock of IN.

    • Shawn Huckaby

      And on the subject of science, skepticism, and reason, I’d like to drop a short plug for one of my favorite podcasts, the SGU:

      Occasionally sophomoric, frequently geeky, but usually both challenging and entertaining.

      I think we could all do with a bit more critical thinking now and then.

    • John Sawyer

      “the model does not account for their special feelings”


    • Obama 2012

      1) it doesn’t seem reasonable to compare House races with the presidential race. there’s a ton more polling data for the presidential race.

      2) I’m not sure about the 2010 House performance here – but 538 got close in the end.

      3) his bias (such as it was) was in favor of the GOP in 2008 (only state he missed was Indiana which he called for McCain and went for Obama) and in 2010 (senate races called for GOP that went Dem.)

    • wheelers cat


      this is latest instantiation of polltrutherism– that the poll aggregators are some how “all in” or “doubling down” or “bookies”.
      The implication is a subjective application of the data in service of the aggregator’s secret agenda, ie that Linzer, Silver, Blumenthal and Dr. Wang are somehow gambling on the outcome.
      Actually they are all using sophisticated mathematics to peel away the uncertainty to see the true shape of the race.
      Citizens like Skeptical here just don’t like what the peeling reveals.
      Sean Trende and Jay Cost have constructed an elaborate bimodal model where the good polls (pro-Romney) form one curve while the bad polls (pro-Obama) form another.
      But the problem is, like Dr. Wang shows in his election stasis graph, that the good polls for Romney are few. And if you remove just Rasmussen, the result is remarkably consistent.

  • Terrence Sherry

    Nate references baseball; Sam links to Looney Tunes shorts. I’m staying here :-)

  • Randall

    Nate Silver gets a lot more respect than he deserves. His model is close to unintelligible. He got 49 states right in 2008 when it was Obama versus McCain / Palin. I’m pretty sure anyone commenting here could have done the same if they had thought about ir beforehand and tried to do it. Right now, Silver has Romney at about 25% . Silver is hedging. And when he is not writing about the data, he invariably takes hard partisan positions on the Democrat side of the issues. Speculation that he is running an agenda is reasonable.

    • DanF

      Do you read Nate’s blog? His model is pretty well articulated for anyone with a reasonable understanding of statistics. As for his tone, he has clearly stated that the tone of his blog is dictated by the poll results on any given day. If the results are good for Mr. Romney, he speaks positively about Mr. Romney. If they are good about Mr. Obama, he speaks positively about Mr. Obama. Lately, polls have been better for Obama. He is exceedingly careful to maintain his neutral stance. How anyone can call his posts partisan is beyond me.

      By contrast, Sam wears his beliefs on his sleeve, but like Nate, he lets the numbers fall where they may. He’s model is completely transparent.

      Ultimately, I think both Sam and Nate are honest brokers with nothing to gain by rigging the game, and more to lose.

      And no, most people could not have predicted 49 of 50 in 2008. Professional pundits were all over the map as were pollsters. You don’t make your bones by doing what any average Morning Joe can do.

    • Eric

      Randall, you’re just repeating talking points. This site has Obama as a better chance than 538! If anything, Nate leans conservative when compared to similar sites.

    • Randall

      What “talking point” am I repeating?

      This is tiresome.

      The claim that criticism of Silver or speculation about his motives is ideologically-motivated, or a so-called “talking point,” is itself a *talking point*.

    • orchidmantis

      If his miss on the presidential in 2008 was to call Indiana R when it went D by a whisker, and his miss on 3 Senate races in 2010 was to call them R when they went D, then isn’t it logical to conclude that he is erring Republican?

      His agenda, like Sam et al, would appear to be to call the election right, which will result in people paying attention to the model in the next elections. Exactly what would be served by predicting an EV win for Obama (along with everyone else) when the data don’t show it?

    • Travis

      You must feel like a genius now. Hard to do better than the 95% chance he predicated in 2008, but 100% is certainly better than 95.

  • StatsJunkie

    I don’t know about “running an agenda,” but a serious caveat about Nate’s work is that he’s never really covered a “neck and neck” election in real time. (He’s only written about 2008 and 2010, both of which featured rather healthy margins that weren’t that difficult to call. We don’t know how accurate he might be when the race is “too close to call.”) By contrast, Sam did call the much-closer 2004 election to perfection.

    • Sam Wang

      That is an interesting point about 2008 and 2010 being relatively easy races. I ran into a little trouble in real time in 2004, but I agree that I learned a lot from the challenge. This race looks a little like that race, only shifted about 2% toward the incumbent.

    • Matt McIrvin

      The thing is, the top-line electoral-count results that 538, PEC, RCP, and all the other state-poll aggregators cite just come from state polls, and with the possible exception of Votamatic with its strong prior weighting, they’re just not that different from each other. They mostly differ in how much certainty they have in the result.

      So this is all about whether you can trust aggregated state polls, and anyone can aggregate state polls. Kvetching about Nate Silver or somebody having a partisan bias is completely beside the point.

      I am definitely interested in the hypothesis that 2004, ’06 and ’08 were races that were unusually easy to call with state polls (and races more like 2010, where the aggregates were further off from the results, are the norm).

      Silver recently made a fairly convincing post arguing that the only presidential race where state polls were horrendously off was 1980, and that was in a world that did much, much less state-level polling than we do. Conservatives have been hoping for another 1980 ever since the beginning of this campaign, but that year looks like a real outlier, and not very similar to 2012.

    • Sam Wang

      Matt, I agree. Poll aggregation is not rocket science or weather prediction. It is much simpler.

      I do think there are semi-optimal ways to analyze the data. I would say that RCP/ could use improvement. I would be delighted to get them together for brainstorming. I like, though I want to dig into their methods a bit to see if they could do better. My own approach are dictated by (a) a desire to keep things transparent and simple, and to prove the point that a simple approach works, and (b) the fact that I don’t want to be forever adding little things here and there. It’s at a level that Andrew Ferguson and I can handle as a large hobby.

    • Matt McIrvin

      Part of this may be just a reaction to the general hilarity with which Unskewed Polls was received: a poll aggregator who was clearly tweaking all the poll numbers he got in service of a Republican pipe dream. Nate Silver, being an avowed liberal, naturally must be doing the same thing in a liberal direction.

  • yellowdog

    Sure, there is a chance Nate Silver is running his model to produce a result he knows his readers want to see. However, his future reputation–that is, what those readers think of him after next week–depends in great measure on his model’s ability to produce a prediction that is close to what actually occurs. A liberal modeler in 1972 might have predicted a McGovern win because he/she agreed with McGovern on the issues, but that person would have a serious lack of credibility about 10 minutes after the polls closed and reality began to bite. I doubt Silver is willing to trade his long-term reputation as a model-builder–which is his livelihood–for short-term popularity with Obama supporters. Silver gained much of his reputation by getting 2008 about right; his 2010 model, though, under-predicted the GOP wave. His 2012 work means a lot to what people think of his analysis in the future. As he himself would probably say, though, this is still small-n territory.

    • ThatGuy

      Exactly, talking about bias in sites like 538 is silly. The site is popular because it proved accurate in the past. If it suddenly fails this time, greatly overestimating Obama’s electoral college advantage, then I, along with many other readers will simply stop paying attention to it.

      Any commentator earns their money by keeping their readers happy, and while on that hand it makes sense for conservatives to blast a site that disagrees with their perception, as that is what their readers want to here, a site like this, or 538′s would do huge harm to themselves by offering biased results. They’d be gaining a short term traffic boost, at the cost of any long term readership/credibility, and as people that deal in numbers, I’m sure they understand that the tradeoff would be very bad.

  • ML Henneman

    The worst part of the right-wing attack on Silver on other sites is that it used to have a high quality discussion thread for those of us interested in the nuances of statistical modeling. Now the trolls have been set upon it with their ignorant opinions about the election based completely on cherry-picking polls (which Silver has written a great post on recently) and the comment thread is not fun for us nerds anymore. Sigh.

  • BlondeWhoLikesStats

    Great reading your opinion about the sometimes critical media being focusd on election math and Nates blog. Observing the mounting increase in intentionally false paid polling groups that pollute among the authentic ones…if too many bad polls are partial to only one political side this would change/tilt the median methodology?….this would unfairly *buy* the stats analyzed/reported through the midst of most of an election cycle until the very last week or day when all the bad polls *magically* align to reasonable truth in order to cover themselves? Controlling the news narrative with cheap false polls can drive down enthusiasm to vote but it is a fast growing problem. Which brings me to the subject of Arizona…. how come more good quality polling is not being reported in Arizona as a swing state this election? Closer attention study of this weeks AZ polling is needed to get rid of old bad polls which election maps are using…..Arizona is currently possibly a tie but reporting polling seems distracted away in EV count updates? Why does the Power of Vote stats on this site not seem to have AZ listed? GOTV power not shown yet for AZ !

  • Justin S

    Scarborough was really at it again this morning. He really has it in for Nate Silver. He wants a 50/50 race. This is not a 50/50 race. Can Romney win? Sure, but an Obama win is far more likely and thats evident to anyone who is looking at the numbers coming out of the states. And he knows it but continues to sell misinformation and trash Silver. I’ve been a long time viewer but I’m about to quit watching his show

    • Michael

      I’m not so sure it isn’t pretty close to a 50/50 race. Aside from Gallup, the recent national polls do pretty much show a dead heat, and there is nowhere near enough polling of the states that aren’t in doubt to say anything definitive about the national polls being either right or wrong. It’s true that the right is pretending that Obama’s structural EC advantage either doesn’t exist or doesn’t matter, but it’s also true that the left is in a bit of denial about the national polls being as close as they are. And finally, there’s the simple fact that Scarborough and other pundits just don’t understand that Silver’s 75/25 actually is saying that the race is close.

    • Matt McIrvin

      And he still doesn’t even realize that Silver, unlike this site, practically *is* describing a 50-50 race. 27.1% probability is a lot! That’s something that could totally plausibly happen.

    • orchidmantis

      Next you guys will claim you’ve seen someone throw two heads on a coin toss. Pff. 25% < 50%, so 25% = 0.

  • ChrisD

    Scott Galupo’s take on “Nate Silver and the Nerd-Haters” from The American Conservative:

  • Niels Rosenquist

    This is all about avoiding a self-fulfilling prophecy on the GOP side, where the current state of the race has an influence on voter participation.

    The sheer ignorance of talking heads (esp GOP ones) about basic statistics and modeling astounds me (and I have a pretty low bar of expectations).

    • orchidmantis

      I see it as one sign of the return of the neocons, convinced that what they say reality is can cause reality to take on that form. So far working only patchily. (See Benghazi fiasco at the 2nd debate, or the blowback on their claim Jeep was closing Ohio plants.)

      So if they can defeat Nate Silver, and get him to say his model predicts a Romney blowout, then there will be a Romney blowout. So long as he (and Sam and the others) are out there claiming reality has a different form, they’re a threat.

  • Norm

    Mr Wang…

    You are an arrogant fool. What will your excuse be when you are wrong. If you are an example of what Princeton passes as “academic community”, said community is loaded with imbeciles. Can’t wait to read your excuses for being so willfully ignorant next week!

  • Ms. Jay Sheckley

    OMG it left out the quote. Sorry. Again
    Sam writes: == The human mind has a large capacity for finding reasons to reject a piece of disagreeable evidence. I’ve written about this in the context of how people form false beliefs in politics (“Your Brain Lies To You,” NYT, June 27, 2008). Polling internals lend themselves very well to such “motivated reasoning.” It is always possible to find something not to like in a poll. This is why I discourage all of you from chewing over single polls. ==

    I wish everyone could read this. I’m going to the article link. But what I imagine the article is about is that we’re all drugging ourselves.

    Facts! Interesting! LOVE THIS SITE

  • BrooklynKevin

    It’s a great site you have here. I find it sad that much of the posts here take so much time comparing your work to Nate’s. Seems awfully hostile, which makes me wonder why folks are so defensive. I have been following Nate since his site materialized and I have found next to no criticism of your work (or others) by Nate or by other posters. I guess it could be helpful to compare notes between you and Nate, and I’m sure he’d have helpful hints for you too. I just want you to know that, while I think your site is helpful and interesting, and I’m very glad I stumbled upon it, I do wish there was less noise about who is more accurate or more transparent or more humble or more…. In short, stop making noise.

    One more thing: Nate is not going to be completely transparent since he actually does want to make a living at this. Why give away trade secrets? This is, after all, his job, not a hobby.

Leave a Comment