Princeton Election Consortium

A first draft of electoral history. Since 2004

Today on the Brian Lehrer Show: 11:10 am, WNYC 93.9 FM

September 12th, 2014, 8:55am by Sam Wang


This morning I’ll be on with Brian Lehrer to talk about poll math, the suspenseful Senate race, and when to accept a bet. Tune in!
[Update 11:23am: that was fun. However, I just found out that Silver claimed that at PEC we use "arbitrary assumptions." OK, that is an out-and-out falsehood. We use polls only and are completely open-source. I do not think a polls-plus-fundamentals outfit like FiveThirtyEight should be throwing that particular stone! -Sam]

In other news, the Senate Meta-Margin’s holding steady at D+1.0%. The Meta Margin’s a better number to watch than the probabilities because it shows how much the race would have to swing to create a perfect toss-up situation.

D+1.0% reflects the close margins at present in a polls-only view. Look at the right sidebar – the top five races are all within 2%. Click on them for rich poll goodness, courtesy of HuffPost!

[*Update 3:13pm: A few of you have called attention to this. Let's clarify: it's affectionate. Nate's friends at DailyKosEschaton, and elsewhere adopted this to make fun of his critic Dean Chambers. It is explained here at Gawker. I see that I have exited the blogosphere and entered the world of actual media figures, and therefore must stop with the inside-y jokes. Point taken. -Sam]

Tags: 2014 Election · Senate

35 Comments so far ↓

  • Phil Drum

    Sam, I am wondering about the Orman caucus issue – I have heard he will caucus with other Independents (currently with the Dems) and that he will caucus with the majority – what if it is GOP 50 – Dems 49? In that case – which appears the most likely – he gives the majority to whom he chooses – anyone have a sense what he will do then?

  • David vun Kannon

    I just saw that some statistics on early voting (party affiliation, number of ballots) is already becoming available. Sam, will data such as this be useful inputs to a model? Even more than polls, it is information on actual voting. Thanks for your thoughts, you do great work.

    • Sam Wang

      Thanks. I believe pollsters ask if people have voted yet.

      Information on early voting might be useful in estimating turnout, useful for superclose races like Arkansas Senate, Cotton v. Pryor. There are others who are good at that kind of estimation.

  • Forrest Collman

    Sam,

    My understanding from reading Nate’s qualitative descriptions of his model is that most or all of the non-poll based assumptions in his model go to zero as we near election day. So we should expect your models to disagree less and less as time goes on. When election day comes around, the remaining pieces that are left to disagree about is the precise manner in which you aggregate polling data.. which is pretty straightforward for your model, and slightly obscure for nate’s?

    We can obviously use the outcome of the election to weigh the relative strength and differences then.. but what about at this point or earlier in time.

    Are there really enough historical predictions from both of you to figure out who has been overfitting and adding in extra variance and who has been ignoring important historical trends?

    • Sam Wang

      That is correct. He says his assumptions go to zero. However, he still “corrects” polls. I generally expect his prediction to move toward mine. Mine will move hardly at all until 10/1, after which the random-walk from the day’s snapshot will contribute more and more.

      I am not overfitting at all because there is no fitting. Fitting is treif. No added variance either that I can identify.

      As for ignoring historical trends…that is a possibility, mainly because there is virtually no relevant data. I’d say only Senate 2006 and 2010 individual-race data are relevant, for predicting Sept-to-Oct movement. For example, there might be a 0-0.5% bias toward the GOP in the coming weeks. Or not.

      As for historical predictions, I’d say his wrong calls (ND, MT) in 2012 are Exhibit A. Getting two races wrong this year would matter.

    • 538 Refugee

      “Getting two races wrong this year would matter.” I almost wrote this same thing last night after reading some comments at 538. Someone said it didn’t matter that Nate called two races wrong because the overall result was correct.

      Welcome to the Princeton Election Collective. You will be assimilated.

  • SFBay

    It sounds like Silver’s starting to sweat.

    Sam, the comments are rude and intended to get a rise out of you. It’s like the saying, if you have facts on your side, use facts, if you don’t have facts, throw dog poop.

    • Sam Wang

      Good advice, thanks.

      It is not fun to see Nate go after POLITICO and so on. Gotta stay substantive. There is plenty to say – and it elevates the conversation.

    • ArcticStones

      I think you’re spot on, Sam.
      My apologies for suggesting that you entertain the notion of a bet. That was probably a seriously bad idea with regards to Nate.

  • Kenny

    Here’s the exact quote by Nate Silver (I transcribed):
    “That model that he [Sam Wang] uses is based on a lot of very arbitrary assumptions. Umm, He just says, ‘Oh well, let’s try this’ and it’s different than us. Everything we do is researched empirically.”

    • J. R. Mole

      Yeah, that’s a pretty unhelpful comment from Silver.

    • 538 Refugee

      I wonder when testing against historical data for president if Nate uses this: http://en.wikipedia.org/wiki/Redskins_Rule While this may be tongue and cheek it illustrates the point. If you look for correlations you will find them. While some things SEEM to be more logically related than football and elections this ‘rule’ probably holds a better track record than presidential approval ratings and individual senate races.

    • ArcticStones

      Nate Silver: “Everything we do is researched empirically.”

      The problem, of course, is that while Dr Sam Wang’s algorithm is open source, with no hidden assumptions, the exact mix of ingredients in Mr Silver’s “Secret Sauce” is a closely guarded secret.

      Translation: there is absolutely no way for anyone else to verify how empirical Nate Silver’s hidden adjustments are, since he is not willing to share them and subject them to peer review.
      .

  • J. R. Mole

    +Raj: Silver’s take on internet polls is more nuanced than that, but Sam’s right that a fundamental difference between the two sites is in “poll-sniffing” or not (great phrase). As I’ve said, it looks like Sam has the better argument on that one.

    +Sam: How much time do you want to spend dissecting the meaning of “arbitrary”? It doesn’t seem a productive exercise. Stating one’s assumptions, and kudos for that, doesn’t make them any more or less arbitrary.

    At the end of the day, one projection or the other will fit the actual results better, and then we can try to analyze why that might be.

    It’s interesting that 538′s projection has lately shifted significantly toward PEC’s, which has remained much steadier. That would seem to directly undermine the fundamentals-based assumption that shifts were more likely to break toward the GOP in states like NC — which did exactly the opposite.

    Maybe that was a bad assumption. Maybe 538′s model is inherently more volatile and we’re just seeing a random fluctuation that will reverse itself.

    • Sam Wang

      Regarding the dissection of “arbitrary” – I don’t want to dwell on it, which is why we’re talking about it down here. I’m trying to avoid the public sphere. However, the implication that we are throwing random things in (see Kenny’s transcription) is maddening.

      I totally agree that this is all about to get sorted out by new data!

  • Tony Roberts

    Good Professor, just to clarify, your model at present predicts Democrats plus Independents to reach fifty WITHOUT knowing with what party the Kansas independent would caucus with if he prevails, correct?

  • RAJ

    What do you think about internet polls like YouGov. In 2012 Nate Silver dismissed internet polls as not represntative because they didn’t use random contacts and weighed them lower than random contact polls. What would your result be if you omitted internet polls like YouGov, Reuters/Ipsos and others? I’ll be watching the Scottish Independance vote since Yougov was the only pollster predicting a Yes vote win in a poll a week ago.

  • Brian

    Could it be considered an ‘assumption’ that you assume all independents (including Orman) would caucus with the Democrats?

    • shma

      I think his choice of a cutoff for polls (7 days prior to the median date of the last poll), his choice of probability for a black swan event, and any temporary fixes for the near total lack of Orman-Roberts polling are probably what Silver was referring to, in that these seem to be a guess/chosen because they worked well last time, rather than motivated from some kind of model.

      Orman has been pretty clear about who he would caucus with, and the output clearly indicates what probability it assigns to an Orman-controlled senate (27%)

    • Sam Wang

      I call bullshit on your statement. Every model has to make assumptions of that sort, every single one. Except they don’t spell them out. You should thank me for the total transparency.

      I furthermore think he was bluffing, and has no idea what we do here. It is offensive.

    • 538 Refugee

      The caucus situation has been discussed previously. Orman isn’t considered a lock but the democrat dropping out against a highly unpopular republican when the polls were actually tightening? That does at least have the appearance of collusion. If republicans get to 50 seats he is king maker.

    • Sam Wang

      No, because the calculation is for Democrats+Independents to reach 50 or more votes. That is why the Orman vote is indicated in green in the right sidebar.

    • Gregory Primosch

      Wait, so Silver ISN’T making the assumption that Orman will caucus with Democrats?

    • Liang Q

      @Gregory,

      I’m under the impression that Mr. Silver assumes there is a 75% chance Orman would caucus with the Democrats should he be elected, as his ideological stances are similar to that of a moderate Democrat.

    • shma

      “I call bullshit on your statement. Every model has to make assumptions of that sort, every single one.”

      Yes, but Silver claimed in the interview he motivates all his choices by testing against historical data. He didn’t say how, exactly, but that’s his claim.

      Anyway, I’m not taking saying he’s making a serious argument. I’m just guessing at what he could have meant when he made that statement, and pointing out it probably had nothing to do with the assumption that Orman would caucus with the Democrats.

      I do, however, think it would be beneficial if you re-ran previous elections using different poll cutoff dates and/or minimum number of polls in order to see the effect it has on the snapshot and long term predictions over the course of the election season. And, of course, in order to optimize your choices.

  • 538 Refugee

    Well done. Host even got your last name right 50% of the time. Interesting that anyone could accuse your model of bias but you have a bullet proof defense on that one. Did Silver initially use that term or did the host introduce that into the mix? I’d be really surprised if it was Nate.

    • Sam Wang

      Silver himself used the words “arbitrary assumptions.” oh, me mad now!!! hrrmrmrmrpr.

      I am OK with the possibility that I might be wrong. But it will not be because I added a bunch of stuff!

    • Amitabh Lath

      Well, maybe things you are *not* doing could be construed as “arbitrary assumptions”. You assume that pollster house effects need not be corrected for. You assume that Presidential popularity, fundraising, blah blah blah have no effect. You assume that medians work better than means. You believe in the Central Limit theorem.

    • Gregory Primosch

      >> You believe in the Central Limit theorem.

      Nice.