Princeton Election Consortium

A first draft of electoral history. Since 2004

Why you’re wrong to get excited about “60%”

August 4th, 2014, 2:35pm by Sam Wang


Some people are excited (positively or negatively) about Nate Silver’s column today giving a probability of a GOP takeover at 60%. To cut to the chase: I do not think that number means what you think it does.

Here are three big things to think about.

1) Five coin tosses. At this point, Senate control comes down to as few as five* races: AK, CO, IA, KY, and LA. Think of these races as coin tosses. Then Democrats have to win 3 out of these 5 tosses to retain control. (I’m simplifying matters, but not by much.) These coins are not perfectly fair, and the overall situation is a little unfavorable to Democrats. That is basically the amount of uncertainty expressed in Silver’s probability.

Fundamentally, any probability in the 40-60% range is a numerical way of saying “I don’t know.” (Just to poke at the scar a bit, “I don’t know” is what Silver should have said when he intimated that Brazil would probably beat Germany in the World Cup. We all know how that turned out.)

2) The certainty fallacy. Silver has done something common among paid writers, which is to do what it takes to attract eyeballs. He has rounded a probability that is barely over 50% to make the statement that one side is ahead.

This problem is endemic to journalism. Nuance with probabilities is just lost on most people. But writers are just trying to appeal to their readers. The real headline is “WE DON’T KNOW” but somehow that is not what gets the lead. Really, he’s caved in to all of you!

Basically, whenever you see a probability like that, you should mentally say “plus or minus 20%” just to get the right idea. Of course, if the probability gets above 80% that doesn’t make sense any more. At that point, it’s OK to say that one side is ahead.

3) One of those coins might say “D” on both sides. As I’ve pointed out before, the biggest problem is that we don’t know how Alaska will turn out. The only recent Alaska poll shows Sen. Mark Begich (D) bounding out to a 12% lead over likely nominee Dan Sullivan (R), probably because of a gaffe on Sullivan’s part.

If Begich really is ahead, that translates to Democrats only having to win 2 out of 4. That gives a pretty accurate intuition for where the Senate race actually is as of today.

Make no mistake: at the moment, the battle for Senate control is super-close. If national opinion swings as little as 2 points in either direction, uncertainty about Senate control will evaporate. As I’ve written before, even if that happens, don’t be surprised if both sides end up with 49 to 51 seats each in 2015.

*Just for purposes of building up a clearer intuition about probability, I’ve left out AR, GA, and NC. Of course, here at PEC our Meta-Analysis considers all races at once. But the bottom-line answer is exactly what I’ve said. So for purposes of discussion today, I’ll leave those races out.

Tags: 2014 Election · Senate

34 Comments so far ↓

  • Mark Buckley

    Had a lot more respect for 538 last cycle when the discussion was mostly politics and the search for America’s best burrito was left to the Food Channel where it belongs. I do miss Intrade though for it’s daily ( sometimes hourly ) entertainment for political junkies . Still think following the betting lines give a more accurate picture of the Race than the talking heads who need to ‘ keep it close ‘ to interest viewers and sell advertising . Thanks for all your hard work Sam sure glad you haven’t changed !

  • 538 Refugee

    I’d get more excited if we didn’t already know one side can pretty much kill anything it wants in the senate. Couple that with a president that doesn’t have election pressures for two more years. Our government has become like a hockey game but instead of a puck the ref drops a dead horse labeled “change” and the two sides proceed to whack at it with toothpicks. Like two large companies engaging in ad wars they both win from the added publicity the ‘rivalry’ brings.

    Even when the Democrats had a clear large majority the best they could get was a pretty flawed, but much better than nothing, health care bill. I’m not getting excited about slim majorities for either side.

    • Sam Wang

      Stasis is what to expect no matter who is in charge. The exception is judicial appointments. That’s a fairly big exception!

  • Bill

    It is a long way until November. A lot can happen during that time period to influence how the November election turns out.

    • Sam Wang

      The thing is, how much can really happen? Measured in Senate control probability, a lot — but that’s only because we’re right on the edge. Measured in terms of public opinion, the likely range of movement is smaller.

      Go look at the range of variation in 2014 of generic House preference (2%)*, Obama job approval (4%), and now my Senate meta-margin (3%). Each of these is moving within a narrow range. Do you really think that much will happen in the coming 3 months?

      I agree that it’s interesting to wonder if we will bust out of the current stasis. I think this can be modeled, but I am holding back until I understand it better.

      *If one goes back to last year’s government shutdown, the two-party generic House margin has a 13% range of movement. That leaves open some possibility of large movement.

  • Disco

    Sam has it right. Judicial appointments. That’s the only thing the senate will be capable of doing before 2017.

  • Stuart Levine

    Ok Sam, play “Dear Abby” for a moment. I have a conservative friend who will most certainly try to throw the new 538 number in my face as soon as he picks up on it. So, do I pre-empt his move by sending him a link to the above right now or do I wait and use it as a rebuttal?

    Signed: Chomping at the Bit.

    • Sam Wang

      Sure, fire it off.

      But…you know what 2014 is like? It’s like 2004 Kerry v. Bush. In early August, we have no way of knowing who’s Kerry and who’s Bush. In terms of winning, that is.

  • Stuart Levine

    Oh, BTW, my friend actually has a degree in statistics. He sent me stuff up to a few days before the 2012 election to the effect that neither Silver nor you knew what you were doing.

    • Sam Wang

      The key to understanding what Silver and I do is that it’s not rocket science. All one needs is a method for looking at all the data at once, one that does not add too much extraneous bullshit. Punditry is mostly extraneous bullshit.

      All I am doing today is pointing out that Silver said something mostly defensible as a number (60%), which people are foolishly rounding up to 100%.

  • neil wilson

    I really miss Intrade.
    There is just nothing that can replace the accuracy of Intrade.
    Intrade was far from perfect but it reflected the consensus better than any poll could.

  • Alan Diede

    Dr. Wang,
    Again your insights are highly appreciated.
    Two quick questions about this mid-term election.
    1. Do you detect any trends in the Senate races, both individually and overall?
    2. After Cantor’s debacle in VA, where an important part of the story was disastrous polling, what is your confidence level in the polls you are seeing? I assume that state-wides are far better than congressional district polls, but are minority populations still being under-represented?

  • Mark Buckley

    Yes poll aggregation is more accurate but Intrade is more accurate than just one poll I believe . Current betting line on 2016 : Clinton 5 /4 Rubio 10/1 Christie 10/1 Paul 14/1 . I see Paul as an underlay currently as he’s had a better year than all other Republicans .

    • ArcticStones

      Current betting line on 2016 : Clinton 5 /4 Rubio 10/1 Christie 10/1 Paul 14/1 .

      Clearly I am in the wrong business! Those numbers don’t add up to anything close to 100 %. Intrade must be making an outrageous profit as a betting agent.

    • Sam Wang

      Well, I agree with that. On short time scales, Betting markets perform like a noisy poll average – about as well as if one looked at a HuffPost polling page for three seconds and then estimated an average. On long time scales, markets collect the wisdom of individual bettors. That amounts to conventional wisdom, which is often right.

  • EthanX

    There’s one wild-card in the Senate control discussion that so far, I have seen little about. While he currently caucuses with the Democrats, Maine Senator Angus King’s “loyalties” have often appeared to be up for sale. At the moment, with Democrats in control, he caucuses with them, but with a 50-50 divided Senate, could he throw the whole thing to the GOP? I’m sure they’d try and offer him the moon and stars if he would. The only problem with that is that he would likely face the wrath of Henry Reid, and the Democrats are likely to retake control in 2016. King isn’t up for reelection until 2018, and 2014 will be long distant memory by then.

    Thoughts?

  • Amitabh Lath

    60% +- 20% sounds about right. Silver would get an F if this was a lab report. We do not take kindly to a reported measurement not accompanied by its uncertainty.

    Looking at your previous post comparing polls with results (which showed that results skewed D+2 compared to polls in close races), and keeping in mind that midterms have low turnout and even a modest bump in voting likelihood among a certain demographic can have a serious effect, I would add ~15% systematic uncertainty in quadrature for pollster mis-weighting. (For a total of a 25% uncertainty)

    • 538 Refugee

      At some point I would expect attentions to center around getting people that aren’t likely to go to the polls to vote absentee. That could make up for lack of enthusiasm.

    • Amitabh Lath

      Yes, people like the African American cohort that voted in the MS runoff for Thad Cochran. The articles I’ve read state these voters carried Cochran, but none discuss the technical details of how their vote was mobilized.

      At some point there needs to be a study of GOTV effectiveness in the era where everyone (but me) is connected to social media on their phones.

      Effective education/mobilization helps both sides of course, so Sam is probably right to trust the median of the polls for now. In the long run I expect the pollsters will change their demographic weighting and likely voter screens towards Dems, since apathetic voters tend to skew D.

  • Bob Grundfest

    Whatever happens in 2014, the Democrats will have a much better chance of regaining the Senate in 2016 when the Republicans have to defend a large number of seats. As stated above, a 51-49 GOP Senate for the next two years will mean little except for judges.

  • Pechmerle

    To me, the single most interesting item in Silver’s column is his forecast of an 80-20 probability of victory for McConnell in Kentucky. That is non-congruent with PEC’s estimate, and I thought with what the polls were showing. Have I missed some statistical info that would support Silver on this one? He could just be weighting McConnell’s incumbency very heavily, but one doesn’t know since he gives no discussion for this assessment.

    • Sam Wang

      It probably has to do with campaign finance, or maybe older polls. Those are not crazy reasons. But I try to keep such handicapping separate from the PEC estimate, which is set up to just use the last 3 polls / N weeks in a blind fashion – a minimalist approach. One thing that is true is that over the course of the campaign, Grimes has had difficulty breaking past 47%.

    • Sam Wang

      Any polls-only based estimate should be more like 50-50. As they state, they are using prior assumptions based on pre-campaign conditions, which is what NYT and WaPo do. That’s where they get their data. In particular, I am skeptical of the WaPo approach. I am against such assumptions on the grounds that they are less accurate than fresh polling data.

      For example, other aggregators are calling Mitch McConnell the favorite. In that case it could be prior assumptions. In that case, my gut says they have a point: Grimes’s HuffPost trendline is fairly flat and and she has had trouble getting above 47%.

      Generally, if one looks at long-term trends, incumbent job approval, and other factors, the impression that emerges in the nine races I listed a few days ago is as follows:
      Republican advantage: Georgia, Kentucky, Louisiana, Montana
      Democratic advantage: Alaska, Colorado, Iowa, Michigan, North Carolina

      This particular scenario leads to a 50-50 split. Now, if you call this a prediction I’ll deny it…for now.

  • Amitabh Lath

    Even with a polls-only estimate, it seems if I play around with the time window I can get whatever mean value for the prediction of any given election.

    Of course the error bars grow/shrink as well, but if I’m only interested in the mean value to create a headline, I’m not going to be particularly invested in the proper uncertainties.

    • Sam Wang

      At the end of the 2008-2012 campaigns, I would often increase the time window on each state until I minimize SEM, i.e. I choose the time window where SD/N is smallest, where N>=3. This secret sauce worked rather well. I wouldn’t do it now – conditions are too fluid.

    • Amitabh Lath

      The presidentials tended to be well behaved except for some large first derivatives around specific catastrophes like the first debate, or 47% etc.

      I can imagine playing with the time window didn’t do much.

      The Senate ’14 polling by comparison so jittery it’s hard to argue for or against any specific time window.

  • bks

    Harry Enten at _538_ claims an observer effect among pollsters:
    http://fivethirtyeight.com/features/are-bad-pollsters-copying-good-pollsters/
    –bks

    • Sam Wang

      The Upshot’s estimate does not move much, probably because it is weighted toward polls.

      The Monkey Cage…I guess I should read that forecast. I wonder if there are enough details to nail down why they think that.

  • Gwen Carter

    Dr. Wang is it possible a quiet giant is waking up and this election could see an increase in voter turnout? The reasons why I ask: women’s health care, voters’ rights, immigration, and a few other issues. I feel like this election will see an increase in voter turnout. Just wondering about your thoughts on this.

    • Sam Wang

      No idea. It would probably show up in polls. In 2004 I thought there’d be some “bonus” above what the polls were saying. Boy, was I wrong. So…just watch the aggregate up there in the banner, and in the graphs in the right-hand column.

Leave a Comment