Exceptionally Low Turnout Can Account For Polling Errors
At The American Prospect: Tweet In the home stretch, I wrote that midterm polling is far less accurate than in Presidential years. Today, in The...
Is Nate Silver a little too excited about his model?
“I would like to place a large wager against that guy…” says @NateSilver538 of @SamWangPhD‘s projection, which heavily favors Dems.
— Brian Lehrer Show (@BrianLehrer) September 8, 2014
I guess when you’re the King of the Nerds, you have to be willing to engage in a little trash talk. I think this is a good time to administer a lesson in probability* – and also question who is doing the “heavy favoring” around here. [*see postscript – Sam]
The PEC Election Day prediction indicates a 70% probability of Democratic+Independent control. That is based on polls alone, plus the assumption that September-October will act like June-August. FiveThirtyEight’s probability is 64% favoring the Republicans, based on a model with polls plus a substantial dose of special sauce (a.k.a. fundamentals).
I guess it is a good time to make a bet, if you are the kind of guy who makes a lot of bets. Personally, I am conservative about these things. It seems best to hold one’s tongue until an outcome gets into 90% territory. In my analysis, the pure polling data suggest that Democrats are favored. But to me, a wager at this time sounds like taking turns playing Russian roulette – for both players.
However, I do see a fairly juicy bet in the FiveThirtyEight calculation.
I have to say, this special sauce is messy stuff. Really, the GOP has an 25% chance (3-1 odds) of getting 54 or more seats? I’d put it at more like 5%. Even 53 GOP seats is a fairly outside outcome. If a betting person were offered the chance to put up $3000 against Nate Silver’s $1000 on that outcome…that would be taking his money.
Joking aside, there are two serious points to be made here. First, nobody should be getting excited about any probability that is in the 20-80% range. That includes Nate Silver, who knows better. Second, the addition of “fundamentals” and other factors adds considerable uncertainty to the projection.
***
In other news, the Meta-Margin has dropped to D+0.9%, mainly because of a YouGov/NYT poll in Alaska that puts Dan Sullivan (R) ahead. Alaska is a problem for any poll-based modeler. However, the Meta-Margin is right at the middle of the range predicted by the polls-only model. As always, we must wait for more data.
Topics:
Stuart Rothenberg says R+7 in the Senate:
http://blogs.rollcall.com/rothenblog/elections-2014-stu-rothenberg-senate-republican-gains/
–bks
I have just challenged Stu Rothenberg to place his money where his mouth is. A gain of 7 Senate seats for the GOP? I don’t think so.
.
I’d gladly take that bet against Rothenberg, as would the vast majority of statistics hobbyists and professionals. His prediction is steadily outlying, with little to no measured evidence supporting his projection.
His prediction is almost entirely based on subjective Bayesian priors (kept conveniently opaque in terms of weighting). There is no data, current or historical, supporting his conclusion. Anywhere.
And why do I make this judgment about his prediction being based upon subjective Bayesian priors? Because none of the measured data supports the picture he’s painted. No historical trends, no current measurements. This inherently leaves nothing *but* subjective Bayesian priors as a basis for his model.
In other words, where Silver has decided to slather his “Special Sauce” fundamentals onto his model, Rothenberg is simply serving up “Special Sauce” ala carte and calling it “data.”
Therefore, his prediction is, in my opinion, business-minded clickbait, designed to bring him back into the polling-pundit fold after years of being ignored in favor of 538 and the PEC. I just don’t find it plausible that a man whose life was spent on gleaning information from measured data would suddenly decide that subjective Bayesian priors (AKA, blind assumptions)
Electoral history has shown that this approach to Bayesian priors creates a horribly inaccurate model of prediction. He knows better than this.
Stay away from this circus, Sam! Anyone with an ounce of statistical common sense knows that the error bars on all these predictions are so large that at this point they essentially all agree with each other within uncertainty.
The fundamentalists (good name for them, no?) would like nothing better than to drag you and the PEC down into the muck of the political talk shows where they shout about unemployment and ISIS and approval ratings.
If you can, put the code up and point people to it. Let them nitpick the MATLAB.
PS: My advisor got caught up in the talk-show wars during the “Star Wars” debates. He battled Buckley, went on Donahue, etc. He really did not enjoy it, and it took a real toll on him. In the end Star Wars up and died of natural causes.
The truth, no matter what excuses Nate Silver makes, is:
“Secret sauce” = Unscientific punditry
.
AL. You mean let the numbers do the talking? Strange concept, but it just may work. We’ll know in a couple of months I guess.
The way I see it, the problem with fundamentals is you are scouring the past looking for connections. You can find them just about anywhere though. http://en.wikipedia.org/wiki/Redskins_Rule While some may seem related they only work until they don’t.
BTW, Election Day is exactly eight weeks from today. –bks
I understand your hesitation to “take on” 538 simply because of how unnecessary it is. But at the same time, like many others here, I started following election predictions with 538. When Nate got a little too big time for me, I found you and have enjoyed your predictions and your prose greatly. There is a large part of me that wants you to take the bet, especially since I am confident you would win.
“There is a large part of me that wants you to take the bet, especially since I am confident you would win.”
I agree. And not because there is money to be made – that’s irrelevant. It would be a great way for the Princeton Election Consortium to gain added attention. As a matter of fact, we could even establish a pool, so that the bet is substantial.
And the winnings could even go to a charity of choice, for instance Médecins Sans Frontières.
.
Sorry, no.
Look at Silver’s curve for Senate seats, and then look at the PEC curve. Silver has a peak at R52 with a FWHM of about 8. PEC has a peak at D50 with a FWHM of about 4.
Of course since they use the same data, they are not really independent measurements, but claiming that the (slight) difference in the mean has any significance at all when the widths are so large is just silly.
Anyone claiming that PEC “heavily favors Dems”
does not understand how to read a probability distribution function.
Sam’s predicting that the probability of a Democrat win is not sufficiently different from even odds to be worth the downside risk for a single event. Based on both of their analyses, there’s a 1/3 chance that whoever’s actually right would still lose the bet.
I’ve enjoyed this site since finding it and it is now my go to for what is going to happen election wise. Your track record is good for me. I also started with 538, but saw the light after reading your blog.
Seems to me that Nate Silver is calling into question “the assumption that September-October will act like June-August”. He may be right or wrong, but he gives clear and detailed reasons for that, builds a model around those reasons and makes a testable prediction based on that model.
What’s the problem?
It’s not at all clear to me whether the “fox” or the “hedgehog” approach is more appropriate for forecasting elections. Unlike the 2012 election at this stage, there’s currently some daylight between 538’s and PEC due to their different methodologies. That means we have an opportunity to learn something here. Not much, maybe, but something.
Most likely, the two will converge on election eve, if only because 538 weights polls more heavily closer to the election. It will be interesting to see whether the polls remain stable and 538 shifts, the polls swing toward the GOP and PEC shifts, or some of both. Then we can argue about what, if anything, it means.
I have no idea what that fox-hedgehog stuff means. I feel like he has made a hash of the concept. But sure, we’ll end up as fedgehoxes.
fedgehox. That your new mascot?
PEC and 538 seem to diverge in two main places:
1) 538 tries to account for the varying quality and biases of pollsters by explicitly accounting for “house effects”, “pollster quality” and such. PEC argues that the best way to account for variations in measurement is to aggregate measurements without trying to model anything specific about them. When trying to answer the question “what do the polls predict?”, PEC and 538 appear to give essentially the same (correct) results on election eve. PEC wins on simplicity or put another way, 538 fails to make a convincing case that their approach produces better results and so justifies the added complexity.
2) PEC assumes that the polls going forward will perform a random walk. 538 explicitly says this is incorrect. Arkansas or North Carolina, say, are more likely to shift toward the Republican side than the Democratic side. 538 gives a plausible reason for this: Those states have tended to vote more heavily Republican (I believe the cite other factors as well).
2a) Along the same lines, 538 also asserts that races in different states are not independent trials. I believe that PEC at least implicitly assumes this (could you please confirm/disconfirm?).
As I understand, PEC assumes an unbiased random walk and (I believe) independent trials on the basis that there is no compelling reason to assume anything more complex. 538 asserts that there is a good empirical basis to assume otherwise.
It’s this second factor that’s causing the two forecasts to diverge at the moment.
This is not “trash-talking”, “special sauce”, “needing media attention” or even “getting excited”. It is disagreeing with your assumptions.
As far as this difference over assumptions is concerned, today’s post is essentially content-free. It promises some sort of “lesson in probability”, then devolves into hypothetical bet-making. The only valid point is that the error bars are wide, which point was never in dispute. Noted.
538 sees a larger chance of 54 Republican Senate seats than PEC. Pointing that out is not imparting a lesson. It’s expressing an opinion about modeling.
So far, there hasn’t been much to pick between these two approaches to forecasting, because there hasn’t been much disagreement between them (at least since I’ve been following this). In the absence of anything more persuasive, simplicity, that is, PEC, wins.
Now that we’re seeing some divergence, we may have an opportunity to learn something.
Please, Sam, don’t spoil this by taking pot-shots.
It is challenging to maintain my cool when I am called biased, and mocked with the offer of a wager to boot. However, thank you for the feedback!
As far as content-free goes…not true! You already understand the point about variability…but many readers don’t.
Anyway, the next post will have enough content for you to sink your teeth into…
There was daylight between the models in 2012, also, particularly in the North Dakota and Montana Senate races. The 538 model predicted Republican victories with 92.5% and 65.6% likelihoods, respectively. Wang (correctly) predicted Democratic victories in both.
North Dakota, in particular, is why I look a little askance at 538 now.
Trash-talking from the guy who predicted the Democrats to regain the House in 2012? That’s rich.
And I re-examined my assumptions, bigtime. What do you do when you are wrong?
Sam, can you expand on your statement “nobody should be getting excited about any probability that is in the 20-80% range.”
Are you referring to this specific topic or all probabilities in general?
For example; a 70% probability sounds like a pretty likely outcome… so I guess my question is: Are all 70% probabilities created equal? And if not, why not?
It’s fairly likely…but 30% of the time it will be wrong.
You could say that 70% probabilities aren’t created equal. It depends on what they’re being used for. If I’m gambling over repeated trials and with a one-to-one payout, then I’d be pretty damn happy about 70% chance to win. But there’s only one 2014 midterm election.
I can remember when 538 (the original) went after pollsters with suspect data or assumptions. Seems to me that Silver is now the one guilty of suspect assumptions. The current version of 538 has nothing to do with accurate data or assumptions. Not sure how their internet traffic is these days, but after being a big follower I have given up on the site. Once Nate started with this bravado I knew he had truly lost his way.
Nate and his fellow writers have been picking a lot of fights lately. Nate himself went after Matthew Yglesias over at Vox and other writers there took issue with the former home of most of Vox’s writers over at the Washington Post’s Wonkblog. Nate even went after your colleague Paul Krugman. He and the rest of FiveThirtyEight seem to be going after all the rest of the data-based journalists in the media, particularly the outlets liberals read. Based on that history, Sam is in good company.
If you personally, not as part of the PEC, had to give fair odds for the Republicans taking the Senate, would you give any weight to the other models, or would you rely solely on the PEC model’s pick?
I would only give weight to odds that did not involve fundamentals: HuffPost, NYT-without-fundamentals, and Drew Linzer’s DailyKos Poll Explorer. However, I would also wait until 5 weeks before the election, when the snapshot should start to become directly predictive.
Have to ask this question. If we are asserting that ‘fundamentals-based’ aggregate polling is flawed and impure, what about the practice of weighting the polls themselves? Individual polls are nearly always weighted for any number of variables. Aggregate polls are based on these weighted polls, so why the fuss over the weighting, raking, etc. of polls in the aggregate? Have to ask.
I thought polls were weighted when the responsible grout doesn’t match up to the known percentage of men, women, age, and political affiliation. That sounds reasonable. The problem arises when pollsters use incorrect weighting to favor an outcome in a specific direction.
Individual polls are weighted demographically, given the likely voter model that the individual pollster uses. I don’t think fundamentals as understood here (say, Obama approval rating) play a role.
“special sauce” lol
If everything aligns perfectly for the GOP, they’ll get 52 seats. But that outcome requires them to retain both KS and GA, and to pick up AK, AR, LA, MT, NC, SD, and WV. The GOP will indeed do some of that, but their “running the table” in that way, just getting to 52, strikes me as extremely unlikely.
Any scenario that gives the GOP 53 seats or more strikes me as sheer fantasyland, because it requires the GOP to “run the table” as above on all the really tight races AND start picking up fairly safe Democratic seats–CO (53)? IA (54)? NH (55)? And yet Silver assigns these outcomes, collectively, something like a 39 percent possibility?
My common sense doesn’t buy it.
I’ve been a big fan of Sam and Nate for the past several election cycles; I think both have made huge contributions to our understanding of election forecasting. Perhaps the biggest being the undercutting of political pundits who either don’t understand data or lie for political reasons or, more likely, both. Think George Will on the eve of the last election predicting a landslide for Romney. What an ass, but at least now, because of Sam, Nate, Drew Linser and several other data analysts, it is more difficult for Will and his ilk to get away with their lies. I have always preferred Sam to the other modelers for the reasons many have stated — transparency and simplicity. Usually, the best models are the simplest that work well. Then we also learn a lot if and when a simple model breaks down. But right now I don’t see much difference between Sam’s and Nate’s predictions. It is currently too close to call who will control the Senate, unlike the last presidential race which was clear cut. Each of the last few elections, there have been slight differences between Sam’s and Nate’s prediction and Sam has consistently done better. I am definitely hopeful the Democrats have a reasonable chance to retain control based on both models and especially since Sam’s model suggest the Dems are a slight favorite. Unfortunately, they are not heavy favorites.
Not to digress to re-arguing the past, but almost no one predicted a Romney landslide – including George Will. He incorrectly predicted MN and Ohio would fall into the GOP column, thinking referenda in those states would result in an over representation of Evangelicals. The word ‘landslide’ has been assigned by Dems and media types in the aftermath. To the present, what I have always liked about the PEC model is that it presumes market corrections for outliers and the vagueries of any individual polls, assuming a robust data field.
George Will and others predicted a landslide on the eve of the election as reported by Republican types prior to the election.
http://www.theblaze.com/stories/2012/11/04/romney-landslide-here-are-the-biggest-names-predicting-it-how-it-will-happen/
Thank you, Djinn, for documenting Will’s prediction of a 321-217 landslide for Romney. I remember very clearly seeing him make this prediction on TV the day before the election, but I didn’t have any documentation other than my memory.
I think people forget that 538 missed the 1st Obama midterm (2010) by a decent amount! I don’t remember what the PEC prediction was but it was quite a shock when I saw it.
PEC didn’t do all that well with the House prediction in 2010 either, not up to the usual presidential-election standards; possibly the polling in House races is just less reliable. But the real difference between the two was that 538 had unnaturally big error bars right up to the end, so in a sense Silver’s prediction was less falsified but only because it was a far less powerful prediction in the first place.
People mostly care about who will get the majority, which was easy that year. Still, detailed House prediction seems better left to Charlie Cook, Mark Blumenthal, Larry Sabato, and David Wasserman.
An issue with Senate races is that polling seems to be of intermediate accuracy, and variable by year. I confess to casting a fishy look at my “systematic=0.7%” parameter.
I feel like that is just reflective of Silver’s high risk tolerance rather than anything else. He’s a former “pro”(?) poker player, so he might just be very aggressive on betting – more comfortable with casually putting money on the line for relatively small odds differences. And his reputation is already on the line – so maybe he feels he’s the “big blind”, so to speak.
Richard, I still see George Will on the Sunday shows plying his delusional wares. Conservatives don’t seem to pay the price for their stupidity…ever. I’m still waiting for just one, say Karl Rove, to admit they were wrong in their hyperbolic predictions about that big Romney win.
Nate / 538 have an issue. ESPN paid a lot to buy them, their launch was a bit inauspicious. They need headlines to attract attention. ‘Nuff said.
(1) I think this is a friendly rivalry.
(2) This is nothing but positive for PEC (when it comes to blogs, all press is good press).
(3) Both camps are doing their level best to be rational and unbiased, which is to their credit. This is in contrast to other outfits, which I’m convinced have their thumb on the scale.
(4) At most, I’d bet a six-pack.
Want to make a bet _and_ a point about uncertainty? Offer the 35-1 odds he’s showing that the Dems get 54 or more seats.
That way even if you lose, you win.
When you continue to predict your hopes and dreams will come true rather than root yourself in reality, you will continue to be incorrect.
Nate happens to be rooted in reality.
Republicans will have 51 seats.
I estimate that FiveThirtyEight adds assumptions that effectively bias their calculation about 1.5 percentage points toward the GOP. Is that rooted in reality? I wonder what your data-based reason is for saying that.
Since their prediction is so broad, even if Democrats+Independents reach 50 seats, that would be ambiguous in terms of testing models. A result outside the range of 48-52 D+I seats would be a clear indication that the PEC calculation did not account for some hidden offset or hidden dynamics (fundamentals?).
I wonder if the reason PEC and 538 differ quite a bit is because of likely voter polls vs registered voter polls? That’s part of Nate’s “special sauce,” trying to align the two so it’s an apples-to-apples comparison. I think he said it was something like a +2% difference favoring Republicans, when registered voter is translated to likely voter (due to higher turnout).
Probably not. We use likely-voter polls whenever they are available. Go browse our data sources over at HuffPollster.
Well, as of today, a lot of the “special sauce” peddlers are lowering the probabilities of Republicans taking the Senate.
I have been suspicious of 538 since they hired a climate denier to report on climate issues. It just has made me feel that Nate has had to sell out, not sure to who. But why would a math head hire a science denier. Something makes me cast a “fishy eye” his direction, Going on Jon Stewart to apologize and try to get the liberal audience to believe him just didn’t fly. So the “special sauce” is a little tainted with conservative money. Just sayin.