Exceptionally Low Turnout Can Account For Polling Errors
At The American Prospect: Tweet In the home stretch, I wrote that midterm polling is far less accurate than in Presidential years. Today, in The...
Senate Conditions Are Back To September 3
As of today, conditions in the battle for Senate control are just about back to where they were on the day after the shake-up in the Kansas Senate race. Using polls alone in a 2-3 week window (see right sidebar), current medians show the following key margins: Alaska D+5%, Colorado D+2%, Iowa D+0.5%, North Carolina D+4%, and Kansas I+5.5%.
In an election based on today’s polled sample, the most likely outcome is 51 votes for Democrats and Independents.
[Update: see comments. At the moment, significant drivers of the difference between PEC and other sites appear to be (1) we’re using all polls, including partisan ones, which changes Alaska; and (2) we’re using Kansas two-candidate matchups and don’t have fundamentals to drag those polls in the GOP direction.]
The first four races are critical for either side to win control, and among the closest of races this year. Money and time would be very well spent there, for both Democrats and Republicans.
There seems to be a fuss over today’s snapshot probability. It’s only there for rookie readers. The far more interesting and informative quantity is the Senate Meta-Margin, shown above. The Meta-Margin is defined as the amount of swing needed to create a perfect tie for Senate control.
In this calculation, Democrats+Independents=50 is scored as Democratic control because Democratic Vice-President Joe Biden, who presides over the Senate, would be able to cast a tiebreaking vote. If you do not like that assumption, subtract 0.83% from the Meta-Margin to assign Orman to the Republican caucus, or subtract 0.41% to split the difference.
The core assumptions of the Princeton Election Consortium calculations are: (1) to take a snapshot, accept all polls; and (2) the future will resemble the range of snapshots taken since June 1 (after accounting for the Kansas shake-up). Today’s change is an example of a swing toward the upper end of the expected range, which is between R+0.2% and D+2.5%, peak-to-peak.
Topics:
Nate Silver is now categorizing PEC’s results as an “outlier” (see http://fivethirtyeight.com/datalab/senate-update-democrats-add-by-subtraction-in-kansas/ ). Assuming his numbers are correct, I guess that’s true….
Indeed we’re an outlier. Is it that we accept everything from HuffPost? Genuinely puzzled here. Froggy, any thoughts?
I find it strange that Silver, who once proudly wore the mantle of an outlier, now attacks other projections not on the merits if their system but on their status as an outlier. Haven’t we established by now that most election projections have little value?
Sam (can I call you Sam?),
Can you explain why PEC is an outlier though if some of those other sites like Daily Kos and Huff Po are also ‘polls only’ — but they show Republicans with the advantage?
It seems odd that there is such a large difference… with PEC showing 70% chance for the Dems and DK and HP giving the GOP 54-56%.
To be really frank, since I have documented everything so thoroughly, I do not feel compelled to answer every pointed/abusive remark made over there. So…maybe you can pore over the HuffPost data feed and report back? It’s all out on the table.
Alaska’s result at PEC is certainly being driven by Democrat-sponsored polls that PEC gives full weight to, that some others discount or ignore. Other than that I can’t think of any other state where your additional data is driving things toward the Democrats.
One thing that always strikes me about PEC is how many of the snapshot results are nearly certain, while other models show many more races as competitive. For example, today’s snapshot shows only three races (IA, LA, and CO) that are not at least 95% for the leader. By contrast, Huffpost Pollster’s polls-only model shows nine races as less than 75% certain.
Some of the uncertainty in some places seems totally overblown — 538 has an 11% chance that the Republicans take Minnesota, and an 11% chance that the Democrats take South Dakota. I don’t think that it’s additional data that’s driving you to an outlier position, just that your model doesn’t build in as much uncertainty as others. But then what do I know — I’m an amphibian.
I thought there was enough sigma_systematic built into the snapshot calculation. I will look again. I view Alaska as the biggest problem. One issue is how tail-y to make the win-probability distribution.
There is also the two-candidate Kansas data, which other sites seem to be discounting by covering it in sauce. Come to think of it, this and Alaska probably account for everything.
Sam,
I guess that’s fair (because you have documented), but to be honest I wasn’t asking specifically due to the “outlier” remak from 538, but as a more general question. I can research it on my own, I just thought there might be a “simple” answer.
I trust Froggy. Protuberant eyes see a lot.
Re Froggy’s comment: I think PEC’s snapshot data are not directly comparable to others’ win percentages, because the others’ are for the actual election date, while PEC’s snapshots are for a hypothetical election today. I think you should expect PEC’s snapshots to be more certain. Also, since no one else calculates a meta-margin, the only apples-to-apples comparison is the “probability of D+I control”. (Even that is complicated by the Orman factor.)
I think you also have to realize that 538 and PEC are measuring slightly different things. PEC is measuring D+I and 538 is measuring Democrats control the Senate. 538 currently says the Kansas “I” is 75% likely to caucus with the Democrats in the event of a win and the Republicans holding 50 seats. I’m not sure how much difference that makes but Kansas is somewhat pivotal.
@Art Brown: “…PEC’s snapshot data are not directly comparable to others’ win percentages, because the others’ are for the actual election date, while PEC’s snapshots are for a hypothetical election today.”
I think you’re on to something.
Other sites are assuming that Iowa, Kansas, or NC will break in the GOP favor. That’s basically placing a finger on the scale. The polls still average slight Dem/Indy leads in each of those states.
The points from Art Brown and Joseph are well taken. Nate Silver used to provide numbers comparable to PEC’s snapshot (the now cast, I think it was called), and it would be nice if he still did, because then we would have a better basis for understanding the differences between the two approaches. (I took the percentages from Pollster’s model as comparable to PEC’s snapshot, but I could have been wrong about that.)
Sam, is there any way to produce for the individual races something comparable to the election-day forecasts 538 and others are putting out? You step the present total results forward with random walks and such; if the resulting election day prediction could be used to produce single-race probabilities (or they could be produced in some other way), it would make for an interesting comparison.
Even recognizing that 538’s predictions are election-day forecasts, does anyone really think that the Democrats have an 11% chance of taking SD, or that the Republicans have a similar chance of taking MN? Short of a scandal or huge gaffe involving a frontrunner, some catastrophic Obama scandal resulting in his resignation, or another Republican government shutdown, I don’t see it happening. My guess is about a 2% chance of this sort of thing occurring.
Wouldn’t be too hard. See the Basic post. I’ll have that soon. Maybe not daily, they wouldn’t move that much.
In fairness, the “outlier” label was not really used as an attack in the above linked article. He lists 6 different projections, including his own, and five of them favor the Republicans (though all only slightly). That the PEC prediction is the outlier of that group seems more of a statement of fact than any kind of attack. What that says about the value of the prediction is, of course, another matter entirely.
But he also tweeted, “A “polls only” model with a Dems at a 93% chance of winning the Senate today is [censored]. No way reasonable assumptions get you there.”
Sam, could PEC be an outlier because you do medians and the Quinnipiac polls in Iowa (R+6) and Colorado (R+8) are really pulling the mean based calculations away from the median?
I was looking at that. The HuffPost two-candidate margin and win probability are noticeably shifted. I think you are right that it’s the Quinnipiac data. I would have thought that house-effects calculations could address that by estimating a house effect using this year’s polling data to inform the estimate. I find it strange that anyone would let their calculation be pulled so far by a single data point.
As for the Tweet…I am now leaning toward not hearing about these things! Thanks, though.
Actually, PEC is not the only site favoring D+I in an election held today. Andrew Tanenbaum’s electoral-vote.com doesn’t give Senate control probabilities, but the front page had D+I having >= 50 seats for 8 out of the last 10 days (roughly equivalent to 80% chance of control, if you wish).
I have no idea why I put parentheses around that url:
http://www.therepublic.com/view/story/4cb54bf5287c40d98eeeb919eac76115/US–Senate-Kansas
Beat me to it, Joseph!
Sam, you can now adjust your model again.
No adjustment required. It started using two-candidate margins the day that Chad Taylor withdrew.
I’m intrigued by Nate’s use of the term “reasonable assumptions.” Are we now injecting our opinions or momentum or conventional wisdom into the debate?
Either you use the dry data or you spread the sauce.
First post by a (very) long term lurker. I think what Nate is doing is make his data fit his hypothesis rather than the other way around.
From the article in question: “For what it’s worth (not a lot) my subjective feeling is that the race is still more like a true tossup…”
So yes, Mr. Silver is now “injecting” his opinions into the debate. Which actually makes me sad. 538 was a lifeline for me when it first started up.
Another first post by a lurker, this one a former reader of 538 & the upshot– can’t take those fundamentals skewing the prediction.
The thought does cross my mind that Silver’s move to ESPN may not be working out as well as he would like, and the amped rhetoric coming from him lately will presumably have a beneficial side effect of driving page views to his site because there’s nothing like a high profile debate to get people interested in a subject.
This however raises the question, what happens in November? Despite Prof. Wang’s protests to the contrary, I get a sense that a lot of people are translating 60% Rep. probability as 60% of the vote goes R. If the Rs win, Wang’s model will be viewed as a “loser” (e.g., see headlines for Scottish vote about how polls failed but bettors didn’t), but that is presumably not too much skin off his back as this site is a hobby. It’s more dangerous for Silver if the D+Is do indeed take 51-52 seats, because that site presumably is a big chunk of his livelihood. You’d think that alone would make him tone down his rhetoric, but maybe he figures the benefits outweigh potential detriments in that sometimes there is no penalty for being wrong. I’ll quit because this is getting awfully “meta,” but we should learn some interesting prediction theory regardless of what happens: Trust polls or trust your gut?
For me, I look back to 2012 when O won by 4%, & barely any pollsters got that correct, most underestimated O’s share of the vote (the “no landline generation” effect?).
http://elections.huffingtonpost.com/pollster/2012-general-election-romney-vs-obama
That suggests if anything, most polls have a bias in favor of Rs. Few mention this but people do seem to bring up the historical notion that young people don’t tend to turn out for mid-terms. Time will tell which view is correct (if either).
I agree with Andrew that 538 is underperforming expectations. Why is it covering topics like “America’s best burrito,” which you can’t answer with data anyway? It seems like a lot of their articles are contrarian just to be contrarian. And of course there was the whole farce with the climate denier. I can go days without looking at the site now; in 2012 I looked at it multiple times a day.
Good lord you people are obsessed with Nate. Let’s all pray he never marries Miley Cyrus, I think the entire internet would collapse on itself.
Who is Miley Cyrus?
What does the Good Lord have to do with anything?
And how can a Flat Internet collapse on itself??
Surely no more than the Flat Earth can!?
I’ve become a devotee of your polls this election cycle. Frankly, I’m a neophyte on all this data analysis — nonetheless, I’m really learning more this election cycle than ever before. After doing of reading on all this, I particularly intrigued by PEC’s forecasts. To be sure, I’m praying that you’re right on these projections (I’m a Dem). As I keep up on the news/polls of the critical Senate races daily, I find your analysis both insightful and extremely helpful. Thanks for the service you are providing for people like myself who are trying hard to be a responsible, well-informed voter. I love following the theatrics of the political arena. A few of these races are a bit wild to be sure. Having your analytical take on all of it keeps me grounded!
I’ve mentioned this in another post and it didn’t pick up any replies — a nice way to measure how the different methods of poll aggregation stack up against each other would be to take the day-before-election predictions of each site (which are different methods of aggregating polls), and seeing how they compare to the actual state-by-state win margins.
A lot of sites talk about who missed what state. I don’t think that’s as informative as being able to nail the margins. For example, 538 saying a state will win by 1% is very different than PEC saying a state will win by 5% and the state ending up being 6%. It will reflect on the techniques used to combine polls together, even if at the very end everyone is polls-only.
A good way to compare Nate Silver and other predictors against each other, is to compare them against themselves.
In other words, not only compare their Election Day Prediction with the actual election result – but also chart how much and in what direction their “prediction” changes in those final weeks.
That’s when Nate Silver & Co remove their “special sauce” (fundamentals), and base their predictions on the polls.
If there is a major shift in their prediction – without the polls changing correspondingly (!) – then that should invalidate the use of fundamentals.
The problem, as I see it, is that these predictors are only held accountable for their Election Day Prediction. Few journalists (and even fewer members of the public) are going to complain that:
“Hey, you’ve changed your prediction recipe at the last minute! It was only your last recipe (polls only, no fundamentals) that was accurate and valid!”
I also think about this – how to call a poll aggregator winner? 🙂
My guess is the 528 will essentially merge its predictions to Sam’s as they switch to polls only in the last month and turn down the secret sauce.
I think what would be more interesting is the delta from the actual final result over time. And, see which model was the most accurate overall during the cycle.
I think the current way aggregators are judged accurate is misleading. Silver can simply push towards polls at the last minute and say “I called ’em all right.” This is misleading because what he is selling is the sauce but that’s not what he’s using at the end (or anyone else). He’s just using the meat/raw data/polls.
Sam would you be willing to put out all your predictions over time so we could look the accuracy of the PEC model over time (and maybe call Silver to do the same).
Maybe Silver is some kind of genius but I think he’s becoming just another pundit/wanna be campaign manager groaning on about the key intanglibles/popularity. Put him next to the big board on CNN with his markers like John King.
Lojo, I am in agreement.
Regarding predictions over time, I think what you want is in this spreadsheet (which is accessible over in the code link at left). Columns A-D are what you want: Julian date, median D/I seats, average D/I seats, and day-by-day probability. I’d have to regenerate the long-term probability, but it’s not like “70%” is hard to remember.
The differences seem astronomical juxtaposing PEC with a Dem victory at 93% and Silver, et al with a slightly higher than even money for the GOP. They are not that different. If we could proclaim a victor in Alaska today, there would be remarkable consensus. And we’re back in Kansas, Toto. Again, if Repubs prevail in even 50 states (which they may well with Alaska), Senator Orman will ostensibly be compelled to caucus with Repubs. To renege on his pledge would be fatal in this deeply red state. Kansas, in this scenario, is a de facto Republican seat. Or not.
“…if Repubs prevail in even 50 states…Senator Orman will ostensibly be compelled to caucus with Repubs.”
By my count, if Repubs only prevail in 50 states, they aren’t in the majority. The “coalition” is in the majority, that is, a coalition of Independents and whoever they choose to support. Also, recall that the VP counts as a Dem vote. Thus, if the Repubs win 49, the Indies win 3, and the Dems win 48, the Dems actually have 49 (with the VP), which is a perfect tie with the Repubs in that circumstance. Who would presumed Senator Orman be “compelled” to vote with then?
IOW, it’s all in the definition of one’s terms….
Joseph – The terms will be defined by the voters of the fair state of Kansas. And that state is (by my count) deeply crimson. If Repubs gain 50, Dems 47, Indies 2; Repubs will be in the majority. You are asserting that no one has a traditional Senate majority at 50. But that is a difference best left to polemics. Can you imagine Senator Orman telling the voters of Kansas, “I said whomever was the MAJORITY PARTY and although Repubs have more seats than anyone, 50 does not technically define a majority?” Consider the context, Toto.
@ Insidious: “But that is a difference best left to polemics.”
I would estimate that a good 90% of politics is polemics.
Also, your numbers don’t add up. Last I looked, 50+47+2 equaled 99, and still leaves out the vote of the VP.
The fact that Orman is an Independent gives him tremendous flexibility. And should he win, there will be 3 Independents in the Senate. Who knows: Maybe they’ll start their own caucus….
The PEC prediction, because it is polls-only, jumps up and down quite swiftly. Look at the first derivative of the meta-margin, for instance at the beginning of September it’s nearly infinite.
Sites that use fundamentals (Upshot, 538) will have a smoother time dependence because fundamental variables simply cannot jump around as much.
So there will be times when new polls come out that the PEC will lead (as it is doing now). The others will slowly catch up to the extent they are allowed to by their constraints.
In other words, being an outlier is a feature, not a bug.
Nicely put.
I’m not sure why people are so offended by Nate’s comments. They have been quite tame compared with normal science discourse.
Okay, look, I’m saying the following in hope that it will be taken constructively.
The PEC model is “open” in the sense that I can go right now and download the m file. I can also get the csv that goes with it. But the PEC “open” model shares the distinction of being one of the most opaque “open” models I’ve ever seen (for instance, the csv file doesn’t label any of the columns, and the code only uses two of them anyway). I’ve made and published a fair number of mathematical models, some good and some bad, and so I’m fairly aware of how to read one. Sam has (in other threads) responded to some questions, but in none of the answers do I get the sense that any of the numbers have any justification whatsoever using empirical data. In each thread, his answers seem incredibly vague to me. He models, for instance, systematic error as a step function on the 35th day before the election. When asked why, he could say “here is the data for the past N years that shows it does that”, but he doesn’t.
If Sam wants to make a post with methodology, he should do so. The post should say things like “systematic error in the polls is assumed to be xxx”, and then show us the data that might suggest that. He also needs to explain why his model defies all notion of Bayesian probability. Why should a single poll result radically change someone’s opinion unless it’s a hugely compelling result? That isn’t how probability works. In asking whether the methodology truly is open, ask yourself whether you could use his data and write the code yourself, arriving at the same answers as his. That’s what’s meant by being open, that’s what reproducibility is. I don’t get the feeling that I would come up with syst_error = 3 on exactly October 1st. If you have found such a formula, please post it.
There would be no real opprobrium if Sam posted his empirical reasoning for the model parameters. The argument would then become totally factual. As it is, critics see ridicule as the only option, since you can’t argue against someone who isn’t arguing back with facts. Blocking Nate on Twitter (and proudly announcing it), and holing up on his site with his fans, isn’t how a scientist behaves, and Sam ought to know that.
Until Sam shows us his reasoning in detail, and again I say this with hope that he will, this site is beginning to get a cultish whiff of the unskewed polls people.
Disagree about the Twitter thing. It was hotheaded and I took it down…but there was little constructive about this week’s dogpile in that medium. As for how a scientist should handle this…no peer reviewer calls a model bad without providing details, and certainly not to a national audience of ~920K people like that. The only thing that saved the whole experience was a thread with Drew Linzer and Hans Noel, which was good though terse.
As for being opaque…really? I’ve been quite open and written several long essays. Many of my methods for 2012 actually went through peer review and will soon be published. It’s linked on this site soneplace – will post it more prominently later. (Update: here it is, text and figures).
Finally..I think you are right that certain technical questions deserve drilling into…but those details do not affect the overall picture that much. Really, it’s what I have said at length about AK, KS, accepting all polls including partisan ones, that kind of thing.
One last thought about your “cultish” remark…that seems like a low risk. You’re new around here, but I think it’s a pretty lively community. I learn a considerable amount in these comment threads. Given the last week, I would say Twitter is like being bullied on the street, and this place is more like a school or a home.
Reuben, I really must disagree vehemently with the “opaque” comment. Sam has been quite honest and open with both the methodology and the code.
I have often collaborated with others where we have had to exchange code and verify/debug results. Frankly, code that comes out of a working research group looks just like this. Elegant pseudocode with copious comments and flowcharts and everything is simply not a high priority.
I was able to download everything under Sam’s /code directory onto a Linux box with a wget command. Then I started with the nightly.sh script and followed where it led. It gave me a reasonably good idea of how the snapshot is made.
I haven’t installed MATLAB on the Linux box so I have not run the code yet, but plan to soon. Then I’ll put some print statements so I can follow along as it executes. This is the best way to get familiar with code someone else wrote.
My openness makes this possible – it was a fantastic catch on the other thread. Read that, and then think about how “unskewed” I am.
Nate’s comments didn’t offend me. I was just surprised that he chose to use the words that he did. Reasonable assumptions must come from the data and not political perceptions. If we went by that, the GOP “enthusiasm gap” would have ensured Romney’s election and a Republican Senate in 2010 and 2012.
As for a cultish whiff, OK, I won’t dress up as Dr. Wang this Halloween.
@ Reuben: “…this site is beginning to get a cultish whiff of the unskewed polls people.”
You never spent much time at 538 in the early years, did you? And “cultish whiff”? Really? As a long term Apple supporter, I’ve already seen this movie….
Open source by it’s very structure is about looking for collaborators and other like minded people, not disciples.
I guess I can understand the “cultish” remark based on the dialog after Silver’s mis-characterization of Sam’s work even as his own prediction started to move closer to the PEC prediction. Timing is everything. I don’t watch sitcoms. Real life is much funnier.
Intrade 2014 senate betting appears to agree with PEC…showing 69 percent chance dems retain control.
Intrade shut down in 2013. You’re looking at a single trade of 11 shares that took place 18 months ago. –bks
I believe PaddyPower and Ladbrokes will have current odds. Like Scotland, they will not be independent.
PaddyPower is currently showing Dems Senate Control at 11 -10; Repubs Control at 4-6.
I couldn’t find this on Ladbrokes site, but maybe I just didn’t know where to look.
I’ve learned from posters here that 538, which I have not been visiting lately, will stop pouring “special sauce” over polling data they aggregate in the final month before the election. So what is the purpose of “special sauce” besides adding brand identity on otherwise identical data ? When I decide not to order “special sauce” on my final death row meal simply means I never liked it in the first place and that I now have the once in a lifetime opportunity of letting the chef know how bad it actually was.
Nate’s polls have already moved quite a bit in the last 10-12 days. From 65% repub. to 53% the other day. It’s now at 54.8%
I don’t think Sam’s has moved nearly as much in the same period.
Nate no longer displays a graph so you can track the movement of his number over time. Perhaps i am missing a link. He used to provide much more data – including the no-sauce numbers. Now wonder his traffic must be down this cycle. 😐
|
From a long time reader, first time poster, “special sauce” is one of those newish political catch-phrases that I absolutely cannot stand, much like “game change.” It really doesn’t mean anything, other than some arbitrary definition of a set of fundamentals. Candidate A might lead candidate B in the Rasmussen poll, but hey, candidate B is actually winning because Nate Silver (or worse, a political pundit) poured some special sauce on the poll. Whatever.
The biggest question is how Orman will caucus if it’s 50 R to 49 D. That’s where the uncertainty is in many of the non-538 poll based models. But that ignores the fact that there will be a runoff in Louisiana. I’m election night it’s quite possible that it will be 49-49. Then Orman is free to caucus with whoever he chooses and it will be D.