A final unskewing
Mother Nature is our best teacher and the only one who is always right. – Viktor Hamburger, biologist In yesterday’s Los Angeles Times...
FiveThirtyEight’s blurred sense of time
Yesterday I reported early indications of a negative post-convention bounce for Romney. Some of you cited a claim by FiveThirtyEight of a positive bounce. However, that interpretation is incorrect. Because of limits in his statistical approach, he is reporting cumulative change since the end of July, a period dominated by the event I reported to you over a week ago: the post-Ryan-VP bounce.
A more accurate interpretation of recent events is as follows: Romney’s post-Ryan-VP bounce has slipped quite a bit toward pre-Ryan levels. The GOP convention bounce itself is zero at best and possibly negative.
First, let’s look at the data cited there.
Nate’s bounce error
This list contains eight before-and-after comparisons: five state polls and three US tracking polls. However, the pre-convention state polls were done ending on July 23 through August 5. So they are reporting the total net change since August 5. What else happened since then? Paul Ryan’s VP nomination:
At the time, I told you that Ryan-VP led to a three-point bounce for Romney. Yesterday’s report at FiveThirtyEight confirms this finding, but without information about when the bounce occurred.
What about the three national polls? They are more closely spaced in time, and report bounces of (positive means a pro-Romney bounce) +6.0%, +5.5%, and -2.0%, for an average of +3.2 +/- 2.6 % (mean +/- SEM), giving p=0.15 by one-tailed test — not statistically significant. So they are not sufficient to answer the question of whether or when a bounce occurred.
The fundamental problem here is that paired comparisons are very hard to come by. With that constraint, before/after comparisons must by necessity have poor time resolution. In contrast, by pooling of all state polls using the Meta-analysis, we were able to resolve the post-Ryan VP bounce in time.
>>>
Finally, a postscript on why the state poll Meta-analysis is so sensitive. Basically, it is an extremely effective means of reducing noise. Uncertainty goes down proportionally to the square root of N, where N is the number of polls. At any given moment we are using dozens of polls. The resulting estimator has an equivalent precision of about 0.2% in margin.
I know this to be the case for three reasons: (1) In 2004 and 2008, the Election Eve Meta-analysis came within 0 EV and 1* EV of the final outcome. (2) The typical daily fluctuation of the EV and Meta-margin history are quite small. (3) When there is a turning point, for instance with the Bain attacks and the Ryan VP nomnation, the results of the Meta-analysis change in a largely monotonic fashion, i.e. the amount by which they jitter up and down is <5 EV and <0.5%.
*In comments, Christian reminds me that attaining a 1-EV error required variance minimization over the final several weeks of the campaign. The final single-day estimates were 360/363/364/352 EV.
Topics:
Hi Sam, I’ve been enjoying your recent posts. I have a question abou the Meta-Analysis. In the 2008 results, you left a note that the election-eve snapshot was noisy. It predicted Obama EV = 352, but you had to do a variance minimization to hit Obama EV = 364. Do you apply a variance minimization for every Meta-Analysis result? If not all, what are the criteria for its use? These questions are in reference to your postscript. Thanks!
I apologize, and thank you for this correction. I had forgotten that such an averaging was necessary. I see here that the day-by-day EV estimator was 360, 364, 363…and landed at 352 EV, as you remind me. The difficulty was the “lumpiness” of the distribution since IN and MO had 11 EV each. So I am claiming too much. The confidence of the single-day estimator clearly depends on Electoral College quantization.
It appears that some averaging is needed. This gets back to what I found a few weeks ago regarding the fact that opinion is correlated for up to 40 days. A better approach would be to take that into account. Variance-minimization was an approximation to that, kind of seat-of-the-pants. How to do better?
Meanwhile, Todd Akin’s anti-bounce appears to be ending; he’s at parity with McCaskill in recent polls.
Sic transit.
For Akin need one more data point I think?
…ah, if only Dr Wang could obtain polls on request.
I have to say, although I really enjoy both of your sites, and have always found the little tit-for-tats between you kind of silly, that this criticism of 538 seems to be right on. It is pretty obvious that the method he has used to adjust for convention bounce is flawed and he is trying to cover up that flaw by finding a bounce where there is none.
Leo – totally reasonable on your part. To be honest, I don’t particularly enjoy it either. But when fanboys/girls cite a statement that is statistically subpar, it’s hard to hold back. He gets you interested, I try to correct the wrong analysis.
I’ll get back to my own math shortly.
Wait…he doesn’t link to me, does he? I don’t read his site regularly.
The funny thing is Nate’s own model is showing that there was no meaningful bounce. His version that is based only on polling has been flat while his version that is correcting for something that isn’t there is showing an Obama bounce.
I do appreciate what he is trying to do, if only as an experiment. I mean his prediction model is trying to tell the future, after all, and it is reasonable to claim a lack of a Romney bounce means that Obama is more likely to win. I am skeptical that he can take everything into account in a mathematically meaningful way. So yes Obama is probably some unknown percent more likely to win because Romney failed to get momentum coming out of his convention. Is he the couple percentage points likelier to win that Nate’s model has calculated? I’m not so sure of that.
I also appreciate Dr. Wang’s straight forward approach. I think his simpler is better approach is probably right, especially given how robust his simpler model has proven to be. I think his critiques of Nate have helped me understand the problem better than I had before with my very basic understanding of statistics.
I know that most of my statements are statistically subpar, but I believe it’s just because of a string of bad luck.
I am a former researcher and current practicing forensic psychologist, I am continually amazed at the rather obvious discrepancy between cited data (e.g. testing results) and stated conclusions.
Evidenced-based practice requires consistency. Nate blew it on this one; he doesn’t get a pass.
Sam — I don’t think he links to you know from his NYT site. I thought that he had done so back in 2004 when he was new kid on the block but I could be wrong about that.
That is not possible because I was doing this in 2004. He started in 2008.
Oops, I meant 2008.
Nate never links Dr. Wang.
Sometimes the statistics nerdcore does in the comments.
there was one link or maybe just a citation, usng Sam’s name, I think, not the PEC label, a few weeks ago.
xian, no cite. PEC was mentioned in the average of poll aggregators post.
Dr Wang: I would think, because the sample sets are different sizes, that the proper way to average polls is to resynthesize the quantities, sum the totals and then get the new percentage.
Sam,
Two off-topic questions:
To what extent can very large and asymmetric expenditures on negative advertising this late in an election cycle (e.g. what we will see from the Romney campaign and his SuperPacs) move the needle on performance? Is this accounted for in your model?
Thanks!
Sam Champion – The only input to the calculation is polls. I regard everything else as being “upstream” of polls or just adding noise. Read this to get my take on stuff like what you are describing.
Campaign spending might be informative in two ways: (1) filling in missing data, as in Senate/House races; and (2) predicting future movement. At the level of the Presidential race it seems unlikely to have an effect. Think about it. Do you think that Obama lacks for a platform?
I have written about all this over the last month…
There’s something undeniably odd going on with the national polls: the rising trend toward Romney has completely closed the gap, and he’s even with Obama or slightly ahead, while he remains behind in EV.
This isn’t Republican convention bounce; my best theory is that it’s continued Ryanmania. The rise started when Romney picked Ryan.
But it barely registers in the EV aggregates or the Meta-Margin, so it might be mostly red-state Santorum or Gingrich voters coming home and making the places unbelievably red, with little effect on the actual election.
Still, Ohio and Wisconsin are now registering as toss-ups, so maybe the convention did give the Republicans a little Great Lakes bounce after all.
It will be interesting to see what happens in the post-DNC polling, given the amazing reception Michelle Obama’s and Bill Clinton’s speeches got. It’s hard to keep it all in perspective and remember that speeches and conventions have temporary effects at best.
…More generally, if there’s anything the history of this campaign demonstrates, it’s that the 2000-inspired lore that the Electoral College favors Republicans is not necessarily true.
Matt, couldnt it just be robopoll bias?
There’s probably some robopoll bias in there (Rasmussen has Romney ahead by 3), but that doesn’t explain changes over time, since the robopolls have been heavily weighted in those averages all along. And some of the polls showing Romney even with Obama aren’t robopolls.
Matt, the old electoral college lock dated from the Reagan/Bush period, when California was still a red state (imho).
Yeah, one thing that the Electoral College obviously does is amplify clear majorities into massive blowouts. And in the Reagan/Bush days, that was what Republicans had.