A final unskewing
Mother Nature is our best teacher and the only one who is always right. – Viktor Hamburger, biologist In yesterday’s Los Angeles Times...
75 days
A few of you have written asking if I plan to change my Presidential prediction based on recent changes. The answer is no.
Some envelope scribblings today to show the basic reason: until the end of September, the daily snapshot is not correlated with the final outcome. Read on.
As an illustration, here is the 2008 Obama-McCain race, measured in units of Meta-Margin (defined as how much popular-vote swing would tie the race):
That race’s major events included the Palin VP nomination, both party conventions, the first debate, and the September economic collapse.
It is possible to calculate how much, and how quickly, the 2008 race tended to move. Here is a plot of average changes going forward in time, calculated for the entire campaign season:
The shaded gray area indicates the +/- 1 standard deviation band. Like a hurricane strike zone, the gray area gets wider over time. (Update: for more discussion see the comments.)
Here is the important part: the gray band reaches a maximum width in 40 days. Only when the election was less than 40 days off (September 27th) did the snapshot have any short-term predictive power. The same tendency is apparent in 2004 data.
This loss of predictive power over time explains why the Princeton Election Consortium’s statements are consistent regarding (1) a snapshot of conditions today, and (2) a long-term prediction of the November 6th election outcome:
The snapshot today is Obama 285, Romney 253 EV, and a Meta-margin of Obama +1.1%. An election today would be nerve-wracking for both sides. The long-term prediction, based on the fact that voter sentiment mostly stays within a finite range, uses a past Meta-margin standard deviation (MMSD) of 2.2% and this year’s average Meta-margin of Obama +3.25% to give a November re-elect probability of 88%.
The bottom line: These two measures, snapshot and long-term prediction, are potentially divergent from one another until the end of September. After that, a short-term prediction will become possible that is driven by the current snapshot.
Topics:
Sam,
As usual, very interesting comments. But I’m not quite sure I understand the math today, in particular what your plot represents.
For example, in your graph, at about 30 ‘days going forward’ the ‘average change in popular meta-margin’ is almost 1%. Does it mean that you average the 2008 meta-margin difference for all points separated by 30 days?, i.e. for example you look at the difference in the meta-margin between June 5th and July 5th, the difference between June 6th and July 6th, then June 7th and July 7th, and average all these numbers, which gave you the 1% data point?
Also, are you suggesting that today, with the meta-margin at 1.1%, the election would be quite close (from your EV histogram of outcomes, it seems like a 70-75% chance of Obama reelection), BUT that in the long run, it is actually more likely that Obama will regain ground for a higher reelection probability of 88%? The math is certainly right, but it doesn’t seem intuitive at all that from the current situation at 1.1% margin, we can infer that the race is more likely to evolve in Obama’s favor from now on. If anything, it seems to me that since there are more days before the election, there is also more uncertainty.
Would be great if you could clarify a little. Thanks a lot!
Another question: some time ago, you provided an analysis of the re-elect probability (i.e. in November). I was thinking it is easy to know that probability if the election were held TODAY (from your EV histogram, adding all the probabilites of Obama getting at least 270 EV). Wouldn’t that be a useful indicator to display on your graph? (of course making clear that this is the probability win today)
Pat – I am traveling and can’t write you something concise. Pardon the long answer.
“for example you look at the difference in the meta-margin between June 5th and July 5th, the difference between June 6th and July 6th, then June 7th and July 7th, and average all these numbers, which gave you the 1% data point?”
This is correct.
(from your EV histogram of outcomes, it seems like a 70-75% chance of Obama reelection), BUT that in the long run, it is actually more likely that Obama will regain ground for a higher reelection probability of 88%? The math is certainly right, but it doesn’t seem intuitive at all that from the current situation at 1.1% margin, we can infer that the race is more likely to evolve in Obama’s favor from now on.
This a perceptive comment on your part. I am traveling and so I will only unpack it partially.
The gray-zone confidence interval on the daily snapshot. This parameter is a little misleading because it combines both true uncertainty and pollster-pollster variability. And pollster house biases mostly average out. To arrive at a true uncertainty estimate, I would have to pre-subtract those. Thus the true precision of the EV estimator is quite good, probably +/- 5 EV. It’s been a low priority because only rarely do I get a probing question like yours.
At any given moment, the leading candidate’s true probabilty of victory is well over 95%, even today. More typically it’s >99%. I have come to the conclusion that this is not a useful parameter to tell people since it leads to false certainty about the future.
The autocorrelation. The autocorrelation (i.e. the forward evolution graph shown in this post) is calculated using only the Meta-margin, with no consideration of today’s confidence interval. It does not include anything about the gray-zone confidence interval described above.
The prediction. This prediction also only uses the Meta-margin. Therefore, implicitly it is equivalent to assuming that on any given day, the uncertainty is very small. This is reasonable because by November, opinion may shift by +/-2%. The daily snapshot’s true uncertainty is smaller — <0.5%, as evidenced by its performance in 2004 and 2008.
@Wheeler’s Cat wrote on Tuesday: “But there are at least two distinct cell phone demographics– cell-only households, and smartphone users. I think both are resistant to traditional polling techniques. But just because those demos are unlikely to be polled, doesnt mean they are unlikely to vote.”
Sam, I know you touched on this four years ago, but do you have any thoughts on whether (and how) the under-representation of cell-phone and smart-phone users might be skewing the polls we’re getting?
Also, you mentioned that there was a host of Republican-leaning polls released shortly after Romney’s pick of Ryan as his VP choice.
Are we still seeing a dominance of Republican-leaning polls?
I’m just not feelin’ it Dr. Wang.
Holy kurtosis, Romney gets 0% of blacks.
http://www.politico.com/news/stories/0812/80015.html
Michael Medved wrote an article in 2010 that shows Romney needs an impossible 65% of the white vote to beat Obama in 2012. That means O needs only 36% to shut him out. Kerry got 37% of white males and 41% of white females.
Take my demographic– college-educated white female smartphone users. Does anyone think we are unlikely to vote?
And we go Obama by +60%.
Certainly we are unlikely to be robopolled.
I think smartphone useage and cell only households have invalidated the traditional sampling methodology. And the pollsters dont really care because they are paid to propagate the horse race mythology.
Demographically, Romney is toast. The election is already over if liberals can turnout their base.
and Olav, here is simple model for skewness.
We can call it landline bias.
Robopolls can access landlines, but not cells.
12.9% are landline only, 55% are landline + cell, and 31% are cell-only.
Who has landlines? White male property owners and old people. The only two demographics Romney carries.
Consider the robopoll Akin commissioned that convinced him he should stay in the race. Landlines only.
The concurrent PPP poll had something like R + 9, when their previous polling had D + 2.
Landline bias.
@Wheeler’s Cat: “Demographically, Romney is toast. The election is already over if liberals can turnout their base.”
You’ve hit the nail on the head, as far as I can see — but that’s a pretty big IF. In addition you have the systematic efforts to reduce the integrity of the election process, a critical topic that I hope Dr Wang will address thoroughly later.
Well I think there is a “systematic” collusion between pollsters and pundits to headfake the electorate into thinking Romney has a chance.
dig this comment from 538, Olav.
http://fivethirtyeight.blogs.nytimes.com/2012/08/20/aug-20-when-the-polling-gets-weird/?comments#permid=82
Eric FosterMichigan
The reason why it is difficult for polling firms to contact people via cell phones:
The Telephone Consumer Protection Act (TCPA) (47 U.S.C. 227 , 47 CFR 64.1200) prohibits the use of an “automatic telephone dialing system” to contact “any telephone number assigned to a …cellular telephone service” without “express prior consent” from the party being called.
Automatic telephone dialing systems (known as “autodialers”) are an essential tool of survey and opinion research. Almost every telephone research call requires them, in order to reduce both the time it takes to dial a number manually (a significant cost) and the chance of error. Automatic dialing also synchs better with the industry standard “computer-assisted telephone interviewing” (CATI) software that helps researchers properly execute and conduct calls.
Sam, What’s your take on this?
http://www.huffingtonpost.com/2012/08/22/university-of-colorado-pr_n_1822933.html?utm_hp_ref=politics&ir=Politics#comments
I think that political scientists do not have the capacity to make predictions that good. During a campaign, polls are the superior instrument. I do not take that pronouncement seriously.
If Romney wins, it will be very close. Let’s revisit their prediction on November 7th.
Dr. Wang, you say….
“At any given moment, the leading candidate’s true probabilty of victory is well over 95%, even today. More typically it’s >99%. I have come to the conclusion that this is not a useful parameter to tell people since it leads to false certainty about the future.”
Its not JUST that….eg, if Nate Silver came out and said Obama has a 99% chance of winning, wouldn’t that change him from an observer to an actor?
Because one consequence might be that the neo-corporatists stop funding Romneys campaign.
Doesnt that kind of forecasting disenfranchise half the voters in this country?
Political forecasting is like q-physics.
Matter changes under observation.
Wheelers cat – I am unaware of any evidence that poll analysts have enough power to influence public opinion. Sports analysts cannot psych out ball players.
I do, however, think that we get most of our information about a race through mass media, more broadly speaking. Does mass media influence public opinion? Sort of, but the media also follows along behind opinion. Sarah Palin spoke in a way that touched people directly, and the press had to follow. In the run-up to the Iraq war, what fraction of the press had the nerve to question our leaders’ misleading statements on WMDs? I view the media as both a mirror and amplifier of opinion.
To summarize: the smartest gas molecule inside the piston is still just a molecule.
@Wheeler’s Cat: There are times when I have wonder whether what you describe might be the very strategy of Rasmussen Reports/Polls and, of course, the election coverage of Fox News.
Case in point: I find it striking that just after Ryan is added to the ticket, we get a whole sleight of Purple Strategies and Rasmussen Polls, showing a rather impressive Romney bounce.
I cannot help but suspect this timing is as much about writing the narrative as it is about polling.
I suspect the over-sampling of Dems we’ve seen that have consistently shown Obama with substantial leads are due to the pollsters attempting to estimate the preferences of the cell/smartphone users they do not poll
wufwugy, Olav, wheelers cat – I wonder if you are all reading your own desires into the polling data. I do not think I will win a prize for guessing that wufugy favors Romney, and Olav/wheelers cat favor Obama.
There is nothing wrong with approaching these questions with a point of view. We all have one. However, it is always good to stop and ask yourself whether you would have the same belief if the data argued against your preference. This is a well-known cognitive bias. For example, wufwugy, if cell-phone-users largely favored Romney, would you have the same opinion? I have similar questions for Olav and wheelers cat.
The fact of the matter is that polls become very accurate when enough of them are aggregated. In my direct observation, this was true in 2010, 2008, 2006, and 2004. It is wishful thinking to imagine that Obama is magically behind in public opinion. National and state polls give two independent measures of the same fact: President Obama has been ahead for this whole election season, and his lead has narrowed in the last two weeks. Today’s data suggests a slowing or stopping of that change.
I have spoken with a survey expert, Ed Freeland at the Survey Research Center, about cell-phone sampling. Professional pollsters are very interested in the cell-phone-sampling issue, which is a specific example of the stratification problem: how to infer the properties of the actual-voter sample using the sample that is available. This is apparently a lively topic of discussion at the American Association of Public Opinion Research (AAPOR) conference.
There is always a possibility that one voter group or another is oversampled or undersampled. However, pollsters are very well-informed actors in a job that puts them constantly in the public eye. I think there is only a very little bit of value in second-guessing them. My opinion places me at odds with many polling fans, yourselves included. It accounts for why I am unfavorable to most questions regarding house effects, voter ID laws, undecided voters, and so on. I trust the wisdom of the crowd, analyzed by proper methods.
Sam,
Thanks a lot for your long answer. I am still confused. You say that the leading candidate’s true probabilty of victory is well over 95%, even today. The EV histogram of outcomes does not reflect that. Does that mean that if you removed the uncertainty due to the pollster house biases which in fact average out, then the “true” histogram would be actually much narrower with much fewer outcomes on the “Romney wins today” side?
If so, it would certainly be interesting to have an idea of how it looks like. I suppose it would just have a smaller number of spikes, each of which at a rather high probability?
Exactly. You get it. In fact, in mid-October the histogram will start looking very sharp indeed as polls become abundant.
@Sam, I think you misunderstand me. I favor Obama rather heavily, and am just trying to figure out 1) why so many polls have higher Dem sampling, and 2) how to account for cell users. One of my guesses is that because they aren’t including cell users in the polls, the Dem oversampling is deliberate by the pollsters in order to present their analysis of what including cell users would do to the data. That may not be accurate though, and I don’t know much about the sampling process
Aha, I see. Well, that was an interesting misunderstanding on my part. Reveals my own biases in how I read questions.
Generally speaking, pollsters have to make assumptions about what the voting population ought to be like: age and gender distribution, and lots of other factors. Then they re-weight their sample to bring it into line. As a simple example, a pollster might reach a disproportionate number of senior citizens during the day, then weight their answers less to compensate. This is stratification.
As for cell-phone sampling…I don’t think it’s likely that the entire profession of pollsters went off a cliff compared with 2-4 years ago. In 2010 they did make one error in the Nevada Senate race (Reid v. Angle), which could have been because of the exceptionally large number of people moving to and from Nevada (and therefore lots of cell-phone-only users). So there is that.
I think it’s important to know that (as corroborated by Nate Silver too), there just hasn’t been that much state-wide polling in this election cycle. You can see this effect in several swing states in the past few weeks that were alternating colors back and forth (ie. Colorado, Florida, Ohio) due to single polls that report significant margins for either candidate. Obviously people don’t change their minds like that, but that is a sampling issue that more frequent polls from more agencies would get rid of.
I’m actually surprised at the lack of Senate polling, considering there is a very real possibility there could be a 50/50 or a 49D/51R split.
Billy: Low polling density is of less consequence with proper aggregation methods. See EV history. It’s pretty smooth in time.
/sigh
Dr. Wang. Do you know what my reply to Eric Foster’s comment on the difficulty of polling cell-onlies and smartphone users was?
I said does that mean you are not polling 31.6 % of the electorate? (cell-onlies)
He had no answer.
Thats almost a third of the electorate.
You know very well there is asymmetrical political behavior bias operating in the pollster community.
House effect is not symmetrical, it doesnt “cancel each other out”.
And there is no market force pressing Ras to do better polling. Everyone consumes his data.
And we are Not. The. Same.
So cut out the “both sides do it” bullshytt.
wufwugy
most polls have higher numbers of of democrat respondents BECAUSE THERE ARE MORE DEMOCRATS.
And this was exquisitely explained in the sampling post that Nate hastily retracted three weeks ago.
To summarize: the smartest gas molecule inside the piston is still just a molecule.
Yet the nanoparticles inside the gas molecule may very well be involved in spooky action at a distance. Its all about the granularity.
Does predicting the future with absolute certainty create the future? That was my question about “true probability”.
What if Nate said to his much larger audience (he is the pre-emminent statistician of known blogspace) what you said?
That Obama has a >99% to win?
Would Romney’s superpacs cut him off? Would Mitt drop like a stone on Intrade? Would Sheldon Adelson start giving millions to down ticket candidates?
What is the meaning of self-fullfilling prophecy?
Sure forecasting can shape the environment.
Havent you read Dune?
😉
@ wheeler’s cat: m_c, it is good to see you again. It hasn’t been the same without you. I’m glad you’re still kickin’.
Dr Wang, I have been contemplating your responses, but still suspect that some pollsters — especially Scott Rasmussen — are intent on writing the narrative. In fact, if you read the *narrative* that Rasmussen himself writes on http://www.rasmussenreports.com/ , this becomes very clear. To call it tendentious would be a huge understatement.
And that carefully slanted narrative just happens to play well with Scott Rasmussen’s clients. In other words, I am accusing Mr Rasmussen of what you call *cognitive bias*.
@Wheeler’s Cat: Do you have a copy of Nate Silver’s post? Surely it might be circulating on the web somewhere. Could you perhaps provide a summary.
Interesting that Akin trails McCaskill by 10 % — in other words, a 21 % movement in the polls since before his infamous comment.
I am still wondering to what degree there are signs of Akin expressed position — which are very much in line with the GOP platform — are influencing other Senate or House races, or the Presidential Election. Any clear evidence of this? Or has Romney’s distancing been effective?
The question is, how many women voters are going to take advantage of this option:
https://sphotos-b.xx.fbcdn.net/hphotos-ash3/564210_4358643374253_567236758_n.jpg
Olav….sadly no, and Im kicking myself.
I think hes simply never going to repost it.
I think someone made him pull it.
The jist was 13 points all about how oversampling was untrue.
I cant find it in any cache, every comment I make asking for it gets deleted by Micah. Probably Micah deletes all the regulars comments about it.
Nate promised to have it up in a week…that was three weeks ago.
Why would Nate Silver do that? Who told him to retract it?
I sure dont know. I am sure it wasnt a mistake. He made the tiny url for twitter with the correct name of the post.
And Rasmussen isnt deliberately evil…..he just doesnt care. Robopolling delivers the only demographics he cares to sample, white male landholders and old people. His sampling is consistantly +R, unlike other polling houses. And as long as poll aggregators consume his product he has no incentive to improve his sampling methodology.
I was semi-joking about a collusion– there isnt overt conspiracy– there is just the invisible hand of the free market. Rasmussen and Nate Sliver are both the product of market forces. Its like regulatory capture in the “freed” market. The conservative half of the country just wont pay for media that doesnt reflect their worldview.
And that raises a question….Dr. Wang, how do you keep this site free of the low information conservative trolls that infest 538 everytime Sully or Politico or HotAir links Nate?
I don’t know. When Andrew Sullivan links here, the commenters still keep it clean.
In general I am interested in all readers who want quantitative analysis. Why not go over there and tell ’em to come over?
New Silver.
Seven Ways to evaluate a poll.
http://fivethirtyeight.blogs.nytimes.com/2012/08/24/aug-23-seven-ways-to-evaluate-a-poll/?utm_source=twitterfeed&utm_medium=twitter
All Nate’s posts this season have that mushy “if on the one hand then the other” equivocating.
Total capture by market forces.
He would NEVAH say something like Dr. Wang said in this post, about “true probability ” and Obama’s real chances being > 95%.
Thanks for looking!
I do wonder, however, whether the 538 blog entry that you mention might be quoted on other fora. Any chance of searching based on text strings or keywords that you remember?
Nate Silver’s bottom line today does not see so mushy:
“…the story of Thursday’s polls wasn’t all that complicated: Mr. Obama held leads in all six of the polls released of voters in Ohio, Florida, Pennsylvania, Wisconsin and Michigan — and Mr. Romney will need to win at least two of those states, and possibly three, to have much of a chance in the Electoral College.”
It was only up for moments.
I didnt even have a chance to comment.
🙁
I think….Obama’s small but seemingly immutable lead is what makes the “true probability” of an Obama win >99%.
Unless Romney can flip the curve, Obama has pole position in the horserace. Romney is no Seabiscuit or Secretariat.
He just doesnt have the substrate to come from behind, and if Ryan and the convention cant give it to him, he just cant win.
Its the appearance of invulnerability…..
heres a text string.
“Let me make 13 relatively brief but interrelated points that explains my philosophy on this issue”
but all the links go to the replacement post.
🙁
heres the original title
“Why Charges of Poll “Oversampling” Are Usually Misguided”
I think I found it!
This appears to be Nate Silver’s blog entry:
http://finance.xyrm.com/2012/08/why-charges-of-poll-oversampling-are.html
Olav: Those all seem like perfectly sensible points in the lost post of Silver. Good work and thank you!
Holy kurtosis Olav!
I bow before your search-fu.
Dr. Wang, you dont want ’em. They are like parrots or low level AI bots.
They just repeat the same talking points and waste Nates bandwidth.
Yes Olav, that is it! the 13 points I remember.
Is that a 538 mirror site?
So Dr. Wang….why did Nate hastily retract the post and never repost it like he said he would?
What is in that post that is objectionable or unfinished?
Is it this?
“Recently – perhaps because Mitt Romney still narrowly trails President Obama in most state and national surveys – we have seen a bit more of this from conservatives. They will sometimes allege that these polls are “oversampling” Democrats, including too many of them in their surveys, and perhaps biasing their results toward Mr. Obama because of this.”
Because that is a widespread conservative meme. It is consistently used to denigrate poll results that conservatives dont like.
Obviously Rasmussen’s results have a Republican lean relative to most other polls, but his track record at actually predicting results on election eve isn’t much worse than anyone else’s, is it? Once you decide to start correcting polls for house effects you’re introducing a guess about where the true center is.
The main problematic thing about Rasmussen is just that his tracking poll (and Gallup’s, which is also somewhat Republican-leaning) releases results much more frequently than anyone else’s, so if one weights every released result equally (which Prof. Wang does not), they dominate the averages. But more intelligent aggregation fixes that.
You know Dr. Wang….
You are the one that called Nate a “color commenter” this season.
What happens if Nate debunks the “dem oversampling” myth?
Would conservatives be forced to acknowledge the horserace is already over?
@Matt
Poll saturation is right out of How to Lie With Statistics. Another thing Rasmussen does is poll selection, where focuses attention on specific polls where the GOP is more likely to do well.
The main bad thing Rasmussen does is that he is a robopolling house, the only place he doesnt use robopolling is where its illegal.
And intelligent aggregation can fix everything…. except I think ignoring whole demographics, like Eric Fosters poll ignored cell-only households (31.6% of the electorate).
At what point do aggregators and analysts ignore Rasmussen? His numbers on Akin losing 20 to McCaskill are suspect but not entirely unreasonable and an acceptable coincidence, but Romney now losing 11 points to Obama in the state just a day after the post-Akin debacle PPP poll with Obama/Romney at 52/42 has got to be the most obvious lie I’ve seen in polling data. Rasmussen isn’t a pollster; he’s a lobbyist masquerading as a pollster
wufwugy
you cant ignore him, hes everywhere. Robopolling is the cheapest form of polling.
Information is valuable, it costs money, so use it. Good aggregation techniques can smooth outliers and damp noise.
And like Dr. Wang asserts, the law of large numbers will take care of most of the problems….except for asymmetrical political behavior bias.
@wheelers, the problem is that Ras is lying. Perhaps not always, but the Obama +1 in MO is a blatant lie. He pulled that number right out of his ass in pursuit of his agenda.
There is no way in hell Romney lost 11 points to Obama because of what Akin said. Every other poll completely disagrees with Rasmussen, there are way too many coincidences which we already know about, and the shift isn’t seen anywhere other than MO. The last point is the most important actually. If Akin hurt Romney in MO by 11 points, it would reflect in other states as well. Only in Rasmussen’s lying-ass fantasy land are MO Republicans turned off to Romney because of Akin yet every other Republican in every other state is not