Princeton Election Consortium

A first draft of electoral history. Since 2004

An apology to poll-dissecters

October 9th, 2012, 9:30am by Sam Wang


The appearance of an exceptionally favorable Pew poll to Mitt Romney has many of you in a tizzy. You have asked me to comment further. I am really not good for that kind of thing. I respectfully submit: neither are you! Here is why.

Human beings engage in motivated reasoning: we look harder at evidence that disagrees with our existing (or desired) beliefs. From a statistical standpoint this makes sense: any one piece of information that we receive might be wrong. We carry a model of the world in our brains, and rely on that to interpret everything that comes in. This is sometimes called Bayesian inference.

This even permeates to sensory perception: in the above phenomenon demonstrated by Prof. Patricia Kuhl, the McGurk effect (click here if the video doesn’t work), we rely on visual cues (in this case mismatched) to help infer what is being said.This principle also underlies most visual illusions (cool site!).

If motivated reasoning succeeds, a closely related consequence is biased assimilation, in which agreeable facts get in, but disagreeable ones don’t. This can account for the coincidence that people who don’t like President Obama’s policies also mistakenly believe he was born in Kenya. Sandra Aamodt and I wrote about this in 2008 (“Your Brain Lies To You,” NY Times). in  Nobody is immune to this kind of thing, including me. This is why I devise statistical measures that attempt to resist such biases – automatically.

Left-learning readers of this site are sniffing around the internal details of the Pew survey. None of your suspicions change the fact at that present, President Obama has taken a 4-point drop in Popular Meta-Margin. It looks a lot like the effect of the Ryan VP nomination, which lasted for a while; or the Palin VP nomination, which did not.

Ryan and Palin bounces in Meta-margin

Detailed comments on an individual poll (i.e. poll-dissection or poll-sniffing) are well-suited for discussions at FiveThirtyEight (and will be heavily moderated here). A sabermetrician like Nate Silver thrives on those details, and when it comes to analysis he is pretty good at suppressing his own biases. The approach here at the Princeton Election Consortium is to compile lots of polls to remove noise. It’s a bit bloodless, but it has worked well in the past. Anyway, give it all some time.

Tags: 2012 Election · President

17 Comments so far ↓

  • Michael Worley

    Sam,

    Isn’t the drop in the MM now bigger then the Ryan drop? I think that’s worth noting.

    Michael

  • wheelers cat

    so….instead of poll sniffing Pew….maybe this could be a place to talk about Nassim Nicholas Taleb and black swan events and Solon’s Warning?

    • bks

      It should have been easy for Obama to eke out a split decision in the debate. Instead the punditocracy called it unanimously for Romney. That is a black swan event. Strangely, no one can point to anything that Obama said that was damaging, it was merely his demeanor (or the contrast in his demeanor versus Romney’s) that swung the decision. I think that is what Sam is getting at with the psycholinguistic video.

      –bks

  • Ms. Jay Sheckley

    As for the National mood, Steve from the Rachel Maddow Show ran down the polls, concluding,”It’s a close race. It’s going to remain a close race. It was always expected to be a close race. Those on the verge of falling to pieces need to get a grip.”
    [ Rachel Maddow Show quoting itself on Facebook just now, referencing http://maddowblog.msnbc.com/_news/2012/10/09/14316102-with-four-weeks-to-go ]

    We maybe should factor in that Sunday was this election’s “Pulpit Freedom Sunday” Evangelicals in over a thousand churches advocated particular candidates and sent evidence to taunt the IRS. Being perhaps odd, I think maybe thouse churches should consider _offering_ to pay taxes, because Jesus said, “Render unto Caesar that which is Caesar’s.” Then again, I’m such an extreme moderate I don’t get why the kids aren’t spraypainting “Matthew 23″ everywhere….
    Does anyone know what effect “Pulpit Freedom Sunday” had on polls in 2008? 2110? Info please!

  • Paul

    I have a general methodological question about polling aggregation, unrelated to the recent Romney surge.

    Are there any criteria that a polling organization has to meet to be included in the averages? Can anybody with a website who claims to be conducting polls get included, even if they have no track-record, reputation, clients (news or political), or evidence of expertise in polling or politics? And are you concerned about the possible effects of less-than-professional partisan pollsters flooding the zone with favorable polls, as polling averages become an increasingly important part of the media coverage of campaigns?

    Again, this question is not related to the recent Romney surge, which seems to have been confirmed by a number of reputable pollsters. But do you think this may be an important concern going forward?

    • Sam Wang

      It is not that expensive to open up a polling shop. Median-based statistics get you so far, but if there are too many bad data sources, then a problem arises. One possible screen would be to follow a precedent set by Mark Blumenthal and others. Ask pollsters if they belong to AAPOR, if they subscribe to the principles, and so on.

      In past years it’s been outfits like the Columbus Dispatch and Zogby Interactive. This year, maybe Gravis, though they seem more hapless than anything else. If poll aggregation took over the world, the problem might arise.

  • Alan Cobo-Lewis

    Sam,

    Nice citation of McGurk effect.

    A couple weeks ago the right didn’t like the polls so engaged in that silly “unskewed polls” nonsense. They did things like saying that polls oversampled Dems, and they “corrected” for it by reweighting them. (The big fallacy there was that the unskewers regarded party affiliation as a demographic rather than an attitude. They didn’t understand that party affiliation changes, and typically correlates with the presidential preference expressed by the respondent. As respondents change their minds about presidential preference, some follow on by changing their self-identified party affiliation. See good non-technical discussion by my colleague Amy Fried at http://pollways.bangordailynews.com/2012/09/24/national/big-drop-in-people-calling-themselves-republicans/

    Now the shoe’s on the other foot, and some on the left would like to unskew a particular recent poll. Leaving aside the tendency to overinterpret a single poll (which your median-based poll aggregation helps us guard against), the link you posted to http://www.electoral-vote.com/evp2012/Pres/Maps/Oct09.html#item-1 seems to show the electoral-votes.com site master actually trying to unwrap party affiliation. It seems pretty much like a mirror-image from what some on the right were doing a couple weeks ago.

    So the left correctly ridicules the right when they do handstands to ignore data that makes them uncomfortable. OK. But the left might have some trouble recognizing when they’re making the same error.

    Reminds me of the fish-fis phenomenon (Berko & Brown, 1990). Lefties can detect the error in righties but can’t keep themselves from making the same error.

    alan

    • xian

      isn’t the electoral-vote guy a libertarian? I was also surprised to see him fall for that same fallacy. Then again, I find his editorial content a fairly bland recitation of conventional wisdom.

      I also saw lefty commenters (of which I am one) claim that the party identification doesn’t pass their sniff test, an exact mirror to the right-wing critique.

      I think much more we are seeing people drop their shame at being Republicans and identify with someone whom they now believe might possibly be a winner, but I also expect this effect to subside unless Obama makes more mistakes.

  • Matt McIrvin

    I always got the impression that Tanenbaum is a centrist Democrat, but that’s reasoning backwards from his editorial comments.

  • Ms. Jay Sheckley

    Steve from the Rachel Maddow Show ran down the polls, concluding,”It’s a close race. It’s going to remain a close race. It was always expected to be a close race. Those on the verge of falling to pieces need to get a grip.”
    [ Rachel Maddow Show quoting itself on Facebook just now, referencing http://maddowblog.msnbc.com/_news/2012/10/09/14316102-with-four-weeks-to-go ]

  • JohnJacobs

    Hey Sam,

    Unfortunately the auditory illusion didn’t work for me :-(

    Interesting to see Obama’s continued (and rapid) deterioration in your projections. Brings you right in line with my other – I won’t say who’s #1 ;-) – favorite prognosticator, Nate Stein.

    What would it take (in terms of meta – margin days of movement longevity or scale) to convince you that the dynamics of the race have conclusively shifted. I ask, because it seems that both you and that other guy are hesitant to cause a shift partly due to a preconcieved notion that single debates can’t be game changers so close to election time.

    I propose an analysis of the “Ryan bounce” (which you claim resembled the current one)

    P.S. Have recent events changed ANY of your prior hypothesis about elections, October, debates, or the nature of love in a fallen world?

  • Jack Rems

    What is happening with the Jerseyvote?

    btw, have you ever considered using cash values in “the power of your vote”? One approximation would be to set the sum of all votes at the $2b that the candidates will raise and spend. I’m sure both sides have done this calculation.

  • Brian MacDougall

    I was watching the video and my wife stuck her head in the office door and said, “WTF?” Pretty sure I heard that correctly when I opened my eyes. But who knows? Maybe she asked me if I wanted to buy a duck.

    Seriously, I come here when I start hyperventilating, just like I did in 2008. Always informative and entertaining. And calming. Very calming.

  • Obama 2012

    Looking at 538, Nate Silver says there’s now a nearly 10% chance of a recount … (within 0.5% in a deciding state…) I don’t think I could handle that.

    We liberals tend to be a paranoid lot. While those on the right look at polls where they are losing and figure out how they’re winning, we tend to do the opposite.

    Or we focus on the one poll with bad news (Pew national in this case) while ignoring lots of good polls (Ohio +6, Virginia +5… etc)

  • Olav Grinde

    Interesting observation from Nate Silver at FiveThirtyEight.com today:

    “…even if the typical state poll is slightly less accurate the typical national poll, the collective sum of state polls may be more worthwhile than the collective sum of national polls. Also, the state polls come from a more diverse set of polling firms, and may provide for a greater degree of independence.

    Very much in support of Dr Wang’s approach — even if that wasn’t Mr Silver’s intent. :)

  • Ms. Jay Sheckley

    Film doesnt work for me. I’ve tried it when you posted it and since. And all i ever hear is baba, though from the first, it looked dubbed.

Leave a Comment