Princeton Election Consortium

Innovations in democracy since 2004

Outcome: Biden 306 EV (D+1.2% from toss-up), Senate 50 D (D+1.0%)
Nov 3 polls: Biden 342 EV (D+5.3%), Senate 50-55 D (D+3.9%), House control D+4.6%
Moneyball states: President AZ NE-2 NV, Senate MT ME AK, Legislatures KS TX NC

The cell phone effect: about 1 percent

September 25th, 2008, 5:27pm by Sam Wang

Improvements are in progress. Go to the interactive map and right-click a state. You will see options: access to original polling data, results from 2004, and the effects of adding support to your favorite candidate. More features will come.

And now for today’s question. How much has cell phone usage affected the reliability of polls? The answer may surprise you: Depending on what pollsters do about it, not much at all. Obama’s support may be understated by as little as 1%.

The question of whether polls have systematic errors is a continuing one. In the recent polling news is a Pew Center study that hits hard on the question of cell phone users. According to the survey, failing to survey people who have cell phones but no landline leads to a net underestimate of Obama’s support relative to McCain. According to a previous Pew/AP survey, cell-onlys comprised nearly 13% of households at the end of 2006. Cell-onlys prefer Obama over McCain by 18-19% (compared with an even split in the landline sample). Uncorrected, this leads to an error of about 0.13*0.185 = 2.4% in the Obama-McCain margin. Clearly this is significant, which is the Pew Center’s conclusion.

Pollsters have two strategies for dealing with this problem.

Strategy 1: Weighting.

National pollsters usually compensate for undersampled populations by weighting their results by age, race, marital status, and other factors. For example, landline registered voters aged 18-29 prefer Obama over McCain by 13% (52% to 39%; n=250). Since they are underrepresented in surveys, pollsters count their responses more heavily.

But there is a rub: cell-only registered voters don’t have the same sentiments as their landliner counterparts. Matched by age, they prefer Obama by a far larger margin of 35% (62% to 27%; n=146). That’s a discrepancy of 22%, which throws the weighting approach into question. What’s different about cell-onlys?

The Pew/AP survey is illuminating. It shows that of people aged 18-25, landliners are far more likely to live with parents than to rent (50% vs. 29%) than cell-onlys (19% vs. 57%). Cell-onlys are more likely to be in school, use Facebook/MySpace/email, and drink alcohol. Finally, cell-onlys are less likely to be married than landliners (8% vs. 15%).

This last factor helps pollsters. One of the greatest predictors of Obama/McCain support is marital status. Marrieds tend to favor McCain, and nonmarrieds favor Obama, for both men and women. The swing is huge, around 30%. Marital status is a much bigger correlate than gender! On this factor alone, pollsters who weight by marital status might get a correction of 30% * (15%-8%) = 2.1%. That would reduce the discrepancy to 20%. Still bad, but it’s an improvement.

Summary: With weighting, pollsters could conceivably get the systematic error arising from missed cell-onlys down to 1-2%. (The Pew survey itself puts the post-weighting systematic error at 2%.) It’s possible to imagine getting the error down even further with clever weighting on other demographic variables.

But why not try the more direct approach…

Strategy 2: Call cell phones.

This is an obvious solution that’s been avoided because autodialing technology to call cell phones is illegal, necessitating manual dialing, which costs more. But more pollsters are starting to do it, as reported by Mark Blumenthal.

Based on examination of organizations that include cell phones, Nate Silver estimates the cell-phone discrepancy as being rather large, 2.8%. His estimate has a standard deviation of 2.6% and an SEM of 1.0%. So I would take his estimate as being unreliable. Nonetheless, calling cell phones is a feature of polling is likely to improve matters, and is here to stay.

Conclusions (for now)

National polls that don’t call cell phones may understate the Obama-McCain margin by 1-2%. Some are on their way to overcoming this discrepancy by calling cell phones. Conservatively, I estimate that the overall error in national surveys is about 1%. It would be a mistake to assume anything larger. Obama supporters, the number of ponies here is small.

A big unanswered question is whether state polls perform weighting. Unfortunately, this is what we really need to know in order to gauge the effect of cell-onlys on the Meta-Analysis. Perhaps someone can enlighten me on this point?

Tags: 2008 Election

18 Comments so far ↓

  • hardheaded liberal

    If Gore had had enough “ponies” to equal 1%, we would not have suffered through the last eight years – at least not as miserably as we have.

    One % = 110% of your current “meta-margin.”

    Two % = 220% of your current “meta-margin.

    Nate Silver at has estimated that if one candidate beats the other by 4% in the national popular vote, it is virtually certain that the leader will also win the electoral vote, given the distribution of state poll results this year.

    In other words, if Pew’s estimate is correct, your model suggests that Obama would have a lock on the electoral vote if the election were held today.

    A lot can change between now and November 4, but at this point pollsters who omit calling cell phones are building an important systematic error into their results.

    Whether the error is 1% or 2%, I think you do your readers a bit of a disservice to discount this systematic error.

  • Ron E.

    Another question is whether the cell phone error varies by state. I could imagine that some states (with younger, less married overall populations) would have higher cell only populations. The error rate for not calling cell phones for state level polls for those states could be larger than your conservative 1% and could be the difference between a state appearing to be a toss up and actually being a lean Obama state.

  • Richard

    It is interesting to speculate and try to correct for systematic errors in polls. Problem is we tend to have limited knowledge about such errors, and we probably aren’t able to anticipate all of them. It is possible for systematic errors to cancel one another. Perhaps, the cell phone effect and a better than usual get-out-the-vote effort by Democrats will cancel the “Bradley” effect. But you well know the danger of trying to add in effects such as the trend for independents to break against the incumbent, an effect that didn’t materialize in 2004. I will start to feel good about this race when Obama has a large lead in the national polls and in the state-by-state polls. It is fun to make projections based on trends and events are trending Obama’s direction, but his cushion is still small and the EV race maddeningly tighter than the national polls.

  • BirdLives

    There is another systematic error in telephone polls, particularly the daily trackers. Proper sampling requires that the sample be divided into replicates, and each replicate has to be worked with at least 3 callbacks before moving to the next replicate. If you don’t do that, you get a bias to people who are at home in the evening. I don’t know this for certain, but I would expect them to skew toward McCain. Daily trackers by definition don’t have time to work the sample with this method. Why not include a cell phone sample to reach people who are not at home even if they also have a landline? That 2% could suddenly become 4-5%.

  • Sam Wang

    hardheaded liberal: Of course in a close race every vote is important. But I do not manufacture ponies – I just count them. What I am doing here is trying to arrive at a data-based estimate of the effect that people love to gossip about.

    Richard/BirdLives: your points are interesting but speculative. I especially contest the size and even the existence of the “Bradley effect.” If you have some actual documentation of this phenomenon that is not based on exit polls, I promise to make use of it in a future posting.

  • Michael Coppola


    If you dispute the existence of the Bradley effect, what is your take on the 1989 gubernatorial election in Virginia? Doug Wilder had a 4-15 point lead in all the pre-election polls, but won by >0.5%.

  • William

    According to the Bradley effect hasn’t been seen since the mid-90s.

    In fact, in the 2008 primaries, Southern states saw a “Reverse Bradley Effect”. We don’t yet know if it will carry over to the general though.

  • Sam Wang

    Michael Coppola, that is a well-known anecdote told to support the argument. For this reason it’s sometimes called the Bradley/Wilder effect.

    However, you are guilty of selectively accepting one piece of evidence that agrees with your position. This is called biased assimilation. There have been African-American candidates besides Wilder. What happened in those cases? Is there a systematic survey of the evidence?

    A contrary bit of evidence is that Obama is said to have outperformed polls in his primary contests. Does this suggest that there is an anti-Bradley effect? Nate Silver has suggested as much, and is now assigning undecided African-American voters to Obama. I consider this to be an error as well.

    To reiterate, I am looking for evidence of an unbiased nature. I am certainly interested in getting to the bottom of the subject. I am open to the whole range of options – Bradley, no effect, anti-Bradley. But the existing lore does not meet my standard of proof.

    That having been said, if you have articles with actual data to send me, including documentation of the Wilder case, by all means do so.

  • Richard

    I don’t know that the Bradley effect will occur in this election, even if it has occurred in previous elections. My point is there may be various systematic effects that could throw the poll results off. These effects might turn out to reinforce or cancel one another out. The validity of your “meta-analysis” rests on the typical statistical assumptions that the polls are “independent events” and there are no systematic errors. I don’t think you can account for systematic effects much better than what you already do, which is to bracket your results with +-2%. If systematic effects break one way or another by this amount, we can see what happens to the initial result that didn’t assume such effects. As you say, you don’t have much confidence in Nate Silver’s estimate of the cell phone effect. But I agree it is fun to gossip about these possibilities.

  • Michael Coppola

    you are guilty of selectively accepting one piece of evidence that agrees with your position

    Actually, I wasn’t taking a position, just asking a question. Having lived through the Wilder election, I can tell you it certainly seemed like something strange was going on. You have a lt. gov. in an overwhelmingly popular and successful administration running for gov. He never trails in any published poll. Leads by more than MOE in most of them. Is called the winner early on election night by all three networks based on exit polling that predicts a 10-point win. Election ends up in re-count and final margin is ~6,500 votes.

    I suppose that’s all anecdotal in a way, and I don’t really know where to track down the internals of those polls almost twenty years later. But it’s something I’d think should be studied rather than dismissed.

  • William

    Michael, no offense, but it has been studied: look at my earlier post.

  • Sam Wang

    Everyone with an interest in Bradley/Wilder, let’s please stick with actual evidence. Michael, I’m sorry I was a little tart with you.

    Here is an example of evidence, though still only a single case. In a New York City mayoral primary between David Dinkins and Ed Koch, the Dinkins led by 0 to 5% in the weeks before the election, but ended up winning by 8%. See pages 487 and 490 of this PDF.

  • Sam Wang

    I just read the paper William provided. It’s quite good. Thank you! I have great readers.

  • Michael Coppola

    No offense taken, William. Thanks for the article. Data is always a good thing. Sorry I missed it the first time. Of course, now I have to read it, and Sam’s link too.

  • Jeff Rogerson

    I’ve been wondering for a while about how the huge growth in cell phone use would affect these national polls. I know for me and most of my friends, a land line doesn’t even come into the picture. I wouldn’t be surprised if the error margin WAS more than 1%…

  • Ralph Reinhold

    @BirdLives: I just looked at the coverage map of some blue areas in Iowa, Colorado, Ohio, and Wisconsin. While these are areas with sparse population, they are lacking in coverage.

  • Ralph Reinhold

    The landline usage statistics that I can find are over a year old. In 2010, 26 percent had opted out of landline. The phone companies say they are loosing about 10% of their customers per year. I don’t know how you would relate those two numbers, but it would seem that roughly half rely on cell phone only.

    Everyone that I know looks at their caller ID and, if it is not someone they recognize, let it ring (or cancel the call). I would be willing to bet a quarter or two that cell phone responses are much less than the 9% they’re getting with the landline calls.

    However, Pew disagrees with my conjecture:

  • nospam

    Personally I get so many junk (spam) calls on my landline that I automatically hang up on unsolicited calls- and that would include pollsters. I can’t believe that I am at all unusual in this. So until I see a serious accounting for the spam-call rejecting segment, I’m going to heavily discount all phone-poll based predictions.

Leave a Comment