Princeton Election Consortium

A first draft of electoral history. Since 2004

The Virtues of the L.A. Times Poll

October 19th, 2016, 9:07am by Sam Wang


Yesterday I visited a journalism class. The question arose of how to interpret the L.A. Times/Dornsife/USC poll, which has been unusually favorable to Donald Trump. I said that polls should be treated the way reporters treat other sources of information: get confirmation from a second source. In the case of polls, find two other sources and take the median. The reason is that a polling result is not a pure number descended from heaven; it reflect the professional judgment of one pollster…and such judgments can vary.

Because the Dornsife/USC poll is an outlier compared to other surveys, its other impressive qualities are often overlooked. Let me get into those a little bit.

The Dornsife survey tracks the same respondents over people repeatedly in time, which reduces swings due to respondent bias. This is a great help, since nonresponder bias is often claimed to create false swings in the polls.

Dornsife’s unusual result appears to arise from weighting that is more detailed than what other pollsters do. This “microweighting” has been claimed to give unrepresentative results, for example because of one 19-year-old respondent in Chicago who represents his entire demographic. For the record, the L.A. Times people do not think this is such a big problem.

Luckily, Dornsife has done something remarkable: they have made their detailed response data publicly available. Using this data, Ernie Tedeschi has used Census/American Community Survey demographics to reweight the responses. Perhaps unsurprisingly, the resulting output more closely resembles polling averages. See the graph above.

None of this is to criticize the Dornsife methods. In fact, those methods provide valuable evidence that the small swings in the race this year are in fact real. I think it is one of the more interesting surveys this year. I just don’t think it means that Trump has ever been ahead in the general election, an idea that is contradicted by other polling evidence.

As readers know, I do not think it is appropriate to second-guess pollster methods. It is too easy to only do so when results are disagreeable. For this reason, here at PEC I report poll medians based on data reported by pollsters. The assumption is to respect their judgment as a whole. In this case, I think that Tedeschi’s analysis is interesting because it gives a salient case study in the judgments that all pollsters must make.

Tags: 2016 Election · President

54 Comments so far ↓

  • Bert

    Y’all are being much too intellectual about this poll.

    The importance of this poll has nothing to do with its veracity, but its use as an instrument to willfully undermine and invalidate the upcoming election.

    It is pure toxic waste.

    If his supporters and Trump go quietly into the good night, I will retract my comment, but y’all are missing a much bigger picture, and it’s a picture that paints a thousand words, none of them very good for America.

  • Lorem

    I’m not convinced that this poll does much to eliminate nonresponder bias.

    As far as I can tell, they ask their panel for responses once a week, and then some portion of that panel actually responds. Seems like exactly the same potential non-response issues are present as with asking a random sample. At best these issues may be somewhat less severe.

    • Commentor

      And according to the Cohn analysis, the result of the poll has fluctuated at times based on whether the single young black man has participated that week.

      I don’t see how a poll in which one respondent’s participation can push the result that much can be said to have “nailed it” if by chance its right.

  • Erik

    Imagine the day before the election RCP is Hillary +6, LA Times is Trump +1, and Trump wins. Every other pollster would either have to stop polling for the presidency (like Gallup) or emulate the LA Times methodology. Not a Trump fan, but I would love for that to happen.

    • Joel

      Assume the opposite happens; Clinton wins by +13; those pollsters would be similarly wrong but *there would be no pressure to fix their models* from the general public.

  • Hem13

    I ask the following with no criticism for including the L. A. Times poll in this election aggregate, but to comprehend future PEC policy and procedure.

    Will PEC never cease the inclusion of any pollster, even if that pollster is found to be employing an egregious methodology?
    Will these be among the essential reasons for the continued inclusion of every pollster:

    1. It was included from the beginning.
    2. The pollster is more transparent about the specifics of its methodology than other pollsters.
    3. The trendline of the pollster’s metrics is useful.
    4. The internals are interesting.
    5. The pollster’s inaccurate snapshot of the electorate will be compensated by other pollsters employing a sound methodology.

    • Joel

      PEC just takes the poll feed from HuffPo (formerly Pollster). They have screened bad pollsters in the past, and Mark Blumenthal ran a tight ship there. I think he’s gone to work for one of the pollster though, so I don’t know who is in charge now.

  • AAF

    The problem with including the LA Times poll in a polling average is in including it more than once.

    Most tracking polls add in the most recent days’ respondents, and roll off the oldest ones, whether it’s for a 3-day rolling average or a week. Polling averages only include such polls as a new poll each time a day’s results include an entirely different set of respondents – either once every three days or once a week, in the examples above, so that the same respondents don’t get triple or septuple counted.

    By including every instance of the LA Times poll, a polling average radically overweights the LA Times’ set of respondents as they get included over and over again. I would estimate that these individuals are overweighted by a factor of 90 or more in any polling average that includes each instance of the poll.

    • Stephen Hartley

      Sam is using the median not the mean of the polls I think and I’d be surprised if his results don’t contain some normalization factors to account for different data rates

    • Sam Wang

      I don’t use national polls at all.

      The rule for integrating state polls is described elsewhere on this site: median, last N days, no duplicate entries from the same pollster. N is variable depending on how close the election is.

    • AAF

      Sam doesn’t use national polls. I meant Huffington Post, RCP, and anyone else that provides national polling averages.

  • JamesE

    I lurk at right wing blogs pretending that I’m a cultural anthropologist visiting an alien culture!
    One that I visit frequently is freerepublic.com, where the Birther movement really originated. At that web site, they question every poll that is bad for Trump by claiming that its “internals” always show that the poll has oversampled Democrats.
    Today they are challenging Reuters/Ipsos saying their sample was 40% Democrats, 26% Republicans and 34% “other.”

    • BrianTH

      That’s the good old “unskewing”.

      Recently some of them have taken to looking at percentages in the unweighted sample, or reporting something like D 30 R25 as “D+20″ (because 30 is 120% of 25).

  • Amitabh Lath

    Would the students in the journalism class be asking about the LAT poll if it had Clinton +6? I suspect not. Finding things “wrong” with this poll (it’s a panel, it’s got microweights, it’s RAND-like…) is impermissible. The only loophole is if one had decided to subject every pollster to equivalent scrutiny before looking at their results.

    Personally I love that the LAT/USC folks are willing to innovate public opinion estimation by using non-random sampling. They are probing an entirely different set of systematic uncertainties than traditional polling houses.

    • Joel

      The analogous example would be Clinton +13 or so. I think that people would question their modeling methods, although it wouldn’t be the same subset of people.

    • Commentor

      The question in my mind is whether such a poll is apples to apples to other polls using randomly selected respondents each time. Whether it is valid is not the question, but whether it should be averaged with other polls which are fundamentally different.

    • Commentor

      As an additional comment, if you were performing a meta-analysis of pharmaceutical trials, you’d probably want all of the studies to be of a similar type, i.e. all randomized controlled trials. My question is whether there is a mathematical argument that such considerations should be considered in averaging polls.

    • Ryan

      I’m far from an expert, but from what I understand the methodology of the LAT/USC poll is sound, they just got unlucky in that they picked a sample that wasn’t exactly representative of the population. Plus, with the way the poll works, sticking with the same group throughout the election, they got stuck with the outlier-type results.

    • Jeremiah

      @Ryan How do you know they picked an unrepresentative sample? Surely this is the same problem that “traditional” pollsters face in that they have to weight their poll to represent the voting population. Everybody still has that problem to a larger or lesser degree these days.

    • Ed Wittens Cat

      Does anyone know if the LAT poll maintains their respondent pool by paying for responses like RAND did?
      it seems like the only way to maintain their panel week to week.
      FYI Sarah Palin will attend the debate. Its like a freakshow of every quasi-conservative aberration starring Trump as a sideshow barker! Maybe Milo Yianopoulis will attend!
      and then Trump will bite the head off a live chicken on stage as a finale…so embarrassed for my country.

    • Amitabh Lath

      Commentor, a decision on which polls to include/reject has to be made before the aggregation engine is started. Chucking polls out in the middle of the game (no matter the justification) is about the most biased thing I can think of.

      I’m not big on taxonomy but my opinion is that the LA/USC poll is also an apple, but it’s one of the first granny smiths we have seen in the bushel of Fujis and Macintoshes.

    • Commentor

      Amitabh Lath, to use my analogy again, if you were performing a meta analysis of randomized clinical trials of a particular drug, and while in the process found that the selection of patients was not random for a particular study, wouldn’t it be okay to exclude that study from your meta-analysis? The key in my mind is to have a set of criteria to select polls that you stick with, not a list of polls that you stick with.

  • Greg

    Right, so PEC (and numerous others) have established that the LA Times / Dornsife has extremely useful qualities, and employs methods that may gain new insights into public opinion over the course of an election. Cool!

    The issue seems to be that the vast majority of the public fixates on what appears to be the least useful qualities / insights of the survey: its accuracy as a measure of the current state of national support for either Trump or Clinton.

    In my opinion it is unreasonable to assume that most people will take it upon themselves to investigate deeper into the L.A. Times results. Instead the outlying results are being seized upon like so many bales of oats to fuel the horserace (narrative). So I think the L.A. Times should adjust the nomenclature to adjust the expectations, and stop presenting this survey as if it was a finished product, comparable to other national pollsters.

    Currently, the L.A. Times stands by their poll as a literal forecast of the 2016 election: “Obviously, the poll’s results have been an outlier compared with other surveys, but if ever there was a year when the outlier might be right, it’s this year.”

    It’s their right to support their own experiment, and it’s everyone else’s right to be skeptical.

    • Michael Smith

      I mainly agree. But you gotta admit, if they called it right, they will have something to crow about. I think that’s really unlikely, though. Almost everybody else would have to be very wrong… tremendously unlikely.

    • Shawn Huckaby

      Well said. Once you start down the “this time is different” rationalization path, you can always find a justification for your claim.

      This cycle seems to be, “I know we always say ‘this time is different’, but this time is REALLY different!”.

    • Joel

      I think this is a great example of precision versus accuracy.

      At the end of the day, polling consumers don’t give a damn about precision. But they care a lot about accuracy, especially when a swing in results is involved.

      Obviously, accuracy is more important, but precision matters, too.

    • Paul Weeldreyer

      The LA Times poll was also an outlier poll towards Obama in 2012 and ended up being much closer to correct than the other major polls. So don’t be too quick to assume that the LAT poll is wrong and all the others are right.

    • Sam Wang

      No, that’s not what is going on here. The best estimate that escapes your personal biases is the median. Since this poll is quite a distance away from the median, it is not crazy to wonder why.

      Anyway, national polls in the aggregate don’t do as well as state polls. And state polls are quite clear at this point. I will say that if this poll ends up being closest, I will eat a bug. A big one.

  • LorinK

    I have read recently that the LA Times sample was weighted using retrospective reports of 2012 Presidential election voting choices to replicate the actual vote split in that election; the article explained that this is problematic because of biased recall–some people “remember” voting for the winner even though they did not.

  • Commentor

    Although the LA Times poll does provide useful information, should its results be included in poll averages if its respondents are not selected randomly?

    Other polls have the benefit both of their respective weighing systems and that particularly unrepresentative samples aren’t used more than once. Averaging is our answer to differences in weighting, but does it address a repeated unrepresentative sample?

    • Harry Schwarz

      It might be an interesting poll but aggregators who average polls use it in their calculations. Only Nate Silver actually adjusts the poll. So, if averaged in with other polls it actually creates a false average, lowering the poll average. RCP acts as though it is just another poll. It is not.

    • BrianTH

      So originally my inclination was no, don’t include it in the averages. Panel polls are different from random sample polls, and this is possibility of being stuck with a somewhat nonrepresentative panel is a known problem with panel polls. But panel polls are useful for other purposes and therefore we aren’t just throwing away the data, but using it the right way.

      However, over time it became clear to me that despite all this, the LA Times really did at least kinda sorta want us to treat it as comparable to random sample polls. And that raises Sam’s issue about not substituting your own judgment for the judgment of pollsters.

      That said, I’m still not 100% clear what the LA Times really believes about this poll and how it should be used.

      Fortunately, it is just one poll in a well-polled area, so ultimately it doesn’t much matter whether it is in or out.

    • Scott J. Tepper

      I’ve had correspondence with the reporters who write about the poll. Their responses to me were very defensive. Their position in the correspondence has been “Maybe everybody else is an outlier.”

      I’ve objected to the headlines they use to characterize their poll once it became obvious the 3000+ initially-selected participants were heavily Trump. (Ask yourself: who is more likely to agree to be questioned repeatedly, for months on end: somebody who works or goes to school or someone who is retired or not working?) The response to that has been that everybody knows that their descriptions are off. Therefore, they don’t have to change the headlines which claim to show where the election is today.

    • Matt McIrvin

      “However, over time it became clear to me that despite all this, the LA Times really did at least kinda sorta want us to treat it as comparable to random sample polls. And that raises Sam’s issue about not substituting your own judgment for the judgment of pollsters.”

      I’ve always figured that only applies up to a point. I mean, I could create a poll of my own that was constructed by a sample of tic-tac-toe-playing chickens, and it probably wouldn’t do anything good if somebody threw it into a poll aggregator. What do you do if the judgment of the poll provider is based on manifestly erroneous theory?

    • Some Body

      @Matt McIrwin: Same goes for fake polls, and it’s even much worse with pollster herding.

      But, ay, there’s the rub: so long as you’re not submitting all pollsters to rigorous quality and method control (while ensuring your criteria are not so strict that you could potentially end up pre-determining outcomes), any qualms about any individual pollster just introduces bias (as Amitabh very clearly noted above). You’re not equally likely to suspect a pollster’s methods regardless of the numbers they are putting out.

    • Matt McIrvin

      See, I think I would have objected to including the 2012 RAND poll in a poll aggregate, even though, as it happened, it had Obama ahead by a reasonable amount and its absolute bias was not far off at all. It was clearly not the right kind of poll for that purpose.

    • BrianTH

      Whatever you do, I think it is important to make the decision using pre-set criteria, meaning before knowing what a potentially affected poll might be saying.

      So I’ve got zero problem with people who had pre-set criteria that would rule out panel polls. The problem is if you didn’t have that issue covered in advance and now you know what this poll says . . . .

  • Sifcell

    Perhaps the one thing that needs to be explicitly restated (though it is often implied) is that since the LA Times poll tracks the same respondents over time, any bias in the original sample – say an over-representation of Trump support – will be conserved over the life of the poll.

    In that case the interest is less in how well the LA Time poll agrees with other polls than in how the LA Times poll numbers shift over time. And in this case the LA Times poll does, indeed show similar loss of support for Trump after the conventions, after the first debate, and after the Access Hollywood tape.

    The fact that the poll continues to “favor” Trump is less important than the fact that it reflects the erosion of Trump support in much the same way as the other polls.

  • James Moore

    I would think there would be an interesting risk of survivor bias here. How many people want to keep answering the same question over and over? Seems like the pool would select for only keeping serious political junkies around. Normal humans would just tell the pollster to go away and stop annoying them.

    • Matt McIrvin

      What I’ve wondered about the RAND poll that used a similar technique was whether there would be an effect on the respondents themselves that came from them no longer being naive subjects. Does being asked about politics all the time make you a political junkie? Does that change your opinions?

      The correspondence of the re-weighted LA Times poll with the RCP national averages suggests to me that it doesn’t have a huge effect.

    • Virgilio

      In a given day, only 1/7 of the sub-population participate, so that a given participant is only asked once a week, and only a few questions. They say:
      “Every day, we invite one-seventh of the members of the UAS election panel to answer three predictive questions: What is the percent chance that… (1) you will vote in the presidential election? (2) you will vote for Clinton, Trump, or someone else? and (3) Clinton, Trump or someone else will win? As their answers come in, we update the charts daily (just after midnight) with an average of all of the prior week’s responses. To find out more about what lies behind the vote, each week we also ask respondents one or two extra questions about their preferences and values”.
      http://cesrusc.org/election/

  • Bill

    I think the poll is interesting. I believe an opportunity with the poll is explained in this interview with David Lauder of the LA Times.

    My takeaway was that they recognize the potential risk in not asking a binary question and weighting by intensity. If I’m hearing his explanation right, a 100-0 response is given twice the weighting as a 51-49 response. Take a look and tell me if I’m hearing this wrong.

    I do love their transparency and willingness to discuss the reasons for making the choices they made.

    https://youtu.be/K8kpRh_CSyo

    • Bill

      Clarification- I wasn’t criticizing the poll. Mr. Lauder from the LA Times seemed to be suggesting that their weighting by intensity was having an impact. He didn’t discuss the demographic micro weighting.

    • Roger

      That is how I read it. The pollster should explain that.

      Also, If it does weight be a vote by 0-100 scale, I think it would make difference between educated and uneducated people. My premise is that I think uneducated people would be more likely to put 0 or 100 with few putting anything between. Whereas, I think more educated people might actually utilize the full 0-100 scale more in their answers. This is my opinion because I think uneducated look at the world in a more simple yes/no way. While a educated person would look at it in a more complex way. So an enthusiastic uneducated person would be more likely to put 100 on his chance to vote, whereas an equally enthusiastic educated person might put 75. In their eyes. their intent to vote is the same, but under the methodology of that poll the 100 is going to have 25% more weight.

  • Michael

    Sean Trende at RCP wrote that the LA Times poll is essentially the Rand poll from 2012, saying the following: “In the end, though, the RAND poll basically got it right. The national polls (though not so much the state polls) were off in 2012.” Is more explanation needed?
    http://www.realclearpolitics.com/articles/2016/10/14/why_pay_attention_to_the_la_times_poll.html

    • Sam Wang

      No, the comparison to RAND sounds like motivated reasoning. As I said, so much depends on weighting. He is correct that state polls did very well, which is why PEC uses state polls.

    • BrianTH

      So even assuming that was true, that doesn’t really address the issue. With a panel poll, you might get lucky if your panel is close to representative. But you might get unlucky if it is not, and it could be off in either direction.

      So if you have one prior instance and your panel that time happened to be close to representative, that tells you nothing in particular about whether you got lucky with your panel again.

    • Ed Wittens Cat

      i may not remember this correctly…but didnt the original RAND study pay the respondents?
      and Sean Trende was a big missing-white-voter guy in 2012 and picked some hyuuge fights with Silver.

    • Virgilio

      Hi, Sam and all:
      In Trende’s article mentioned above he brings up an interesting point. The poll keeps on sub-sampling the same “sub population” of voters. Those in the sub-population know they are part of a experiment, in which periodically they are asked about their political preferences. The initial sub-population may be perfectly representative of the greater voting population. BUT the fact that those in the experiment know they are periodically polled probably induces them to be more “curious” about the political campaign, to read more, follow certain Internet sites more regularly, etc. and in essence become better informed and more sophisticated than the average Joe. This in principle is BAD because that means the poll results diverge from those that would be obtained by sampling the greater population.
      Nevertheless, Trende makes clear that last time around this poll (as RAND’s) did extremely well on election night, although it seemed to yield “outlying” results through the campaign (as may be happening this cycle).
      In several twitter comments (https://twitter.com/SeanTrende/status/786911929503711232 ) I argue that there is a reasonable theory that explains why this poll produces outlying results through the campaign yet does extremely well on election night. The argument is that as election day approaches, the typical voter starts focusing more on the race, reading more, doing more Internet searches, etc, and by election day more or less “catches up” with the level of sophistication of the sub-population being sampled by the the USC poll. In other words, the typical voters does over the finals days of the campaign, what those participating in the experiment do much earlier. Hence, this poll may NOT be a good reflection of voter’s opinion today, yet it may still be a great predictor for what will happen on election night.
      What do you all think?

  • Ken Arnold

    Sam,
    Do leads tend to exhibit (1) reversion to the mean, (2) momentum, (3) or just a pure random walk? HRC has climbed to +7 after being stuck in a range from +2 to +5 for most of the season. Is it more likely that she will fall back into that range or continue to climb?
    Thanks for your terrific site. It is so rare to find a forum for civil and erudite discussion.

    • BrianTH

      How about none of the above?

      Polls are trying to measure a real world thing (public sentiment), and public sentiment can be changing for reasons not in any way captured in the polling history.

      Now to be sure, when you are aggregating lots of polls with some random error, you would expect to see something like a random walk around the actual value. But again, that actual value could be trending for actual reasons, and any sort of trend is possible depending on what is actually happening in the world.

    • alurin

      And yet reversion to the mean is a real thing. We intrinsically want to ascribe a cause to every shift in the polls, but I’m not sure that’s true. Consider: Hillary’s lead was already trending upward before Debate #1, so it is impossible to attribute her late September recovery entirely to her debate performance (or T—p’s meltdown). Conversely, L’Affair Access Hollywood did not lead to a noticeably change in the polls… despite an unprecedented collapse in his support from prominent members of his own party, and the apparent meltdown of his campaign apparatus.

    • BrianTH

      There is definitely random noise in the polls, and sometimes what people are describing by “reversion to the mean” is just that if you look backward and calculate a smoothed trend of some sort, you can see what was probably just random variation around a trend.

      On the other hand, just because we may not be able to provably identify exactly why public sentiment moves around the way it does, that doesn’t mean it isn’t really moving around, or for real reasons. And there are some real things that are pretty easy to detect–the conventions, for example, pretty reliably create a back and forth effect, although the magnitude and net outcome may vary.

    • Matt McIrvin

      Yes, that’s what “regression to the mean” literally means in statistics: if you have a variable that has some random component, extreme values will probably be followed by less extreme ones, because some of that identification of extremes is just selection applied to random noise.

      (I think the term was introduced by Galton, and he was originally comparing statistics such as, say, height between parents and their children. Taller parents will tend to have taller children, but the curve correlating them has a slope less than 1, because some of the variation is just random and not heritable. “Linear regression” is called that because it was the mathematical technique he was using.)