Princeton Election Consortium

Innovations in democracy since 2004

Outcome: Biden 306 EV (D+1.2% from toss-up), Senate 50 D (D+1.0%)
Nov 3 polls: Biden 342 EV (D+5.3%), Senate 50-55 D (D+3.9%), House control D+4.6%
Moneyball states: President AZ NE-2 NV, Senate MT ME AK, Legislatures KS TX NC

The paint continues to dry; VP debate comment thread

October 4th, 2016, 9:28pm by Sam Wang

It does not seem likely that much change in the race will come from this debate. However, there is some chance that viewers will learn about the candidates’ positions on issues. Weird, huh?

Since reaching high-water marks for Republicans in the third week of September, three measures have moved in the same direction:

  • the generic Congressional ballot, by 3.0% toward Democrats, to D+5.0%;
  • the Senate Meta-Margin, by 2.0% toward Democrats, to D+0.8%; and
  • the Clinton-over-Trump Meta-Margin by 1.2%, to Clinton +3.0%.

Only the generic Congressional ballot is composed entirely of post-first-debate data – and it shows the largest change. That suggests that in the coming week, Presidential state polls and Senate polls may continue to move toward Democrats.

These changes are in the direction that I suggested a week ago would occur, a regression toward the mean- though I wasn’t expecting quite such a strong win by Clinton in the first debate.

However, the changes are also not very large. They represent a continuation of the big untold story of the 2016 race: low volatility, in what is the most stable national race in 65 years of polling. This stability is a major contributor to the high Presidential win probability calculated here at the Princeton Election Consortium.

Tags: 2016 Election · House · President · Senate

123 Comments so far ↓

  • Anthony Shanks

    This is definitely not a boring debate. Full of policy (from Kaine) and policy denial (from Pence). His debate strategy (besides the interruptions) is solid. Force Pence into defending undefendable positions all night.

  • Ed Wittens Cat

    Battle of the Blands
    cant wait to see the SNL skit on saturday
    SNL powerfully affects presidential elections i think…its like our cultural bible

    • Ed Wittens Cat

      Argental Satan:
      “Nate Silver 10:10 PM
      Another thought: I think Pence is doing a better job of executing his strategy, which is basically just to pirouette around any time he’d have to defend Trump. Whereas Kaine is swinging at a lot of pitches but not really following through. So judged on the basis, I’d give it to Pence, so far.

      But I’m not sure that it’s a great strategy, because I think Pence’s nonresponsiveness could make for some good commercials for Clinton’s team in Brooklyn after the debate. So Pence could be setting himself up to win the actual debate but lose the post-debate debate.”
      What its going to “make for” is a truly epic SNL skit

    • Sam Wang

      How sad to see a numbers guy descend into play-by-play commentary over an event that he knows makes no difference.

    • Ed Wittens Cat

      yup–Dr Wang is correct– i dont think the needle moves–
      and the devolution of Silver from quant to pundit is just sad– i think Trump’s unimaginable (to Silver & the pundit class) rise has broken his nerve.
      i was surprised to see Pence even try the “There you go again” oneliner– if Kaine had been a little more nimble he could have responded with “you sir, are no Ronald Reagan”
      but it still fell flat because the audience wasnt allowed to make any noise.
      to me it was a kangaroo slap fight btwn two grampaws– full of sound and fury, signifying nothing– and they were very rude to the moderator

  • Greg Gross

    I agree with Anthony Shanks. Kaine repeatedly pointed out policy and made it clear that Pence’s “spinning” (per Nate S) was occurring only because Pence had no good way of defending Mussolini in an orange clown wig. Pence was poised, but Kaine was more energetic, persuasive on policy, and on pointing out Pence’s inability to defend his boss.

  • Hem13

    Thanks for these insights regarding Veep Debate. It’s unfathomable why Nate Silver would consider himself remotely qualified to adjudicate a political debate when he’s a stats expert, not a policy or communication/ public relations expert.

    • josh f

      NS signal+noise book was precursor to espn and present-day 538- opine on anything and everything, vs old 538 where they focused in-depth on polling and occasionally sports. A significant portion of present-day 538 is not even data-driven.

    • Andrew

      As a potential explanation, 538 is not a “hobby” site like this and needs visitors to pay the bills. My guess is that a boring, stable polling environment is death for them and they do better with lots of volatility and unskewing going on. Hence the play-by-play, headlines about what needs to happen for Trump to win, etc. (Of course Prof. Wang’s page views were up this year over past ones IIRC.).

  • Larry Guy

    So, a validation that regression to the mean can indeed occur in a low volatility environment. It is a relief. The model works!

  • Sean Patrick Santos

    The effect on polls might be dull, but the debate wasn’t. Kaine was a bit too aggressive at times, especially over tax returns, but he seemed to have a big advantage over Pence (pointing out the obvious – Pence doesn’t actually want to defend Trump’s many outrageous statements). If this debate had any effect at all, it probably helped Clinton.

    I was a little surprised over the heavy foreign policy focus. I missed a few minutes of the debate, but I heard little about trade and nothing about climate change, both of which are key international cooperation issues that depend heavily on the executive branch to negotiate.

    • Sean Patrick Santos

      I feel obliged to clarify that I don’t mean that “Kaine” won, as a brand. Rather, I don’t feel that Clinton lost here. It is entirely plausible to me that Kaine “lost” by gaining less respect than Pence, while Clinton won in the sense of being more respectable than Trump. I don’t know if that counts as winning or losing the debate, but who cares?

  • DonC

    Can’t see this moving any votes. Pence’s strategy was to point out the failures of the Obama Administration and to blame Clinton for these. Basically an extension of Trump’s main line of attack.

    Kaine’s strategy was to bring up Trump’s positions/statements, forcing Pence to either deny the statements were made or positions taken, or to disagree with them.

    Pence ended up denying that Trump said many of the things he said, which opens the doors to a few days of “fact checking” and tape replays, both of which keep the focus on issues that Trump likely wants forgotten. Consequently, regardless of the tactical outcome, the debate seems to have been a strategic success for Clinton.

    Completely OT: Changing the fundraising goal seems to have really moved the goalposts! That’s what success can do.

    • Sam Wang

      ActBlue: I tried to set it to $916,000,000, but it wouldn’t let me. So I settled on a more manageable number.

    • WildIrish

      Professor Wong! OMG, that’s funny. I nearly choked on my tea. You are wicked. :D

    • Olav Grinde

      Professor Wang is doing is utmost to have PEC match the Koch brothers!

  • Allan

    This was a headache of a debate.

    Pence seems to be under the impression that no one’s going to fact check him on all the denial he did tonight.

  • Matt McIrvin

    I never watch these things; I watch other people’s reaction to them. Based on that, it sounds as if Pence “won” in the purely stylistic sense in which people traditionally lose or win TV-era political debates, but it may not actually help his ticket much, to the extent that these things ever do.

    And if people go around talking about how much better a debater Pence is than Trump, it might get Trump’s goat again…

  • Jeremiah

    Pence’s performance was mainly style over substance. Kaine was fairly aggressive but he kept painting Pence into a corner on Trump’s policies and statements. There were too many waffling answers from Pence when he realized he had nothing to hit back with.

  • JPI

    The abortion discussion near the end was striking as both candidates were quite honest in their views on the issue. Not the usual obfuscation that hopes to avoid alienating half of the electorate.

    • linda

      True. Pence showing he is a true christian conservative, speaking to the base and not trying to appeal to the middle at all. Suggests he is aware they are highly unlikely to win.

      In contrast, Kaine’s position (as genuinely held as Pence’s) fortuitously pleases both the Democratic base as well as a healthy chunk of the middle.

  • A

    Have been heartened to see the meta-margin moving in this direction, and to see the very positive uptick in the probability of a Clinton win.

    It’s still hard for me to fathom how Clinton’s total dismantling of Trump in that first debate didn’t move the needle more. I understand it’s due to polarization, but it’s still mind-boggling to me.

    But we take what we can get.

    I wonder, if we factor in Clinton’s ground game advantage, which is a slight unknown, how much more likely would her winning probability be?

    Seems to me that based on what we know, she might just get 1-3% advantage, based on what I’ve heard analysts give to a ground game advantage.

    Since we just look at polls, I suppose the ground game factor isn’t necessary to make a concise prediction…but I am consoling myself that perhaps the real number right now is closer to 98% when you factor in ground game advantage…

    • Sam Wang

      I take a dim view of assertions about “ground game.” However, the Clinton+2% link in the right sidebar can show you how that would turn out.

    • Michael

      Sam is certainly the expert here, but I think it’s important to note that in 2012, Obama beat virtually all of the aggregators’ final prediction numbers, including PEC’s, and by a not insignificant margin. I think that turning people out has to matter some on the margins.

    • Sam Wang

      Your statement is not true.

    • Josh

      I’ve posted this elsewhere, but in having spoken to numerous vets from the vaunted Obama GOTV operations in 2008 and 2012, a robust “ground game” can be good for an extra 2-3% increase in turnout…in the states where it is successfully implemented.

      There are only about a dozen truly competitive states. I’ll leave the math to you, but the idea that the disparity in GOTV between Clinton and Trump’s campaigns could result in a major increase in the national popular vote MoV for Clinton is probably unrealistic.

    • Matt McIrvin

      The thing about ground game is that Trump also had that disadvantage in the primaries, and it didn’t seem to hurt him that much. However, the primaries were less close in the first place, so whatever small advantage you get from good GOTV organization may have been swamped by the margin. I recall that Ted Cruz did seem to be consistently outperforming his polling for a while, and better organization might have had something to do with that.

    • Matt McIrvin

      Keep in mind, too, that the PEC model won’t fully reflect the post-first-debate period until there are enough recent polls in all the relevant states that they push the older ones out of the median. We’re still getting some new polls whose survey periods go back to before the debate (like UPI/CVOTER, whose results were dramatically better for Trump across the board than the polls that are wholly post-debate).

    • JeffE

      Would love to be proven wrong, but I think the “ground game” is essentially baked into the polls. Sure, there is some amount of just turning out your voters on election day, but it also means knocking on doors and making contact with and registering voters BEFORE election day, thus creating “likely voters”. Those people maybe weren’t “likely voters” before but become “likely voters” after they’ve been contacted, etc.

  • Brad

    My apologies for the potentially ignorant question, but can someone provide a link to the 2012 Meta-Margin? Thanks.

    • Colin

      On the top left-margin of the page there is a link to the archive of the blog, going back to 2004.

    • Michael

      Sam, am I mistaken, or is the final meta-margin number for 2012 not a popular vote prediction? If I’m incorrect about this, then ok. But if that is a prediction of popular vote percentages, then I don’t see how I’m incorrect. Maybe I should have clarified that I was speaking about popular vote only.

    • Slartibartfast


      As I understand it, the final meta-margin number is exactly what it says on the tin: the predicted state of the race after the last pre-election poll is in (i.e. on election day). While it will probably be similar to the final margin, it is a prediction of the size of swing necessary to make the election a coin flip.

      You can think of it this way: if all of the strongly Democratic states became 100% Democratic, it would have a big effect on the popular vote margin while having little impact on the meta-margin.

      On the other hand, Sam’s state-by-state margins should be close to the statewide popular vote margins—at least in the battleground states where there was a lot of polling done. I think this is one of the strengths of Sam’s methodology—boiling all the data down to one number that encapsulates the current state of the electorate. Because of the winner-take-all nature of the Electoral College, the popular vote margin isn’t necessarily correlated with the EC margin very well and thus lacks the power to predict the outcome.

      On the other hand, you can always use Sam’s methodology to predict the final margin: look at the most recent national polls and take the median.

  • Michael

    Is there a historical example of some event in the final month of a campaign that has changed a “settling” race of the kind we see now enough to cause the front runner to lose?

  • Melissa

    Is climate change and environment ever going to be brought forward?

    • Fritz

      Unfortunately no. We have to hope someone from the town hall debate will ask a question, but if MSM vetts the questions beforehand, we’ll be out of luck there too.

  • 538 Refugee

    I’ve missed about a week of evening runs because of illness so decided that would be a better use of my time tonight. My wife echoed the style vs. substance but I always felt these two guys would come away about even with ‘winning’ breaking pretty much along the lines of who people intend to vote for anyhow. From what I see here I’ll expect polling to come out essentially even.

  • NickT

    My guess is that the Clinton people saw the debate as an opportunity to push ideas/memes about Trump firmly into the public’s mind and that’s why Kaine kept forcing them into the discussion. Trump/Pence were working with an old-fashioned idea of verbal victories and so made the error of letting Kaine set the agenda, even though it looked as if Pence was successfully evading the issues in the short-term. I suspect that the more sophisticated approach was the long-term winner here, even though the debate itself was a fairly inconsequential event in terms of immediate polling.

    • Phoenix Woman

      Yup. Kaine had one task, which was to get Pence to say things that the Clinton campaign could put in ads.

      He succeeded admirably.

    • Avattoir


      Trump and his campaign surrogates have been fixate on winning moments & mere skirmishes. The Clinton campaign from the beginning has been all about winning this election.

      We don’t really know if anyone directs the Trump campaign. We do know that the Clinton campaign is directed by herself, Podesta, Mook & Plouffe. The difference shows.

      Top trial lawyers and football head coaches, and Bill Russell (who had to spend much of his career in the NBA dealing with Wilt Chamberlain) know all about this: it’s not necessary to win every skirmish, so long as every part of the process aims to serve the ultimate goal. This is starkly evident in how Gerry Spence conducted himself, in his typical approach to long trials, in the 1986 TV play, “On Trial: Lee Harvey Oswald”:
      Watch in particular as Spence spent a significant percentage of the trial conditioning the judge and the jury to allow him the room to develop his defense. He incurred frequent rebukes and something approaching wrath from the judge, and no doubt that initial response was mirrored in the jury panel. But he ended up making his case.

  • Jake Ingram

    Just so you know, PEC was referenced in an article at the Canadian Broadcasting Corporation’s website. The article is about the bogus online “polls” after the first debate and how Clinton has been leading pretty much from the beginning, and the specific reference to PEC is about the stability of the polls since 1996.

    You guys might also be interested in the following long-form article written by the same correspondent, entitled, “Make America White Again: Donald Trump has spent much of his U.S. presidential campaign whipping up racial fear. Regardless of whether he wins in November, he has empowered angry white voters.”

    While you’re at it, the CBC has its own poll aggregation site:

    • Arthur Klassen

      Keith Boag is one of the CBC’s better reporters, but I wonder if he gets it right to call Breitbart and Gannon “alt.right”. I mean, from up here in Canada they look like agitprop makers, and not newsguys at all but I thought a.r. meant, essentially “crypto-Klan”, a step or two beyond what they’ve shown and done. Thoughts?

    • Ed Wittens Cat

      Arthur, google the Unz Review
      Breitbart is definitely AltRight

      and read this
      and this– Milo Yiannopoulis writes for Brietbart

    • Arthur Klassen

      In retrospect, after talking to some friends, I realize that the question I so innocently posted earlier could be taken as trolling. It was not. It was a question from ignorance.

      A co-worker pointed out that Breitbart and “alt.right” occur together pretty regularly in other fora. I will go and check for myself, but first I wanted to disavow trolling. I didn’t know and asked a question — and I recognize that questions like this are regularly submitted so as to spark flame wars. My intent was self-enlightenment, not provocation.

  • bks

    The POTUS debate on 9 October will take place just about the time that Hurricane Matthew is hitting South Carolina, a day or so after it is projected to hit Florida:
    (URL may be volatile.)

    • Robert Del Medico

      projected to sideswipe FL, that is

    • bks

      It’s headed straight for Mar-a-Lago!

    • Jay Sheckley

      Huh. Hurricane Matthew could deliver a bigger October Surprise than WikiFizzle: It’s WikiLeaks 10 Year Anniversary! Whee! I mean, attention brought to human suffering, the purpose of government or even, strangely, the humanitarian message of the book of Matthew (not to mention Matthew 23’s admonition against public piety) could focus the media imagination on much-skirted issues:
      What are the odds that a photogenic family devastated by the Orlando shooter is made homeless by Matthew in time to bring up the elephant in Pence’s closet? .
      If not, close polls with a horserace spin _might_ get out the vote, and reduce 3rd party interest.
      Dang, Sam, it may just be paint drying, but as we gaze upon these sticky violet walls, inquiring minds want to know, is that paint red or blue??? The numbers seem clear, math is beautiful, but turnout is everything.
      PS We’re mailing out toy taco trucks. One goes to your lab. Trying to get one on every corner.

    • Tom_b

      No joke. It will sideswipe Miami-Dade, which is perhaps the most critical county in the state. And it could smack RTP (which would suck for me) if it fails to make the projected right-angle turn in SC.

      All said, though, Sandy did not keep NYC from helping Obama to earn his second term.

    • Phoenix Woman

      Early voting by mail is possible 8n Florida.

      In-person early voting has to wait until late October.

  • Michael

    Many pundits are citing the alleged polling failure before the “Brexit” vote as Trump’s ace-in-the-hole, also saying his campaign is thematically similar. As support, they cite the recent election in Israel, Colombia’s war-anti-war vote, and the Michigan primary, where polls supposedly “got it all wrong.” They also throw in some form of the Bradley Effect to round out their argument. Is there any wheat among this chaff?

    • Sam Wang

      Brexit polling was okay – it was too close to call, with 9% undecided.

      The Michigan Democratic primary used a bad voter model because there was not a real primary in 2008.

      GOP primary polls predicted Trump support almost perfectly.

      These are all documented here at PEC if you use the Search function.

      The other cases…I don’t know, at some point, shooting down the b.s. gets tiring. Who are these “many pundits”, anyway? Name some.

    • 538 Refugee

      How about this for “Bradley Effect”? Men who won’t publicly admit they would vote for a woman? Conservatives who won’t admit publicly they would vote for a Democrat but are truly afraid of what a Trump candidacy could do? White women who are afraid to let their husbands find out they are voting for Hillary? Trump supporters don’t sound like they are the shrinking violet types afraid to let their views be known. They are more the type you hide your views from to avoid conflict.

    • fred flint

      This is the same “skewed polling” logic that happened in 2012. Nothing to see here.

  • Roger

    Something that might last more than one news cycle from the VP debate: Governor Pence’s comment, “You whipped out that Mexican thing again.”

    It seems to be the most trending internet “thing” from the VP debate. Also, it could form into a rallying call for Latinos to vote and find its way into a Clinton campaign ad.

    A group has already bought the URL, which directs you to the main Clinton’s campaign site and started a Twitter hashtag.

    Also, I suspect Saturday Night Live will emphasize that Governor Pence snafu in its likely to come VP debate skit.

    • fred flint

      Any latino votes that were going to leave would have left Trump after the all the other outrageous and offensive comments he has already made. So anything said in a VP debate is not going to make any more Latinos leave. Those votes aren’t going anywhere.

    • alurin

      The question is not which way Latinos vote, but whether they show up to vote. Trump hasn’t called Mexicans rapists in a while, so it’s probably good to have a little reminder.

  • Olav Grinde

    Given the currently slim state-level margins, I wonder what the odds are of new polls moving Iowa and Ohio (and perhaps also Arizona) back to a Clinton lead?

    • Josh

      Ohio will almost certainly flip back to blue–it’s about 3-4% to the right of the country as a whole, so a Clinton lead of 5% nationally (roughly where she’s at right now) would include Ohio.

      Iowa is a bit further right–maybe 6 points. It’s probably a tossup.

      Arizona is more like 10% to the right of the country as a whole. Hillary will only win it in a blowout (given the parameters of this election cycle).

  • Steve Scarborough

    As a PEC disciple, and election modeler since back in the 1990’s, I should like to weigh in on the VP debate by saying it will not make a difference. Second, I wish to add support to Dr. Wang’s claim about the race being stable. I do my own modeling, where I use medians for up to 12 most recent polls state by state (use polls with Johnson and Stein.) Compute the Clinton median, compute the Trump median, take the differences. Weight the differences state by state using the electoral college votes for each state, add up across the states and then divide by 538. The result is what I call the National Percent Lead — similar to the Meta-Margin. I do my work weekly versus daily. Net result is that the NPLs I compute are very stable. So far, the standard deviation of my weekly figures going back to early July is 0.76%. Same data points for Meta Margin show standard deviation of 1.30%. Hence, I am showing the race to be even more stable than PEC.

  • Lee

    If every senate contest (including the vice-president as tie breaker) moves by the same amount so as to counteract the senatorial meta-margin, which state will be the one with the tie?

    Is there a meta-margin for the House of Representatives?

    • Steven

      Not as nuanced as the Meta-margin, but the generic congressional chart on the left points to a R +3.0% meta margin (assuming the D’s need an 8% PV to win the majority of the House).

    • Froggy

      Just shift the Senate races numbers by the Senate MM and the closest races would be NC (Burr +1.7), PA (McGinty +0.3), and NH (Hassan +0.8). Democrats would need to win two of those three, and would be have a narrow edge in PA and NH.

  • Gil

    Folks if Hillary is going to win the election then I urge all of her supporters to make an effort to help with the Senate races. A presidency without the senate will be absolutely terrible.

    Please consider the following:

    – the Senate map is atrocious for Dems in 2018; unfortunately it’s a guarantee takeover for Reps; Dems will lose anywhere from 3 to maybe 6 seats;

    – midterm elections have been a disaster for Dems, especially considering the fatigue after 2.5 terms (Obama plus Hillary); I am concerned it will probably the biggest blowout in modern history ; if Reps keep the Senate now they will block everything, effectively turning Hillary’s term into lame duck from day one;

    – even if Dems take over the Senate 50/50 there will be a special election to replace Kaine in 2017; this would be The Mother Of All Elections with all on the line , drawing potentially 100s of millions of dollars from special interest groups;

    – realistically Hillary will have 2 years to replace Supreme Court justices and that’s it; IMO that’s the best case scenario for her, unfortunately ;

    Please donate to the winnable Senate races (NC, PA, NH, etc) or to the Senate Majority PAC whose focus is solely on turning the senate blue.

  • A

    Looking at the various prediction markets and aggregators, interesting to see many places such as Daily Kos and NYT and Predictwise all starting to congregate around the 80% probability mark for Hillary to win.

    And then I look at PEC which has essentially hung out around that same mark, give or take, for a very long time now–with only moderate fluctuation!

    I think it’s a little sad that in the end, people will likely only remember the prediction models’ final numbers they give close to the election, rather than their track records further out, when there’s a lot more noise to cut through.

    If you actually kept track of this site versus many of the others, this one has held that line much closer to the mark, through a lot more supposed volatility in the polling then anyone else, just about.

    And that would seem to once again validate Sam’s no-nonsense approach, which while it gives anxious folks like myself less to pick over from one hour to the next, actually does the job it’s supposed to do.

    I hope people do take notice of that when this is all said and done.

    • Paul Ruston

      When comparing PEC, Daily Kos, NYT and other aggregators after Nov. 8th, my question is there a way to score their respective performance based from how far out their numbers matched the election results? Which forecast had the closest match to final outcome from the farthest date from the election. Also as “A” stated “If you actually kept track of this site versus many of the others, this one has held that line much closer to the mark, through a lot more supposed volatility in the polling then anyone else, just about”.
      When Sam won his t-shirt, it was based on being closest to right answer on who the nominee and vice president pick for both parties were. This prediction however had to be made by a specific date long before this was decided.

    • Matt McIrvin

      In 2012, it was Drew Linzer (now at Kos) who was exactly on the nose freakishly far out. But this time around he’s been much more bearish on Clinton.

  • Michael

    Sam, Please do not take my reference to pundits as agreement with them. The ones who keep referring to the “Brexit” bomb supposedly awaiting Clinton are Jeffrey Lord on CNN, and the crew of “Beyond the Beltway,” a satellite radio show, among right-leaning pundits. Question: If the reason the Brexit polling ultimately came out slightly off was the number of undecideds, how does that compare to the Clinton-Trump race? Thanks.

    • Amitabh Lath

      Brexit, Scottish Independence, the Colombian FARC vote, are all one-offs. As such quite hard to model I imagine. There isn’t any history to hang your Likely Voter model on.

      Date from US Presidential elections have been collected and poked at for ages. Some still get it wrong (Gallup just up and quit) but the ensemble of pollsters with their wide range of biases tend to straddle the correct answer. What’s the probability that everyone is wrong in the same direction?

    • truedson

      Brexit was basically a tie in polls before the vote…which a lot of folks forget….the attitude that Yes couldn’t possibly win is what most remember. The relation to this election is rather small.

    • Sam Wang

      It is a good thing for the GOP that those people are not in charge of allocating campaign resources.

      Brexit had 9% undecideds in the home stretch. Now we have 4% undecideds, and it will decline by November 8th.

  • mediaglyphic

    Not sure if Dr. Wang is still looking at this thread, but i notice your comment about GOTV. Are you saying that GOTV in general in your opinion? Or is it that in this particular case a GOTV advantage is unlikely to shift prediction values? If its the former, it would be interesting to know why. Most election campaign practitioners seem to put a lot of emphasis on GOTV, they might be informed by your reasoning.

    • Amitabh Lath

      What is this argument for GOTV to cause the polls to underestimate one way or another? I presume registering and persuading voters has been part of the electoral landscape for ages, and pollsters know this, and have techniques and metrics for estimating how big (and relevant) it is, and apply whatever first or second order corrections to LV models they need to.

      Why would a GOTV effort that is clearly visible to us (and CNN) not be apparent to a pollster whose bread and butter depends on getting it right?

    • Ed Wittens Cat

      Because its not clearly visible.
      jealously guarded trade secrets.

      to this–>

      in 2012 OFA leveraged a dream team of behavioral scientists– the natural progression would be data scientists and machine learning in 2016.

      but we wont know until the election

    • Amit

      Amitabh, there are 2 possible reasons why GOTV may skew the polls. One, it may change the mix of people voting. for instance, if the typical well-designed ‘likely voter’ polls assume a certain % of hispanics to vote, but GOTV changes that number, then the polls may be off. Second, even if the demographic mix does not change, it may alter the willingness to vote.
      In going door to door in FL, i see both of these effects, obviously on a very small retail scale.
      Assuming GOTV does not provide an advantage would also mean that voter supression should not provide a disadvantage vis-a-vis the polls (your logic of pollsters adjustment should carry over there too).

    • mediaglyphic

      how would pollsters adjust a poll for GOTV asymmetries? Would they model is explicitly? Or are you saying that in principle there should be no asymmetries.

      I am just trying to clarify the reasoning.

    • Amitabh Lath

      Amit, first of all thank you for your efforts in extending participatory democracy, it makes me proud to share your name.

      I agree that pollsters who fix their percentages might be off, but I am assuming the serious ones are able to craft filters that can discern highly motivated new voters, and adjust LV fractions.

      (In particle physics we have “constants” that refuse to remain constant. Called running constants.)

      Your observations are very valuable, and unless you landed in some atypically enthusiastic neighborhood what you are seeing with new voters and newly willing voters should apply at a wider scale.

    • 538 Refugee

      I was just reading articles showing heavy interest in early voting. The election has already started. Trump is still gearing up with GOTV while Clinton is marking people as ‘banked’ and moving down the list. This difference may not be picked up by the pollsters yet. State polling is still a bit unsettled. In Ohio you have two pollsters with good track records differing by 7 points.

    • Amitabh Lath

      Media, my point is simply that GOTV is nothing new. Campaigns have been signing up new voters and getting reluctant voters to the polls in every cycle. GOTV is not something new this cycle.

      And since polls have been reasonably accurate in the past, I assume they have ways to discern a newly registered, recently motivated voter with their panel of questions.

      I agree that if all they do is ask “did you vote last election” they would miss these new voters. But I cannot imagine them doing that. They can probably estimate the magnitude of the GOTV better than most of us. And adjust for it.

      I don’t know any of this but if I was designing a system to estimate voting outcomes I would start with zeroth-order estimate based on long time voters and then make perturbative corrections for new voters, and also negative corrections for disenchanted regular voters.

    • Olav Grinde

      “…GOTV is nothing new.”

      This is true, but there has usually been reasonable symmetry in GOTV efforts. In this year’s election, all the available evidence indicates that this is simply not the case!

      While Clinton’s campaign has invested heavily in GOTV, Trump’s effort seem negligible at best.

      I have not seen any evidence that pollsters are able to take that asymmetry into account – and I really don’t see how they would model for it. How can questions aiming to “filter” for likely voters possibly accurately reflect election-day GOTV measures?

      I think Mediaglyphic and other posters in this thread are onto something!

    • Davey

      Amitabh – I appreciated your questions and comments. I don’t directly disagree with anything you said, but here’s my take as someone who worked the GOTV efforts in four Congressional elections (cooperating with the presidential team in one).

      We have very low turnout in general, leaving a massive pile of potential voters to “mine.” In working GOtV, I was surprised how many voters wanted to vote, intended to vote, but just needed a knock at the door. I mean…thousands of “I kinda forgot, I was maybe going to skip, but you’re right I’m going to go vote now.”

      Seriously…it’s like herding cats.

      My point is that my experiences left me feeling like GOTV is an essential pressure exerted by campaigns before and on Election Day. Without it, it was always clear that thousands of people in a district would skip voting.

      Pollsters obviously try to account for this, and I think they do a good job generally. However, their models are based on past elections with both sides exerting relatively the same pressure. And these models have at times been able to show better performance in campaigns with better GOTV efforts.

      If this election truly has a candidate with stellar GOTV and one with awful, there’s no methodology to account for this. True, polls shouldmcarch some of it, but I can’t imagine they’re catching all of it. If Trump doesn’t have people knocking on doors, I think he’s going to lose a lot of votes. How many? No clue. It’s a social experiment we seem to be running untested in a presidential election. Fun! We’re stocking enough nachos to last until 4 am on Election night (and I’m on pst).

    • Slartibartfast

      Mr. Wittens’ Cat,

      The question I have is what GOTV effects are already baked into the polls and what are missed. In 2012 it was widely perceived that President Obama’s GOTV effort was far superior to Governor Romney’s, but this didn’t case a noticeable error in the predictions of the margins as far as I’m aware.

      You can argue that there is a bigger effect this year due to Trump’s anemic ground game throughout the campaign, but we wont know until election day who will really turn out. Certainly the numbers that are “baked in” (i.e. new voters registered) should favor Clinton, but they are already baked in to the polls and hence the meta-margin.

      It stands to reason that election-day efforts aren’t baked in already, but I have yet to see evidence of a disparity in election-day efforts causing a miss in the predictions. As far as Project Ivy goes, I think it’s a great idea and will give it credit if the Democrats over-perform the generic House ballot, but I’m not sure that it will win out over the kind of money Republicans are spending downticket.

    • mediaglyphic

      This thread is getting a little stale, but i wanted to add, that election practitioners i know (elected officials and senior campaign workers) swear by GOTV, saying that without this effort many people don’t vote.

      I am actually not so sure they are correct (or incorrect) Dr. Wangs “dim view” of the ground game intrigues me.

      Amitabh seems to be saying that pollsters should be able to discern GOTV asymmetries in their models and adjust the LV transform accordingly. I wonder how they do this?

    • Sam Wang

      To be exact, my dim view comes from a narrow reading of the question, which I interpret as: “how much GOTV effect is *not* captured by opinion polls?” My view is that much of it is captured, either because it took place at an earlier stage (i.e. voter registration and reminders) or because it’s approximately matched on both sides. This is why one can get a more accurate view after the election. Even then, one would have to find a way to compare heavy-GOTV with light-GOTV states.

    • Ed Wittens Cat

      if i may cite Olav’s comment and Dr Wangs as well– for me the problem is nonlinearity, or as Olav pointed out, asymmetry of bases. Dem GOTV targets the enthusiasm gap btwn the parties.
      Election night 2012 was obvi an internal polling failure for the GOP– witness Roves’ very public meltdown– may have been due to bad methodology or bias but GOP post-mortem analysts admitted OFA had located Dem voters they didnt know even existed and turned them out.
      So its not really possible to separate turnout due to enthusiasm for Obama from the relational behavioral science OFA database that targetted voters that COULD be turned out with GOTV.
      Remember how OFA worked– it targetted reachable voters only.
      As for the asymmetry of the bases, red/blue brain hypothesis means GOP potential voters are mostly already out there– via the enthusiasm gap.
      Im mostly stringy but also a datanerd– advances in data science and machine learning mean the OFA legacy databases are likely even more powerful today– an example of datamagic showcased here at PEC was the indispensible N’s google correlate.
      So my point is we wont know until the fat lady finishes her song. And even then we will have to look at scale– since GOTV will be applied more heavily in battleground states.
      Dr Wang is correct that not much GOTV shows up in his nonparametric math based multi poll techniques, which really arent designed to deal with non-linearity. More GOTV effect may show up this cycle– Clinton has a larger enthusiasm gap than Obama.
      I guess the test of OFA/Ivy going forward is if it can increase turnout for for the blue tendency part of the electorate– and particularily for downballot races.
      I were i designing a GOP GOTV effort for 2016, i would emphasize registration to turnout the “missing white voters”– but that is a dead end strategy going forward.
      By 2020 the GOP will need to have a relational db targetting minorities with red brain tendency and persuading them– simply impossible with Trumpian level of noise.
      In 2020 the white conservative part of the electorate slips below 50%– demographic doom for the GOP.

    • Olav Grinde

      “Fun! We’re stocking enough nachos to last until 4 am on Election night (and I’m on pst).”

      Cool! Davey, please pass me some more of #ThatMexicanThing.

    • Amitabh Lath

      I don’t know if anyone is scrolling down to this thread anymore but to echo Sam, I would assume a well designed poll would capture the effects of GOTV asymmetry just like it would capture any other asymmetric push factor (say, a candidate’s race, gender, age, misogynistic/racist statements…).

      There have been many races (maybe not at the presidential level) that have had highly asymmetric GOTV. Surely polling firms have quantitative estimates of the GOTV for both sides, and enough years of data to know how relevant and how to apply this to their filters.

    • Amitabh Lath

      Important to remember professional pollsters have been at it for quite a while.

      It’s highly unlikely that there is a large effect that is obvious to us sitting on the couch that they have not devoted significant resources to understanding and correcting for, be it cellphones or Spanish-only speakers or GOTV or whatever.

      In my experience in evaluating other people’s analysis I have found it’s rare that I think of some major issue that the authors themselves had not considered.

    • Davey

      Amitabh – I generally agree with you that pollsters should pick up a lot of GOTV effect. However, I’d love to see a very thorough, nerdy study that provides insight into who these voters are and categorizes them.

      How many are new registrations? These should be picked up in polls. How many are voters reminded to vote? Possibly picked up. And as I said above, how many are voters who get a knock on the door on Election Day because they haven’t signed off as having voted already in their precinct? These are the votes I doubt polls pick up.

      I doubt we’ll get such a study, but it’d be interesting.

  • adrian mckinty


    538 has had Ohio as light blue for the last 2 days now & you’re still at light red. Have I missed a poll from Ohio where HC is ahead? I know you can’t really comment on someone else’s model, but in the absence of a new poll what gives with 538?

    • Rex

      538 uses economic predictors and nation wide polls. Also there was a new Ohio poll today with a HRC +2.

      My advice is to pick one predictor. I recommend PEC because it’s transparent, consistent, no BS, and has the very useful meta-margin. Plus, Sam is much funnier than anyone else making models.

    • CW

      The 538 polls-only prediction does three things PEC does not. It adjusts state polls for trends in national polls- so poorer Ohio polls for Clinton two weeks ago get pro-Clinton bumps and then are averaged in. It also factors in assumed correlations between states. So great polls for Clinton in Pennsylvania impact Ohio. Finally it performs a demographics regression– essentially a check on likely voter screens

      These seem like common sense assumptions, but I’ve never seen 538 provide a robust argument they understand what’s required for these assumptions to be valid, systematic errors they introduce in the model, and improvement in accuracy these assumptions are supposed to provide. Until I do see that, I’ll always take their model with a grain of salt.

    • Sam Wang

      Those adjustments sound like they would each add substantial uncertainty.

      I have been thinking that the Meta-Margin is actually a way to capture all the uncertainties in one place, without introducing extra uncertainty. Since the Meta-Margin is calculated by compounding poll-descriptive-statistics-based probabilities only, its changes over a campaign season give an empirically-based measure of how much the polls, as an aggregate, vary over a campaign. The appropriate amount of correlation structure between states is, in a sense, baked into the MM. It is effectively much less variable than the probabilities reported over there.

      The main problem I see with an MM-based approach is how to determine its systematic error on Election Eve. I only have three data points for that: 2004, 2008, and 2012. I have an estimate of 2000 as well, 0% (Florida was too close to call, and we all know what happened).

    • 538 Refugee

      I bit on this because polling in Ohio seems a bit erratic so I was curious. Actually, as I write this the “Polls+” model shows Trump winning Ohio. 53.1% I include the decimal point for amusement (and derision) purposes only. I don’t know if he includes his poll weights and other bias corrections in his “polls only” forecast. Nor do I really care. Based on the volatility of the polls in Ohio, I’m comfortable with calling it a toss up at this point.

    • Matt McIrvin

      538 makes a lot of adjustments to the state poll numbers it gets. Every individual pollster has an estimate of systematic bias (calculated how, we don’t really know) that they subtract off from the numbers from that pollster, which can often flip the result from one candidate to another.

      But the most significant tweak they make is to try to use the trends in national polls to correct all their older state numbers for being slightly stale. It’s particularly noticeable in the “now-cast,” though it figures into all three models. And it makes their electoral-vote maps jump around a lot when the national polls are changing.

      It makes watching 538 a dramatic experience, but I’m not sure it’s really adding useful information. Is a November prediction that moves around that much in response to day-to-day events really a decent prediction?

      I like Sam’s approach better, of using median-based averaging to try to deal simply with systematic biases, and calculating a Meta-Margin and offering Trump +2 and Clinton +2 maps, so if you have a personal hunch that something’s changed nationally, you can estimate the electoral effect yourself. The adjustment is not hidden behind a veil.

    • Commentor


      With all due respect, I don’t think you’d be able to drive traffic to a website at the same rate as Silver using your boring overly-mathy sounding meta-margin.


    • Sam Wang

      Damn straight.

    • mediaglyphic

      Dear Commentor,
      organizations whose purpose is to drive traffic will be incented to drive traffic (which might mean injecting uncertainty, where there is none).

      its interesting that you think the meta margin is “overly mathy”, i actually feel it simplifies a lot of underlying math. Chaque un a son gout.

    • MNP

      Sam’s model always moves after the others.

  • Tapen

    Back to the subject matter of watching paint dry.

    Here is the picture of what happened to betting
    markets before and after the debate.

    Notice the YUUGE movement?

    <) )╯
    / \

  • Tony Asdourian

    Dr. Wang– what do you think of this article in the NYTimes which asserts final polling averages in aggregate are historically not nearly as accurate as one might expect? It would seem to be in conflict with the assumptions of your work, but maybe I’m misunderstanding…

    • A

      Very interesting article, would be curious to hear Sam’s take as well.

    • hubcap

      I’m not Sam, but just looking at the Presidential charts I see one issue – it tracks the error in the polling average for each state, treating each state as equal.

      But in a Presidential, at best 20(?) states are competitive in any real sense. Those states get polled heavily. Other states…not so much. Certainly not by multiple, high-quality pollsters.

      And so the polling average in Alabama, Hawaii or Utah (for example) is more likely to be wrong, because there will be fewer (and likely worse) polls for those states. But from an overall predictive point of view that’s irrelevant – we can safely guess the outcome.

      Take Utah. The current Pollster average has it Trump +18 (45 – 27). But with Evan MacMullin and Trump’s general crapulence, who knows? That average may very well be off by 5 or more. So Trump wins by 13 and the polling was way wrong. But so what? It was still 99% Trump.

      Whereas the fat part of the chart, all the +/-2% states – I would guess those are the Ohios, Floridas and Pennsylvanias of the world that got polled over and over and over. And those are the states that are drive a Presidential election prediction, because those are the states that might actually swing.

      There’s a feedback loop going on. Uncompetitive races don’t get polled because we know the outcome; therefore polls for uncompetitive races are more likely to be wrong – but not wrong enough to affect a prediction of the outcome. Because the race was uncompetitive in the first place.

      I would guess something similar is going on with the Senate and gubernatorial charts. The big misses are likely races that were never competitive, and so weren’t polled extensively, and so the polling average was more likely to be off. But being off 7 in a blowout means very little. It wouldn’t invalidate any predictions.

      I would be useful to know things like 1) how many polls were in the average for each state or 2) the margin of the error vs the margin of victory. That would help separate misses that might have affected a prediction, and misses that made no difference.

    • Sam Wang

      Basically agree with hubcap’s comment in full.

      I analyzed this in 2008. Rothschild and Goel may have calculated the wrong quantity. They need to calculate a few items:
      (a) How many polls made “sign errors” – wrong predictions. The error rate is probably smaller than their naive statement would suggest.
      (b) What the fractional error is, as a means of determining whether more lopsided states have larger errors.

    • CW

      I don’t see the conflict. This article identifies and attempts to estimate the systematic errors of polling. Sam gets at this through his standard error of the median and shows the impact through the confidence interval on the median EV estimator. From my quick glance, it looks like the Upshot estimate of ~2% in systematic errors that don’t cancel out through averaging is consistent with Sam’s confidence interval on the EV estimator (eyeballed back to the MM).

      I do think it’s great that the NYT is talking about systematic errors in polls- both this article and the Siena poll they recently did. As an amateur (and former physicist), I’m now seeing evidence of folks trying to pick apart the sources of error that actually matter- that will only help in making polls — and their interpretation — more accurate. At the very least, it helps explain why Sam’s error estimate works.

    • Tony Asdourian

      Very helpful responses, clarifies matters for me. Thanks, all.

    • Matt McIrvin

      If people take away one thing from the article, I hope it’s that the “margin of error” cited in poll results isn’t what a layperson should take as the margin of error. It only measures pure sampling error, which is dependent on nothing but the sample size and can be minimized by aggregating polls. But in aggregates particularly, systematic error is more important–it’s probably larger than the MOE to begin with, and it’s entirely possible that there’s some reason everyone is off in the same direction.

      In practice, Sam’s model seems to have done better than you’d expect by taking this article at face value, probably for the reasons various commenters have given.

    • Amitabh Lath

      Nice of Upshot to give a shoutout to systematic uncertainties, but other than pointing out they exist, what? They don’t make an attempt to quantify the various sources of systematic, they just list a few likely culprits.

      All one can decipher from the article is that they estimate total sys ~6.5% (if stat uncertainty is 3% and total is 7%, and they add in quadrature). The overall plots don’t look to have long non-gaussian tails so this is ok.

      The new internet pollsters like Yougov and LAT/USC are interesting because they will have entirely different systematics.

      Ditto if PEC reader N. turns google correlate into a vote prediction engine. Time for a summer workshop?

    • Rudy

      Polling data is stochastic with uncertainties that go well beyond just the sample size or 1/square root (N) spread. Subject behavior, mood are just a few of the other factors; and am sure subject orneriness is neither quantifiable nor predictable… But I love Sam’s website and cant believe how unerring it has been, for example, he predicted more than 3 weeks ago that it was the most stable election and that Clinton’s lead is going to expand again when the true believers were panicking

    • Michael

      Haven’t thousands of polls at the state and national level over decades established a fairly reliable norm?

  • Paul Quirk

    There really needs to be–posted in a prominent location–an explanation of the empirical basis for the key assumption of the Bayesian estimate–i.e., that there is a tendency to regress to the mean values of a very long period. I never heard anyone say that before.

  • Scott J. Tepper

    My only concern right now is that Donald Trump himself is the Black Swan.

    • Paul Griner

      Actually, as a novelist, I prefer the first version. He considers himself King of all he surveys, so speaking of him in terms of royalty seems somehow fitting

    • Matt McIrvin

      A commenter on Lawyers, Guns and Money pointed out that there are fairly energetic vote-suppression efforts going on in Wisconsin, Ohio, NC and Florida, all of which are close states now; if Trump sweeps those (and holds Iowa and ME-2), he wins, possibly with 270 electoral votes exactly.

  • LongStrider

    Since I’m in PA I’ve been following Toomey/McGinty relatively closely. Is there anything statistically significant about polls with high undecideds or races with wildly varying undecideds (5%-22% in recent polls in this race) ?

  • Ed Wittens Cat

    Not sure if anyone is still interested but heres a quantifiable effect of failed GOTV– at least in GOP historical annals
    The Wreck of the Orca

Leave a Reply to Gil (Cancel)