Princeton Election Consortium

A first draft of electoral history. Since 2004

Meta-Margins for control: House D+2.8% Senate R+5.6% Find key elections near you!

Scotland Referendum Fails, Surpassing Polls

September 19th, 2014, 10:21am by Sam Wang


The Scotland referendum on independence has failed by a margin 10.6%. Why was this margin so unexpectedly large?

A few days ago, I pointed out that “No” was ahead of “Aye” by 4.0 ± 1.3 % (5 surveys; 1 sigma uncertainty, p=0.014). Later surveys didn’t change that. It is incontrovertible that the polled demographic was against – but was closely divided. What changed?

Whenever well-executed polls don’t match outcomes, two major possibilities come to mind:

  1. The polled demographic wasn’t representative of who actually voted;
  2. Voters committed or changed their minds.

In this case, possibility #1 seems remote, since turnout was close to 90%, and even undecided voters said they had a high intention to vote. That leaves possibility #2.

In the week of surveys before the election, six pollsters reported a median of 7.0 ± 1.2% undecided voters. It is probable that nearly all of these voters decided at the last minute that they would like Scotland to stay part of Great Britain. The one person held in strong positive regard in Scotland, Queen Elizabeth II, weighed in gently but clearly on the subject. As I suggested before, Her Majesty may have swayed some of these undecideds – and maybe even a few decided voters.

Update: using a very fancy “napkin,” Gregory Primosch reaches the same conclusion.

Tags: 2014 Election

22 Comments so far ↓

  • Steve Schran

    The discrepancy between the polling and result of the Scottish referendum is that the final polls can be attributed to two factors. First, less importantly, there was movement toward “No” in the final polls that wasn’t incorporated in the median. But second, I think it is very reasonable to believe that there was a significant “Silent No” effect, coincidentally perhaps of similar size exhibited in the Quebec ’95 Referendum. http://en.wikipedia.org. /wiki/Quebec_referendum,_1995.

  • Gregory Primosch

    For those interested, my dataset above is from Wikipedia page compiling the polls from the referendum.

    http://en.wikipedia.org/wiki/Opinion_polling_for_the_Scottish_independence_referendum,_2014

    The percent undecided breaking towards no is calculated as follows:

    Undecided Break % = (Final No % – Poll No %) / Poll Undecided %

    You get some extremes where undecided break % is greater than 100%, but the 9 poll moving average cleans that up.

    • shma

      I find this hard to believe as many pollsters asked undecideds what they would do if they had to make a decision and there was no indication that they leaned strongly towards “no”. Certainly not anywhere near 100-0.

  • bks

    There was more reason to lie (to pollsters) about your intention to vote No than your intention to vote Yes. –bks

  • Jennifer Groh

    Hi Sam,
    Technically #1 may still be a possibility, depending on how the pollsters weighted their samples. That is, nearly everyone did show up, but if the pollsters assumed otherwise and weighted their results accordingly, that would have introduced error. Were the pollsters reporting raw numbers or were they weighting their samples based on estimated demographics?

    That said, I tend to agree that some late cold feet kicked in, though.

  • WDR

    Their seem to have been an unusually high proportion of people polled who said they were sure to vote, but undecided. I don’t think most of them were really undecided–there were a number of potential good reasons for someone who wanted Devo-max (which wasn’t on the ballot) to vote No but want to have the polls close.

  • Craigo

    Polling generally overstates the Yes vote in referenda and initiatives; see alternative vote, Wales, Quebec, California…

  • Steve Schran

    Prof. Wang, two points. First, if one is looking to assign individual credit for moving undecideds to the “No” column, the movement of opinion more closely follows Gordon Brown’s very well-received speech in defense of the Union, delivered midday on the 17th. Note the observed changes between polling completed on the 16th vs. 17th. And it sure would be interesting to learn if the afternoon of the latter date saw significantly different results than before. And second, no comment on the “Shy No” phenomenon?

  • Kenny

    I heard someone make a comment about how it’s possible some people liked the idea of Scotland independent in theory, but when reality set in and they had to live with the “consequences” of that decision, they may have changed their vote.

    So perhaps it was a bit of cold feet.

  • Alon Levy

    According to 308, last-minute undecideds actually voted Yes. However turnout mattered: turnout was generally high, but higher in pro-Yes councils than in pro-No councils. In addition, No voters were a bit more reluctant to tell their friends about their vote than Yes voters.

  • Tim in CA

    Sam,

    Justin Wolfers at NYT’s The Upshot see’s the Scotland vote as mark in favor of a polls+”Fundamentals” approach and against a polls only approach. Comments?

    http://www.nytimes.com/2014/09/20/upshot/scotlands-no-vote-a-loss-for-pollsters-and-a-win-for-betting-markets.html

    • shma

      This is complete BS. Wolfers doesn’t even get basic facts right about the polling. For instance he says

      ” The polls were volatile; they often gave conflicting signals;”

      Not even remotely true! Take a look at the polling data. Here’s the median ‘no’ margin of the last 7 days worth of polls (or last three polls if less than three were conducted over the 7 day running window) taken on the morning of every day in September, going backwards from referendum day

      4.00% 4.00% 4.00% 3.00% 4.00% 4.00% 4.00% 4.00% 4.00% 4.00% 4.00% 2.50% 2.50% 4.00% 6.00% 6.00% 6.00% 6.00%

      [Note, I’m using the morning after the last day in the field as the “date of publication”, so I may be assuming some clairvoyance here on the part of the poll aggregator.]

      That’s incredible consistency. There’s barely any volatility at all in the past two weeks, and no conflicting signals.

      “The major polls in the past week ranged from a 6-point lead for the Yes vote to a 7-point lead for the No vote.”

      Using the most extreme two polls out of dozens as a way to judge their aggregate performance shows extreme ignorance.

      “over at Betfair, bettors rated the chances of a No vote at around 80 percent, an estimate that remained remarkably stable over the past week, fluctuating by only a few points”

      An 80% probability of winning and a 10 point margin of victory are almost completely incompatible predictions. You’d have to assume an extremely high variance to come even close.

      “Betting on the likely winning margin also suggested that the No vote was most likely to win by around 4 points. Yes, bettors underestimated the winning margin, but they were still closer than the election-week polling average.”

      The median poll outcome was exactly 4 points, and the average on election eve (again, taking the last week of polls) was just under 3 points. In other words, betting sites did NO BETTER than pollsters.

      In fact, his entire article seems to be based on the premise that this election was incredibly close, when the result shows that it was anything but.

      The polls had an awful miss, underestimating the NO side significantly, but Wolfers doesn’t seem to understand what they got wrong. He doesn’t even seem to understand the referendum result itself. And his praise for betting sites is not based on any factual analysis.

    • 538 Refugee

      I asked before the vote how one models a one off to start with. I’d REALLY be interested in knowing how you come up with “fundamentals” for a one off. Increased or decreased sales of Oiuja Boards?

  • Leading Edge Boomer

    In this case the bookies had it right, and the pollsters did not. Apparently the “No” bettors were being paid before the polls closed.

  • Billy

    Also another thing to consider: how did pollsters choose their demographic? With turnouts of ~90% or above, this actually becomes really important as the pollsters pretty much have to accurately profile the entire population of Scotland. With “regular” elections it’s a bit easier to just figure out who’s going to come out to vote. I don’t think this is something pollsters really have that much experience with, considering that referendums of this magnitude are quite rare.

  • pgvaidya

    Sam,

    You are right. The queen made a difference. There was a report on BBC that said that, the event was planned and well orchestrated.

    There was also a second factor. Gordon Brown came out of his depression and in the last days carried out a terrific campaign for NO. His accent, and his habitus was 100 percent scottish. He made promises that seemed to be quite reasonable.

    I have lived in England before and now live in california. I suddenly realized how fortunate we are in the US that Californians did not need the permission of Alabama to raise taxes to support education. We might soon have a single payer system in Vermont. If Vermont was stuck with the rest of the country, they would have liked to separate too.

    Most likely, we have a new UK now, else, we will be back to referendum again and this time they would give iron clad guarantees to the retirees that their pensions will not go down and everyone else will be told repeatedly that Her Majesty (for all her IQ level) will become the queen of scotland the way she is today the Queen of Canada. As my Canadian friends tell me, she also just happens to be the Queen of some island off shores of France — but all that we care about is that Canada has our Queen.

  • Chatham

    So what’s the timeline on this? The no vote has always been well ahead of the yes vote in polling, and was ahead by about ten point just a few weeks ago. So between then and now, a bunch of undecideds switched to yes, then when it came time to vote all the rest of the undecideds switched to no? That seems a bit odd.

    • Gregory Primosch

      The Yes undecideds “came out”, as it were, after the first debate, August 25th. The remaining undecideds apparently were almost all No voters (if you subscribe to the hypothesis presented above).

  • pechmerle

    // a commenter on Sep 17, 2014 at 10:55 pm:

    Even assuming a 2% overall error and higher chance of a black swan event, 1:5 odds are good enough to take.

    But not as good as the even money bet that YES will get between 45 and 50% of the vote. Unless they’re counting spoiled ballots in that total, that’s just ridiculous odds.

    Turns out the bookies got that one right! The YES vote total fell a shade Under 45% — 44.70% as reported by the BBC. Actually, on the precise arithmetic, they rounded up slightly to get to the .70. So that good -looking bet that YES would fall between 45 and 50% actually didn’t pay off. (And no, adding spoiled ballots — of which there were very few — as all YES wouldn’t have got YES up to 45%.)

  • Pier

    the big outlier was in fact the You Gov poll that gave YES ahead by a healthy margin 53-47. The raw data collected for the same poll told a different story: 47-53 in favour of NO.
    At least in the US you are providing, Sam, pollsters and press with lots of food for thought about “fundamentals”. This is not happening in Europe at the moment where pollsters reign unchallenged over an adoring press

    • 538 Refugee

      Sam’s methods are open source. Encourage people to read and use his code and modify as appropriate for the situation. One size does not fit all. Get a group of like minded individuals together and ‘have at it’.

  • Mark S.

    One possible factor: the final vote was in line with the average polling over the past year. Polls always showed No winning, with margins from 6% to 20%. Only in the last 2 weeks did polls show Yes winning or close. It’s almost as though Scots were answering polls being swept up in the emotions of the moment, but when it came to voting they reverted to their long-held opinions over the past year.

Leave a Comment