Was I wrong about NY-23? Only two polls were available. They came before the full impact of Scozzafava (R)’s last-minute endorsement of Owens (D), which evidently worked. It’s a squeaker. Owens won by 4%. This is a lose-lose for Republicans: loss of the seat is a self-inflicted injury by the party’s unreconstructed right-wingers. Full results are here.

Since 10/23 (i.e. the last 10 days), the race has been static. Polls spanning that entire period give
Final margin: Christie over Corzine by 1.0+/-1.0% (68% CI, 16 polls).
3-way outcome: Christie 45.5%, Corzine 44.5%, Daggett 10.0%.
Christie win probability: 83%, i.e. 5:1 odds.

More details, and analysis of the NY-23 race, after the break.

As usual, I use data from the fine obsessives over at Pollster.com: NJ-Gov. (Note that they omit a late update to a Fairleigh-Dickinson on the grounds that it pools data over 11 days, with 7 days previously reported. Including that poll does not change my bottom line.)

Last November, I suggested that variation in individual measurements arises from two sources: (1) sampling repeatedly from a fixed distribution, and (2) changes in the distribution. On average, sampling from a fixed distribution gives the same standard deviation, independent of the number of samples taken. If the distribution shifts or changes shape, then the standard deviation would be likely to change.

If we plot the standard deviation of a standard quantity (in this case, the Corzine-Christie margin) over various time windows from Date X until Election Eve, allowing X to vary, the standard deviation should stay approximately constant for periods during which the race did not shift – but rise steadily in periods of true change. The plot looks like this:
Please pardon the confusing x-axis label. It indicates the first poll # used. There were 48 polls, so that the leftmost point indicates the SD of polls #1-48, the next point is SD of #2-48, and so on.

The plot starts to take off at #44 (Quinnipiac, 10/27-11/1). It might be that polls after that, taken over the weekend, show more volatility. But there’s not enough data to tell for certain, and in any case these polls still give a median of Christie by 2 +/- 2%. So instead I will go back to the minimum SD at #33 (PPP, 10/23-26). That set of 16 polls gives the median and error bar (estimated using this method) listed above.

It’s right down to the wire – the Christie-Corzine margin is not statistically distinguishable from zero. But if we believe that these polls are representative of voters, the odds do favor Christie.

Considering that Republican-leaning votes are being split by Chris Daggett, a Republican running as an Independent, it is clear that Governor Corzine is not a popular guy. If Corzine pulls this one out of the fire, it will be thanks in part to Daggett. Also, Christie was hurt by his weaknesses – leaving the scene of an accident where he hit a motorcyclist with his car, lack of compelling message, and abuse of his position as U.S. Attorney. We’ll see tomorrow.

For other statistical analysis and color commentary, I refer you to Fivethirtyeight by Nate Silver (Christie win probability 57%) and Stochastic Democracy by David Shor (Christie win probability 53%). They and I point in the same direction, toward Christie. But they are less certain, perhaps in part because they use a shorter time window. I consider this a test of my variance minimization (VM) method.

Postscript: The last 16 polls sampled 12,860 respondents. Binomial sampling suggests that from such a sample, it ought to be possible to get a 68% confidence interval of +/-0.4%. Clearly there are additional uncertainties having to do with methodological differences among pollsters and whether n=16 is the right number of polls to use. But it’s not crazy to imagine making an even more accurate estimate than what I have given.

NY-23: The Conservative candidate, Doug Hoffman, is favored over the Democrat, Bill Owens. The right-wing push for Hoffman, who does not live in the district, has contributed much of the excitement here – as well as the fact that Dede Scozzafava (R) has dropped out and endorsed Owens. However, take a look at the polling data, which show that Owens’s support has stayed in the 33-36% range. Basically, Republican voters are breaking toward the Palin-backed Hoffman, not a surprise. Considering that Scozzafava wants Owens to win, dropping out was a bad move. She would have been better off staying in the race. For more commentary, read Mark Blumenthal.

The fact that the Republican base has rejected the moderate Scozzafava would seem to be a sign of the Republican party being pushed further to the right by its base – seemingly not a good move for them, no matter what happens in this race.

For more on NY-23, see polls at Pollster.com and reporting at TalkingPointsMemo.ons

Inflated voting is the crudest of fraud, cruder than anything implied by my previous post. At this point statistical analysis is interesting for filling in details and estimating the extent of the fraud.

In other news, Boud Roukema has now reported three of the six most populous voting areas in his analysis have vote totals for Karroubi that start with 7, the anomaly he reported before. All three of these have greater proportions of  votes for Ahmadenijad than the other three voting areas.

Analysis of fraud in Iran 2009 is an unfolding story. In this post I focus on Election Day returns themselves, which suggest: Votes may have been, in some sense, transferred from the minor candidates (Karroubi and Rezaee) to one of the major candidates. This could have been legitimate (that is, voters changing their minds when it came time to vote) or illegitimate (for example, minor-candidate votes being counted for Ahmadinejad). It is currently not known whether enough fraud occurred to flip the election.

Obviously, fraud is not necessarily confined to that suggested by this analysis. For summaries and updates, see David Shor and Fivethirtyeight.com (start with basic returns and poll analysis).

> > >

Some crude types of fraud can be identified entirely from the numbers used to commit the fraud. This is a principle known in financial auditing, and relies on a phenomenon called Benford’s Law. The basic idea is that a wide variety of data have first-digit distributions that tilt toward low values. In other words, measurements such as 1345, 198, or 229 (first digits 1, 1, and 2, respectively) are much more likely to occur than 397, 745, or 9618 (first digits 3, 7, and 9). A similar observation even applies to second digits and so on, though the effect is much weaker.

This somewhat counterintuitive phenomenon arises in many situations: areas of islands, financial numbers, and physical constants, and more. An excellent popular article comes from Ted Hill, a mathematician who provided a rigorous proof for how Benford’s Law could arise when one takes random samples from distributions that themselves vary randomly.

A likely mathematical consequence is that observations can be uniformly distributed – on a log scale. To illustrate what logarithmic distribution implies, imagine an investment that doubles every ten years. It would take ten years to get from $1000 to $1999 (first digit = 1 all the way), but only ten years to get from $4000 to $7999 (first digit = 4, 5, 6, or 7). In this case 1 is the far more common leading digit. This is just one example of a logarithmic distribution – many financial quantities do the same thing. But when people cook books, they often don’t pick numbers that lead to Benford’s Law. Failure of numbers to match this expectation is often interpreted as evidence for possible fraud. A similar principle can be used to analyze elections.

Roukema’s claim. Boudewijn Roukema, an astronomer at Nicolaus Copernicus University in Toruń, Poland, made waves last week when he analyzed data from the Iranian Ministry of the Interior for the four candidates (Ahmadinejad, Mousavi, Karroubi, and Rezaee) in 366 voting areas. A PDF of his report is here. In brief, he found several deviations, including too many leading 2’s and not enough leading 1’s in Ahmadinejad’s reported numbers. He also reports quite an odd phenomenon: too many leading 7’s for Karroubi’s totals.

A problem with Roukema? A poltiical scientist at the University of Michigan, Walter Mebane, specializes in the application of Benford’s Law to the detection of electoral fraud. He has pointed out that first-digit distributions often don’t follow Benford’s Law well. Indeed, this was originally noticed by Benford himself (see Table IV of his classic 1938 paper).

In the case of election returns, one big reason Benford’s Law can fail is that voting areas aren’t necessarly randomly sized (recall that Hill assumed that distributions varied randomly; here a “distribution” can be construed as being a voting area). For example, if voting areas are set up to be uniformly sized to contain 100,000 people each, then close races will produce lots of 4’s and 5’s as leading digits. So Roukema’s claim is not yet overwhelmingly convincing.

Beyond first digits? Mebane advocates the use of second digits (for instance, the 5 in 4567; technical article here). Second digits don’t show as strong a trend and therefore require careful statistical analysis. But they are less likely to violate the ideal-case prediction of Benford’s Law for first digits. Mebane has begun to analyze the 2009 returns.

At the level of the 366 districts, he found that the results fell within expectations. However, he has more recently obtained data from 12 provinces including over 11,000 ballot boxes. At this much finer-grained level he has found deviations from expected second-digit distributions for Karroubi and Rezaei (see pages 21-22 of Mebane’s report).

What happened? So did something funny happen to Karroubi and Rezaei’s votes? Maybe, but not necessarily. Over 80% of ballot boxes showed single-digit numbers of votes (i.e. fewer than 10 votes) for Karroubi, and over 70% showed single-digit numbers for Rezaei. For these boxes there is no second digit to test. In such a situation one might imagine exceptions to the second-digit distribution should obey Benford’s Law.

In email with Prof. Mebane, he has expressed faith in the analysis’s applicability in this situation. My own inspection of the ballot box-level data, which he very kindly provided, is that the minor-candidate data appear to have a “scale-invariant” distribution, a condition that appears in proofs of Benford’s Law. I may have more thoughts on this later.

But another line of evidence can approach the question independently: pre-election polls.

Pre-election polls. I previously pointed out that in Tehran, Ahmadinejad outperformed all six pre-election polls by a median of 16 percentage points. Now let’s look at the minor candidates.

In 12 polls taken since May 1st, polled support for minor candidates was Karroubi 7+/-1% and Rezaee 8+/-4% (median and SEM). They showed at least 3% each in Iranian polls (though less in a Western-commissioned poll that showed a large fraction of undecided voters).

In official election returns, Karroubi obtained 0.9% of the vote and Rezaee obtained 1.7%. Where did their support go? Maybe it was never there, though that would require assuming a significant failure of Iranian polling. A second possibility is that minor-party candidates reverted to Ahmadinejad or Mousavi when it came time to vote. Certainly this is a common occurrence in U.S. politics. But it’s been pointed out elsewhere that such a “coming home” does not fit with Iranian politics, and notably it did not happen in Iran’s 2005 presidential race.

A third possibility is that minor-party votes were fraudulently transferred to a major candidate. This possibility is consistent with the statistical analysis of Prof. Mebane. Also, the total support in opinion polls for Karroubi and Rezaee is not far from the 16-point discrepancy in Tehran I identified earlier.

What does this mean? If minor-party candidate votes were transferred to Ahmadinejad, it’s conceivable that he did not originally get 50% of the vote, which would have forced a runoff. However, I would not go so far as to say that the election was stolen. My reading of this evidence is that fraud may have occurred, but not enough to flip the election.

That having been said, the threat of fraud undermines faith in free elections, whether or not it influences the outcome. In Iran, we’re now seeing the bloody aftermath of that loss of faith.

A final note: none of this analysis rules out other forms of fraud that are harder to trace. For that one can turn to other sources of information such as the 2005 election…

People watched all of the debates very carefully and the result was actually very bad for AhmadiNejad…[I] am saying that as an Iranian based on what I saw and heard from people during and after the debates. Rezaei did very well in the debates and got popularity (if not votes) specially for his debate with AhmadiNejad.

I wonder if undecideds felt this way.

At a minimum, hundreds of statistically minded people are poring over Iranian election data (see my previous post and these posts). Sorry for my slow start…but let’s roll.

Three general categories of data are currently available for validating the Iranian election: (1) pre-election polls, (2) statistical methods for analyzing standalone voting data, and (3) statistical comparisons with past elections. Categories (2) and (3) are out on the web already, and I’ll comment soon on those efforts.

First, let’s look at some polls. A simple look at pre-election polls leads to the following assessment: National Iranian polls were highly variable and of suspect quality. But within Tehran, polls were more uniform and allow a comparison. Six Tehran polls gave a median lead for Moussavi by 4%. This differs notably from the official tally for the city, Ahmadinejad by 12%. The 16-point discrepancy suggests an anomaly in Tehran and opens the question of whether fraud occurred here – and elsewhere. However, it is also important to note several caveats, including polling uncertainty and possible shifts in opinion following the Ahmadinejad-Moussavi debate on June 3rd.

More after the jump.

In the US Presidential elections of 2000, 2004, and 2008, election-eve opinion polls were remarkably accurate predictors of the electoral outcome. In each case, aggregated, unadjusted polling data successfully identify key states (Florida, Ohio, and Pennsylvania) and came quite close to the final outcome. It’s a testament to the power of polls.

The Iranian election presents a harder case. Polling is sparse, professional standards of reporting polls are absent, and respondents are potentially unwilling to answer questions or hard to reach. Still, let’s look at the publicly available polls.

National polls are all over the place, even if we only take data after the Ahmadinejad-Moussavi debate on June 3rd, potentially a major decision point for Iranians. One post-June 3rd poll shows Ahmadinejad +16% (47% to 31%). The other shows Moussavi +32% (23% to 54-57%). Earlier polls range from Ahmadinejad +33% to Moussavi +30%. These data are so variable that they are unusable.

Polls within Tehran may be a better source of data. This is plausible because urban areas are easier to sample. The last three Tehran surveys (before June 8th, before June 7th, and June 3rd) show Moussavi +4%, Ahmadinejad +8%, and Moussavi +17%. Before that, three surveys showed Moussavi +12%, +4%, and +2% (May 27, May 26, May 14). The averages are

Last 3 polls: Moussavi +4 +/- 7% (median +/- MAD-based SEM).
All 6 polls: Moussavi +4 +/- 4%.

The announced official result was Ahmadinejad +12% (51.6% to 39.4%), a discrepancy of 16 points. When all 6 polls are used, this discrepancy is highly significant (p=0.003).

For now, my interpretation is that the official returns in Tehran are unbelievable. However, I can think of two three alternate explanations.

(1) Ahmadinejad really mopped the floor with Moussavi in the debate. The experience in U.S. elections is that debates provide a side-by-side comparison that can shift opinion substantially (for a famous example see Carter-Reagan 1980). In the case of Iran 2009 there are only 2 or 3 post-debate polls. A comparison using just 3 polls does not quite reach statistical significance (p=0.07).

(2) Tehran polls have a systematic overall pro-Moussavi bias that prevents a direct comparison with vote counts. For example, as David Shor points out in comments, polls might have been restricted to the actual city of Tehran, which is not all of Tehran province.

(3) Last-minute minor party candidates broke in favor of Ahmadinejad. There is plenty of precedent for third-party (and in this case, fourth-party) candidates to revert to one of the major candidates. U.S. readers, think of Nader supporters in the last two elections, who underperformed the opinion polls. And then there are undecided voters, who usually break against the incumbent in the U.S. but it’s not clear what would happen here.

I should emphasize that Tehran is not representative of the entire nation. It is notably more pro-Moussavi, which can account in part for the public anger there. In fact, if the 16-point discrepancy were corrected nationwide it would still not be enough to alter the overall outcome.

Iranians and other knowledgeable observers, please comment.

This type of data can be used to look for fraud. Crude fraud can be found by applying Benford’s law, which states that for many categories of data, including vote counts, the first digits of lists of observations are not uniformly distributed (as one would naively expect) but instead skewed toward low values. There are more sophisticated tests as well. Prof. Walter Mebane at the University of Michigan is knowledgeable about such analysis and is applying the methods to the data from the recent Iranian election. In addition to intrinsic peculiarities such as Benford’s law, he is also using 2005 election data as a baseline to help discover unexpected anomalies.

He currently says “”I think the results give moderately strong support for a diagnosis that the 2009 election was affected by significant fraud.” I haven’t had time to go over his analysis, but here it is for interested readers, along with a ZIP file of source code and data. Note that he is still updating his analysis, so regard this as an interim report.

Later on I’ll take a critical look at this and other approaches to detecting fraud, with the idea of arriving at a synthesis – and possible conclusions.

Thanks to David Shor and Andrew Gelman for making this connection.