Today I describe how we address pollster heterogeneity. Along the way I will also answer (1) why our state probabilities appear more confident than other aggregators, and (2) why the EV distribution at right is so spiky.
It is well known that pollsters vary in their methods. The American Association of Public Opinion Researchers has established common standards of practice and encouraged transparency, driven in part by Mark Blumenthal of Pollster (our data source). But poll-
sniffers aficionados know that Rasmussen Opinion’s results consistently trend more Republican than other organizations.
Differences like this present a challenge to poll aggregators. An obvious solution is to estimate the size of each pollster’s bias, then subtract it. However, this generates three new problems: (1) Who is the neutral reference point? Gallup? Quinnipiac? Rasmussen? (2) What to do about pollsters who do very few polls? (3) What if the pollster changes methods mid-season?
For my Meta-analysis I have chosen a simple solution that gets rid of most of the bias: use median-based statistics. Here’s how it works. Imagine the two following similar sets of poll margins between candidates A and B:
Data set 1: A +2%, A +4%, tie, A +3%, A +1%.
Data set 2: A +2%, A +4%, tie, A +3%, B +4%.
The difference is that in the second case one pollster is shifted by 5% toward candidate B, approximately corresponding to the Rasmussen effect. This single outlier poll brings the average margin toward candidate B, and increases the uncertainty considerably:
Data set 1 (averages): Candidate A leads by 2.0 +/- 0.7 % (mean +/- SEM), win probability 98%.
Data set 2 (averages): Candidate A leads by 1.0 +/- 1.4%, win probability 74%.
However, now use medians. The two data sets have the same median, 2.0%. Median-based statistics allow calculation of estimated SD, defined as (median absolute deviation)*1.4826. This gives
Data set 1 (medians): Candidate A leads by 2.0 +/- 0.7% (median +/- estimated SEM), win probability 98%.
Data set 2 (medians): Candidate A leads by 2.0 +/- 1.33%, win probability 90%.
Generally speaking, using medians gets rid of most of the bias from a single outlier. In this example, the race is taken most of the way out of the toss-up category.
Which brings me to the second consequence. Increased certainty in individual states makes the EV histogram at right more spiky. This is because at any given moment, few states are actually in play. Today it’s IA, VA, and NC, for a total of 2^3=8 major permutations.
Final question: if medians are so great, then why don’t other aggregators like FiveThirtyEight use them? One reason is that intuitively, readers want uncertainty about the future to be baked into the estimate, even if it’s a snapshot of where things are today. Another is that media organizations are under pressure to attract readers, and artificial uncertainty attracts readers. However, to me that seems like spitting in the soup.