Nate Silver: Public Policy Polling "the biggest herders in the business"

Nate Silver:

It’s time to stop worrying about outliers and start worrying about inliers. Earlier this year, my colleague Harry Enten documented evidence of pollster “herding”— the tendency of polling firms to produce results that closely match one another, especially toward the end of a campaign.1 What’s wrong with the polls agreeing with one another? The problem is that it’s sometimes a case of the blind leading the blind...

A few pollsters are shameless about their herding. One of them is Public Policy Polling (PPP), an polling firm that conducts automated polls for both public consumption and for liberal and Democratic clients.

Take a look at this exchange, for example, between The New York Times’ Nate Cohn13 and PPP’s Tom Jensen. Cohn discovered that in 2012, the racial composition of PPP’s polls was correlated in an unusual way with President Obama’s performance among white voters in their surveys. If Obama was performing especially poorly among whites in one PPP poll, it tended to have a higher share of nonwhite voters, which boosted Obama’s result. And if Obama was doing relatively well among whites, PPP projected less nonwhite turnout, keeping his lead in check. As a result, PPP’s polls tended to show an unusually steady race between Obama and Mitt Romney.

I’m picking on PPP for a reason: They’re the biggest herders in the business...

On average, in states with at least three other recent polls, their polls deviated from the polling average by only 1.6 percentage points. The evidence for herding is extremely clear visually and statistically...

The problem is simple enough to diagnose: When pollsters herd, if the first couple of polls happen to get the outcome wrong, subsequent ones will replicate the mistake. I wonder if these dynamics explain the poor performance of the polls in some states this year. In Kansas, the final polls showed the independent candidate, Greg Orman, ahead in the Senate race by an average of 1 percentage point to 2 percentage points, but the Republican, Pat Roberts, won by 11 percentage points instead.

The first polls conducted of Kansas after the Democratic candidate dropped out of the race were from Public Policy Polling and showed Orman ahead by 10 points. PPP’s polls can be highly inaccurate when they don’t have other polls to herd toward. In this case, however, other pollsters may have herded toward PPP, producing an incorrect consensus about the race.17

This may also be part of why the polls have frequently proved to be “skewed” toward either Democrats or Republicans. In 2012, there was a significant bias toward Republicans and in 2014 a significant one toward Democrats...

So … what to do about it? If you’ve read this far, you’re undoubtedly highly interested in polling. So my message for fellow polling geeks is as follows: Let’s not give pollsters so much grief the next time they publish what looks to be an “outlier.” Polling data is noisy and polling is becoming more challenging. The occasional or even not-so-occasional result that deviates from the consensus is sometimes a sign the pollster is doing good, honest work and trusting its data. It’s the inliers — the polls that always stay implausibly close to the consensus and always conform to the conventional wisdom about a race — that deserve more scrutiny instead.