10 Things We Know About The 2020 Polls So Far
It took mere hours from the close of voting for hot take artists to declare the polling industry had failed, is in crisis, and should never be trusted again. As a survey researcher, I’m always jealous of writers who have the privilege of advancing assertions and theories without the burden of actual evidence, but at least in this instance, I’ll agree with the basic premise: on average, the polls were off. In some instances, they were very off.
There’s a science to survey research and an art to political polling, and in attempting to explain why the polls were off, my intent is to be guided by the science, the art, and the data, and to save the armchair psychology for the lay journalists.
Let’s start with what we know so far.
1) We don’t know how inaccurate the polls were.
Until all the votes are counted, we can’t measure the accuracy of the polls precisely. This seems obvious, but I haven’t seen it in the commentary yet. But it’s not enough to say the polls were off, not if we want to figure out why they were off. We need to be able to measure by how much they were off.
2) The polls consistently overestimated the Democratic vote share
Again, this seems obvious, like a mere restatement of the fact that the polls were off, but there’s a difference between polls being off and polls being off consistently in the same direction.
Being off consistently in the same direction is evidence of statistical bias, which is not the same thing as political bias. I’ll explain statistical bias in the second article of this series. For now, it suffices to say that statistical bias means there’s a source, or sources, of error in the polls that systematically leads to overestimates of the Democratic vote share. We need to find that source.
3) This is not a “close election” issue.
We’ll see how the numbers shake out when all the votes are counted, but it’s improbable the polls were off because this was a close election. Firstly, the overestimate of Democratic vote share doesn’t just show up in the presidential polls. It shows up in down-ballot polls as well. Second, we don’t have an election in the United States; each state holds its own elections. And most importantly, when you look at all of those elections for all of those offices, you notice immediately that polls overestimated Democratic vote share in races where the election was close as well as races in which the Republican won in a landslide and in races the Democrat won in a landslide. The correlation between the degree to which the polls were off and the winning margin may prove to be a crucial clue into what went wrong.
4) The margin of error doesn’t explain this.
For the margin of error to explain why an individual poll is off, the race needs to be close. Beyond that, the margin of error concept is a down-and-dirty estimate of sampling error for individual polls. Polls were off in the aggregate, and they were off in the same direction, which is evidence of statistical bias, not merely sampling error.
5) Polls are not politically biased.
Polling is possibly the only profession other than meteorology in which the entire public has the opportunity to know, objectively, if you were right or wrong. In order to believe pollsters skewed the polls to influence the outcome of the election, you need to believe there’s an advantage to being proven wrong in a profession in which accuracy is easily measured and the most important trait a person can have in order to attract clients.
What’s more, reports so far indicate internal polling, which is not for public release, produced similar results as the polling reported in media. If your job is to use polling to guide campaign strategy, of course you’re not going to put your thumb on the scale, and of course your client doesn’t want you to. Your client needs accurate information, and so do you if you want to keep your job.
The truth is there are a lot of pollsters who are wondering how they’ll reassure clients who are asking themselves why they should hire the pollster again after so big a miss.
6) There’s reason to believe this is not an American phenomenon.
It’s debatable whether each of these was the polling error the media has come to consider them. I tend to think they’re overblown when taken individually, but as a series, they could be telling us something: the Brexit polls; the 2016 Pennsylvania, Wisconsin, and Michigan polls; the 2019 Australia polls; and now the 2020 polls.
The first three were narrow misses in close elections (Brexit is more arguably not a miss), and that makes them different from the 2020 polls. Comparing them to 2020 might prove erroneous. But in each of these instances, the polls erred in predicting left-leaning outcomes. Why?
Perhaps it’s coincidence, but perhaps not. If not, the issue with the 2020 polls may not be unique to American political polling or to Trump.
7) There’s more than one polling industry.
If you narrow survey research down to just political pollsters, you still have three segments of the industry: the data collection houses, the media pollsters, and the internal political pollsters.
The data collection houses play an outsized role in data quality relative to their almost non-existent role in survey design and analysis. Reconciling their practices and analyzing the metadata chaff from their work may tell us why the polls were off, yet looking into that kind of metadata is not a common practice. It’s just not something non-academics pay much attention to, but it’s crucial to measuring things like differential non-response, which may be what caused polls to be consistently off.
Meanwhile, media pollsters — meaning the pollsters who produce polls for media outlets — differ in some practices from the political pollsters who work inside campaigns. They have often different opinions on the sampling approach and on survey weighting, both of which and especially the latter may explain what happened. We need to compare media polls with internal polls, and by definition, that’s not possible unless someone makes their internal polls public.
8) We need much greater transparency if we’re going to figure out what happened.
Political polling doesn’t have uniform reporting requirements, and there are multiple steps in the process that are subject to the pollsters’ judgment.
Case in point: do you weight party ID or not, and if so, what do you weight it to, and what are those figures based on? Not every pollster reports that.
In fact, not every pollster reports demographic characteristics at all. If we’re going to figure out what went wrong, we’re going to need that kind of information, and more. Pollsters will need to share not just their practices and reports but also raw data, including metadata, if we’re going to figure out what went wrong and avoid a repeat.
9) It’s doubtful the polls being off influenced the outcome of the election in any meaningful way.
There’s a twin chorus coming out of Washington right now. On the right, figures have been shouting the newly coined term “suppression poll,” while on the left, some are speculating that the polls are related to losing some House races in red districts. At least one journalist is calling for a change in how progressives approach campaign research. Polling gets used to make decisions and to influence narratives. It’s tremendously valuable, but like any single element of an election, it’s rarely determinative.
What the right is alleging is belied by how polling gets used to shape a narrative. Late in an election, campaigns want to show their supporters the campaign is close in order to motivate them to turn out. A failure to turn out is why Brexit passed and Clinton lost Pennsylvania, Wisconsin, and Michigan in 2016, and in all those cases the polling led Remain and Clinton voters to stay home, thinking the outcome would be in their favor anyway..
On the left, it’s possible some campaigns made decisions or allocated resources based on what turned out to be overestimates of the Democratic vote share, but Hispanic political operatives in Florida knew they were losing ground despite the polls, and the House seats Democrats lost were in districts won by Trump by massive margins in 2016. Call it political gravity: after every wave election (like the one in 2018), the party that benefitted from that wave is going to lose seats in districts they never firmly held.
10) It’s quite possible the polls being off will influence the future in a meaningful way.
We know what cold feels like because we know what hot feels like. It’s relative, and right now the narrative is that the progressive agenda was rejected and Joe Biden has no mandate because Democrats didn’t perform as well as expected, and that narrative is all because Democrats underperformed expectations relative to the polls.
Already, there have been calls to scale back the Democratic agenda, and elected Democrats are tearing into one another over whose fault it is they lost a handful of House seats in red-leaning districts, driven largely by this poll-centric narrative.
But Joe Biden set a record for most total votes, is going to win the popular vote by well over four million votes, and is likely to capture 306 electoral votes. He won back the Blue Wall states that gave Trump the presidency in 2016, and he flipped two states that have been red for decades. The Democrats have netted one Senate seat, and though House Democrats lost a handful of seats in districts Trump won, they’re still in the majority because they held on to more seats in districts Trump won.
That’s an impressive set of accomplishments and evidence of a Democratic, if not hard-left, mandate. If the narrative stays focused on Democratic performance relative to the polls, instead of relative to Republicans candidates, polling’s miss will have an impact, and it will favor Republicans.
