Ann Selzer is better at election forecasting than Nate Silver

Introducing the Selzer+Silver Model

As I write this, the 2022 midterm elections are less than a month away. FiveThirtyEight has started publishing its daily simulations of the elections, which means we can use the Selzer+ model to generate predictions, not just analyze elections after the fact. After all, If I truly believe that Ann Selzer is better at forecasting elections than Nate Silver, then I should trust her predictions for the upcoming election more than I trust FiveThirtyEight’s. Which means it’s time for me to put money where my mouth is and make some quantifiable predictions.

So I did that. Here are the latest Selzer+Silver predictions for the 2022 midterms:

The code for this model is all on GitHub, and the predictions are updated daily.

For the forecast, instead of using Ann Selzer’s final poll (which hasn’t been released yet), we’re using her most recent poll of the Iowa Senate race to re-weight FiveThirtyEight’s national predictions. Otherwise, the methodology for the forecast is exactly the same as for our retrospective analyses:

1. Take the most recent Selzer & Co./Des Moines Register poll of the Iowa senate race.
2. Calculate the Republican 2-party vote share in the Iowa senate race from the poll
3. Use the binomial distribution to calculate the uncertainty in the result
4. Weight each of FiveThirtyEight’s daily national simulations based on how closely it matches the poll result.
5. Profit.

And just in case there’s any doubt, here are my pre-registered hypotheses for how the model will perform in the upcoming election:

The Selzer+Silver model will have a lower Brier score for the 2022 Senate elections than the FiveThirtyEight model, and will more accurately predict the number of House seats won by each party

What’s the probability this is just noise?

There’s a chance that Ann Selzer isn’t magic and has just gotten lucky the past couple of election cycles. After all, there are many pollsters and one of them has to be the best. It would be silly to take this too far before checking to see if this is likely to be a fluke.

Before we start looking at some actual numbers, I want to point out two qualitative factors in favor of this being something real:

1. This wasn’t an exercise in data mining. I didn’t start by looking at all the polls and asking how you could pick and choose which ones to use to get the best results. I started with the question of how well you could do at political forecasting just by using Selzer & Co. polls.

2. Ann Selzer has been extremely accurate for a very long period of time. Politico talked about “Ann Selzer’s secret sauce” in 2015 and FiveThirtyEight called her the “best pollster in politics” in January 2016. If her earlier predictions were just due to chance, we’d expect her to suffer from reversion to the mean. Instead, her results in 2016 and 2020 were some of her most accurate results.

But we should try to be at least somewhat quantitative. Probably the simplest analysis we can do is to calculate the probability that a single pollster could consistently outperform the average of all the polls by chance alone. Specifically we want to know the probability that a pollster could outperform the average at least 8 times in 12 races. I’ll make the simplifying assumptions that Ann Selzer always polls 800 people, that FiveThirtyEight’s Iowa predictions are equivalent to an average of 6 polls of 800 people each, and the error in polls is fully characterized by their sampling error. If you run a million simulations of 12 difference races, you get the following results:

Out of 1,000,000 simulations there were only 1,800 times when a single poll outperformed the aggregate 8 or more times. That sounds like it should be a p-value of 0.0018, but we probably need to apply a correction factor for multiple comparisons, since we’d be equally interested if any pollster consistently outperformed the average. If we look in the FiveThirtyEight poll database, we find that 32 pollsters have run polls in 7 or more election cycles since FiveThirtyEight started in 2008. If we take 32 as our multiple correction factor, our adjusted p-value is suddenly 0.06. But maybe that’s unfair in the other direction, since Ann Selzer was already lauded as the gold standard for polling before the 2016 election. If we only look at her results since 2016, she’s outperformed the aggregate 4 times in 5 elections, with a p-value of 0.013 (with no correction for multiple comparisons).

All that aside, I don’t think we should apply too much weight to either the extremely low p-value without a multiple comparison correction or to the non-significant p-value with the correction. The exact details of how you set up the calculation matter a lot, and there were lots of other questions I could have asked instead. The right thing to do here would be to perform a full Bayesian analysis of the situation, but a) that sounds like way too much work, and b) I have no idea what any of the priors for the parameters should be.

For now I’m inclined to leave my conclusion as: Ann Selzer has outperformed the polling average for long enough that it’s unlikely (but still possible) that it’s a statistical fluke.

Why does this matter?

There are a number of potential takeaways from this.

First, I want you to appreciate how extremely odd this is. Since 2008, we’ve been told over and over that the one true way to forecast an election is by looking at reputable polls, averaging them, and allowing sophisticated people to build even more sophisticated models translating those poll averages into probabilistic forecasts. Under no circumstances should you ever look at a single poll, especially if it’s an outlier. Think I’m exaggerating? Here’s a recent article from the Washington post on how to interpret polls, which features comments from G. Elliott Morris (the Economist) saying to “Aggregate polls together” and Nathaniel Rakich (FiveThirtyEight) saying “don’t pay too much attention to any single poll, especially if it’s an outlier”. Here’s Nate Silver saying “When you see a poll that looks like an outlier, just throw it into the average”.

But here’s a pollster whose polls are consistently outperforming the average! Whose polls of the Iowa presidential race in both 2016 and 2020 were massive outliers, but ended up being right on the money! That goes against everything that political forecasters have been telling us for 14 years!

Second, poll aggregation is hard! Yes, it sounds simple - just take all the polls and average the results - but there are a lot of nuances you have to get right. Start with a super basic question: what is the uncertainty of each poll? The binomial distribution says a poll of 800 votes should have an uncertainty of about 1.8%, but in practice the actual error is often double that. If you want to really understand a poll’s uncertainty, you need to take into account how the raw survey results were weighted. But those numbers aren’t always reported by pollsters, so it might not even be possible for you to calculate the uncertainty.

But even if you can calculate the uncertainty of each poll, you still have to figure out how the errors between polls are correlated. If two polls are perfectly uncorrelated, then averaging them together should reduce your error by 30%. But if the polls are perfectly correlated, then averaging them together doesn’t reduce your error at all. So what determines if two poll results are correlated? Does it depend on the polls’s methodology? Where the poll was conducted? Who conducted the poll? Who the candidates are? Whether there’s currently an ongoing pandemic? The list of possible hypotheses is significantly larger than the number of election cycles we have to evaluate.

Third, (and this is way out of my area of expertise) this suggests to me that there’s so much we don’t know yet about how to conduct polls or forecast elections. Ann Selzer literally is the best pollster in politics, according to FiveThirtyEight’s analysis, and she does it without following any of the rules for good polling!

Here’s one example: Among election forecasters, there’s basically universal agreement that polls that weight by education are better than ones that don’t. That includes Nate Silver, G. Elliott Morris, Nate Cohn, Dave Wasserman, and everyone’s favorite neoliberal shill.

Selzer and Co. polls are not weighted by education. Her poll results are weighted only by “age, sex, and congressional district”. In comparison, the latest New York Times/Siena College (another A+ pollster according to FiveThirtyEight) national poll was weighted by party, age, gender, marital status, home ownership, turnout history, vote method in 2020 election, census block group density, educational attainment, region, and metropolitan area.

So how come Selzer & Co. polls are so accurate, if they don’t weight by education? Why don’t they suffer from the same Democratic bias that affected other polls in 2016 and 2020? Is her approach replicable? If I tried to conduct my own poll in Iowa following her methodology as closely as possible, could I match her performance?

And if polling could be improved, then poll-aggregation can also be improved. Over 12 elections in Iowa, FiveThirtyEight’s average error was 2.55%. That’s actually worse than you would expect from a single ideal poll of 800 voters! To me, that suggests that poll aggregators might have actually become too focused on their error bars and not enough on their point estimates. This isn’t just me speculating either - here’s Nate Silver in 2014:

The FiveThirtyEight model produces probabilistic forecasts as opposed to hard-and-fast predictions. In the same sense a weather forecaster might tell you there’s a 20 percent chance of rain tomorrow, the FiveThirtyEight model might estimate the Democrat has a 20 percent chance of winning the Senate race in Kentucky.

My view is that this is often the most important part of modeling — and often the hardest part. Predictions of the most likely outcome (“The Democrat will win the race by 3 percentage points”) are sometimes relatively immune to changes in methodology. But probabilistic forecasts can be very sensitive to them. Does that 3-point lead translate into a 60 percent chance of winning? Or a 95 percent chance? Or what?

Source

While the error bars are undoubtedly important, and are what generally separates the rigorous forecasts from the fluff, we should remember that the point estimates are what drives the topline result. Take 2016, where all the forecasters predicted a Clinton win, and the differences among them came down to their error bars. Do you know what would have been better than putting bigger error bars on the election forecasts? Having your point estimate be a Trump victory!

Now, I don’t have any magic solutions to propose (if I did, I’d be working on becoming the next Nate Silver), but it seems like we should be thinking a little bigger than “conduct polling the way we’ve done for the past 80 years, but now weight by education”. Maybe we should be counting Google search traffic? Or convening a set of super-forecasters to make predictions about the election? Or subsidizing a prediction market about it? Or conducting a poll with an absurd sample size using random digit dialing in June and November, and have a focus group that you use to interpolate in between. Or ignoring polls entirely and hiring a team of political scientists to come up with their best “fundamentals”-based predictions? Or hiring people to stand on a street corner with a shirt that says “tell me your opinion”? Or maybe it’s to give GPT-3 the prompt “The next president of the United States is” and see what it comes up with.²

Then again, most of these ideas are clearly stupid, and the ones that aren’t obviously misguided would take a large amount of resources. It just seems like somebody should be looking into this! Maybe somebody is already and I haven’t heard about it because they’re either just getting started or they work for a political campaign and haven’t made their results public.³

I don’t know what the future of election forecasting is going to look like, but I can’t believe that FiveThirtyEight’s current approach is the best that we can do.