Combining models to improve COVID-19 forecasts

Soon after the beginning of the pandemic, researchers from the Heidelberg Institute for Theoretical Studies (HITS) and the Karlsruhe Institute of Technology (KIT) developed a web platform for bundling short-term forecasts of COVID-19 cases and deaths in Germany and Poland with contributions by numerous independent modeling teams. Now, these researchers have published their insights from a systematic evaluation study in the open access journal Nature Communications. The results: Combining different models leads to better predictions, but predictions regarding COVID-19 remain highly challenging.

Sometimes, you can obtain a clearer picture by combining different approaches in a collaborative way. Scientists from the German-Polish COVID-19 Forecast Hub took this approach in evaluating the strengths and weaknesses of forecasting models and assessed whether combined ensemble forecasts can improve upon single-model predictions.

The hub: A transparent and fair comparison

"We created a comparison platform for different models and fostered the exchange between modeling teams with the goal of improving individual models and the overall combined prediction," states Melanie Schienle, professor of Statistics and Econometrics at KIT and currently a guest scientist at HITS, who leads the project team together with Tilmann Gneiting from the Computational Statistics group at HITS. The Forecast Hub follows an open science approach. It evaluates forecasts and preserves the entire history of the model outputs. "Models often look a bit better than they are when applied in retrospect, and we wanted to avoid that pitfall," adds Johannes Bracher (HITS and KIT), the first author of the study. "We publicly specified the rules of the comparison in advance, similar to in pharmaceutical or medical trials."

More than the sum of its parts: Ensemble forecasts perform better

The study period on which the team reported in Nature Communications covers the onset of the second wave in Germany and Poland from 12 October to 19 December 2020. This period is characterized by tightening non-pharmaceutical interventions, which subsequently led to a decrease in cases in Poland and a plateau and renewed increase in cases in Germany. Thirteen independent teams provided probabilistic real-time forecasts of COVID-19 cases and deaths for lead times of one to four weeks. Overall, ensemble forecasts showed good relative performance—especially in terms of uncertainty quantification—and were typically on par with the best single-model predictions. Moreover, their performance was less variable across different weeks.

"Thus, we can clearly see the benefit of this collaborative approach, which provides more stable performance," says Johannes Bracher. "It´s like the famous quote by Aristotle: "The whole is more than the sum of its parts.'"

Forecasts: A matter of uncertainty

The evaluations also highlight the fact that COVID-19 remains highly challenging to predict, and ensemble prediction is also sometimes off target. "Overall, the predictions were very heterogeneous across models, so different models did not at all agree about what was going to happen next," Melanie Schienle adds. "By considering the different forecasts together, we can gain a more realistic assessment of the degree of uncertainty."

The German and Polish COVID-19 Forecast Hub was inspired by the US COVID-19 Forecast Hub, a US project in which Johannes Bracher and Tilmann Gneiting have been actively involved from its launch. Since March 2021, the German and Polish COVID-19 Forecast Hub has gradually been integrated into a new European platform—the European COVID-19 Forecast Hub—run by the European Centre for Disease Prevention and Control and the London School of Hygiene and Tropical Medicine.

The study has been published in Nature Communications.