Multiple simulations best for COVID-19 predictions

Computer modelling used to forecast COVID-19 mortality contains significant uncertainty in its predictions, according to a new study led by researchers at UCL and the CWI institute in the Netherlands.

The authors of the study, performed for the Royal Society's RAMP initiative for Rapid Assistance in Modelling the Pandemic, highlighted that however well constructed such models are, they are only ever as robust as the "input" parameters—which include highly uncertain factors relating to how the disease is spread.

The authors said the models should be regarded as "probabilistic" rather than being relied upon to produce a particular and specific outcome. They maintained that future forecasts used to inform government policy should provide the range of possible outcomes in terms of probabilities to provide a more realistic picture of the pandemic framed in terms of uncertainties.

Professor Peter Coveney (UCL Chemistry), who leads the EU H2020 Computational Biomedicine Centre of Excellence as well as the EU VECMA programme on uncertainty quantificationthat undertook the study, said: "There is a large degree of uncertainty in the modelling used to guide governments' responses to the pandemic and this is necessary for decision makers to understand.

"This is not a reason to disregard modelling. It is important that these simulations are understood in terms of providing a range of probabilities for different outcomes, rather than a single fixed prediction of COVID-19 mortality."

"Because of this uncertainty, future forecasts of the death rates of COVID-19 should be based not on an individual simulation, but on lots of different simulations of a code, each with slightly adjusted assumptions. Predictions based on this method, though still highly uncertain, will provide a more realistic picture of the pandemic."

In the study, currently available as a preprint (www.researchsquare.com/article/rs-82122/v3), the researchers sought to determine the level of uncertainty in predictions made by COVIDSim, a sophisticated code developed by Professor Neil Ferguson's team at Imperial College London.

Using a supercomputer located in Poland, they adopted a technique, now standard in weather forecasting and climate science, of performing a large number (known as ensembles) of simulations with varying sets of initial conditions. Inputs include the assumed effectiveness of proposed medical interventions, factors relating to how COVID-19 is spread, and factors relating to population distribution in the UK.

The research team found that adjustments to the input parameters were amplified by up to 300% in the outputs (i.e. the predictions) – meaning that slight differences in, say, the assumed effectiveness of social distancing could lead to larger changes in the simulation's predictions. This is important as the inputs—the knowledge of the state of the pandemic and behaviour of the population—have a significant degree of uncertainty themselves.

They also found that, although the code contained 940 parameters, 60 were important and, of those, only 19 dominated the variance in the output predictions. Half of the overall variation in their results was down to just three of the 940 input parameters (the disease's latency period, the delay in an infected person self-isolating, and the effectiveness of social distancing).

Professor Coveney added: "Our findings are important for government and healthcare policy decision making, given that COVIDSim and other such epidemiological models are—quite rightly—still used in forecasting the spread of COVID-19. Like predicting the weather, forecasting a pandemic carries a high degree of uncertainty and this needs to be recognised.

"Finally, our modelling has only been possible because Professor Neil Ferguson's team open sourced their code. Not all models being used in Government briefings are open in that way. We urge other research groups to follow Imperial's lead and adopt an open science approach."

To test the robustness of COVIDSim, the research team—after discussions with the Imperial College team—selected 60 of the most critical input parameters and adjusted them by increments of up to 20%, analysing how these adjustments affected the predictions.

The study was carried out by researchers at UCL, CWI, the University of Amsterdam, Brunel University London, and the Poznan Supercomputing and Networking Centre in Poland. The Royal Society's RAMP initiative seeks to bring modelling expertise from a diverse range of disciplines to support the pandemic modelling community already working on COVID-19. The availability of powerful computers is critical to be able to conduct this kind of research. Professor Coveney said: "We are more dependent than ever on computer simulations and they will go up another gear in the next generation of so-called exascale supercomputers."