Pandemic forecasting: Predicting events with the help of machine learning

Researchers and data scientists at The Florey have found a way to harness artificial intelligence (AI) to improve people's accuracy at forecasting future events.

The team used AI to enhance the accuracy of crowd-sourced forecasts on a prediction market platform. The team's paper, published in eBioMedicine, reports that the resulting human-machine hybrid model and was more accurate than humans alone for predicting COVID-19-related events.

The paper's senior author, Professor Anne-Louise Ponsonby, said quality forecasting is key to good decision-making.

"Getting an accurate picture of what to expect in the future is important whether we're responding to a pandemic, election results or the economy. The COVID-19 pandemic has highlighted that not only is forecasting a difficult activity, but forecasting related to public health outcomes is particularly challenging."

Professor Ponsonby said prediction markets, which use the wisdom of crowds to predict specific outcomes, have previously outperformed other forecasting methods such as surveys, expert panels and polls in in some studies. The team analyzed deep data from a database of questions on COVID-19 posed on the Almanis forecasting platform run by Dysrupt Labs.

"We used artificial intelligence to detect forecasters' characteristics, patterns and past performances to generate a real-time score of their likely prediction accuracy or 'trade quality' on the prediction market. We gave extra weight to the better forecasts, which led to even more accurate results," Professor Ponsonby said.

This method resulted in improved event prediction across several independent datasets including the Next Generation Social Science Program.

The two systems usually agreed in their predictions, but when they disagreed the hybrid model was likely to outperform the human-only model.

For example, when the two forecasts disagreed by 5 or more percentage points on event likelihood, the Area Under the Curve (AUC) accuracy score was 0.90 for the hybrid model compared to 0.77 for the human-only model (a score of 1 on this metric indicates perfect prediction, while a score of 0.5 is equivalent to chance).

Lead author, Florey data scientist Alex Gruen, said the hybrid approach is likely to be particularly useful for forecasting events or risks where there are no established data sources or there are significant uncertainties relating to human action.

"Every day, people make countless decisions on the individual and group level based on the likelihood of future events," Mr. Gruen said.

"This hybrid model is one way to improve accuracy of predictions and has the potential to improve our responses to emerging risks such as pandemics or climate change," he said.