Online Program

Aggregating Statistical Models and Human Judgment to forecast COVID-19 (309773)

Tamay Besiroglu, Metaculus 
David Braun, Lehigh University 
Juan Cambeiro, Metaculus 
Allison Codi, Lehigh University 
*Damon Luk, Lehigh University 
Thomas McAndrew, Lehigh University 

Keywords: COVID19, forecasting, prediction, consensus, ensemble, human judgment

Forecasting the trajectory of the US COVID-19 pandemic can support public health officials who make decisions with partial knowledge of how the virus will evolve over time. Computational models are capable of processing large, structured datasets. However the trajectory of COVID-19 depends on information inaccessible to computational models such as social, economic, and political influences. We aim to collect monthly probabilistic predictions from experts in the modeling of infectious disease and trained, generalist human forecasters about key targets that signal changes in the trajectory of the US COVID-19 pandemic. Predictions from humans will be (i) aggregated into a consensus forecast and (ii) combined with an ensemble of computational models, called a metaforecast. In January we collected 779 human judgment predictions on 7 targets related to COVID-19. We compared forecasts from: (i) an ensemble of computational models called the COVID-19 Forecasthub (Fhub), (ii) a consensus of human predictions and (iii) a metaforecast–a combination of the consensus and Fhub. Preliminary results for predictions of national incident cases and deaths made for the last week in January show the consensus outperformed the Fhub when predicting deaths but the Fhub made a more accurate prediction of incident cases. The metaforecast’s accuracy was between that of the consensus and Fhub. A consensus can provide forecasts of pandemic targets similar in accuracy to an ensemble of computational models. A consensus, as opposed to an ensemble, is not limited by the type of questions asked and we have asked the crowd a diverse set of questions about the pandemic. But because a consensus requires human effort we are limited by the number of questions we can ask. We aim to combine the best of both worlds by drawing on computational models and consensus of human judgment to support how public health officials translate novel information about the virus into mitigation efforts.