339 – Statistical Models Applied to Defense and National Security Issues
A Sequential Procedure for Aggregation of Expert Judgment Forecasts
John Irvine
Draper Laboratory
Srinivasamurthy R. Prakash
Draper Laboratory
Drazen Prelec
MIT
John Regan
Draper Laboratory
Policy makers rely on forecasts from experts to inform the decision making process. Merging or aggregating the judgments from multiple forecasters poses methodological challenges. New research is exploring ways of combining judgments from multiple experts to arrive at a better overall decision. For forecasts involving a small set of possible categorical outcomes where expert judgments accumulate over time, we propose a sequential procedure that chooses the best single forecast as soon as the expert judgments indicate sufficient evidence for the specific outcome. We present the formulation of this approach for binary forecasting problems. Using forecasting data for a set of real world events, we demonstrate and evaluate this new method. Comparing the performance of the proposed method to the standard unweighted linear average from the pool of subjects demonstrates the benefits of this approach. Once the outcome of the forecasting problem is known, the Brier score provides an objective measure of performance. Based on the Brier score, this method outperforms the unweighted linear average across a number of forecasting problems.
