Evaluation of Probability Forecasts: An Empirical Bayes Approach

• Activity Number: 606
• Type: Invited
• Date/Time: Thursday, August 5, 2010 : 8:30 AM to 10:20 AM
• Sponsor: Biometrics Section
• Author(s): Tze Leung Lai*+ and Shulamith Gross
• Companies: Stanford University and Baruch College, CUNY
• Keywords: probability forecasts ; Brier scores ; default probabilities ; loss function ; weather forecasts

Abstract

Evaluation of probability forecasts: A loss function approach Tze Leung Lai and Shulamith Gross, Stanford University, Stanford, CA and Baruch College, New york, NY

Probability forecasts are routinely used in weather and climate predictions, in forecasting default probabilities of bank loans and in estimating the risk of disease or disease outcome based on prognostic biomarkers. Scoring rules have long been used to assess the performance of probability forecasts. We describe here a new approach that uses the loss function of the true and predicted probabilities instead. Since the true probabilities are unknown, we show how the losses can be estimated and provide confidence intervals for the estimates. Asymptotic theory and simulation studies are provided for this approach and the methods are applied to weather forecast data.