Quantifying Bias in Models for Probability Estimation

• Activity Number: 617
• Type: Contributed
• Date/Time: Thursday, August 7, 2014 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistics in Sports
• Author(s): Thomas Flowerdew*+
• Companies: STOR-i
• Keywords: Bias ; Sports ; Prediction ; Probabilities ; Bayes ; Non-parametric

Abstract

Many statistical models take a certain number of inputs, and output an estimate of the probability of some event. This estimate will usually include some measures of its uncertainty. Given a well-designed model, the user would then have all they require to make informed decisions, given these outputs. This confidence in the model is only valid when its outputs are unbiased. Models can however can accumulate bias via various methods, including missing important model parameters and selection bias.

The talk will discuss techniques used in order to take a series of pairs of model outputs; the probability estimate and the event outcome, and use these alone to estimate the magnitude of the bias. A specific Bayesian Hierarchical Model will be compared in its efficacy against a general non-parametric stochastic approximation approach, both in terms of their power of prediction and sensitivity to assumptions. This analysis will be extended in order to cover the situation where the bias changes through time. This work is set within the context of modelling sporting events.