|
Abstract:
|
Event forecasting typically involves computing risk scores, estimating conditional probabilities of the event given vectors of predictors. An important property in risk scoring is calibration, which requires that the conditional probability of the event given a risk score equals the risk score itself. The extent to which a risk scoring tool is miscalibrated may be visualized by plotting these conditional probabilities against the risk score, and comparing the resulting curve with the 45-degree line. We propose to estimate this curve using penalized regression spline methods, and define a measure of miscalibration equal to the area between the curves, which we term ABC. We compare ABC with other ways of evaluating risk score calibration, particularly the Hosmer-Lemeshow statistic, observing that ABC is not only more easily interpreted, but also provides higher power to detect miscalibration. Inference about ABC is illustrated using data from a postoperative risk score validation study.
|
