←Back

538 – Methods for Risk Prediction

Evaluating Probability Forecasts

at Université René Descartes

Mostly employed in medicine and epidemiology research, the topic we discuss is of interest in any area that is concerned with covariate-based risk evaluation. A variety of indices, such as IDI (Integrated Discrimination Improvement), NRI (Net Reclassification Improvement), the predictiveness curve, the area under the ROC curve and difference in PEV (Proportion Explained Variation), that compare two models' predictive capacity are routinely used. They are often used however without adequate inferential tools. We provide such tools for the IDI and the Brier Improvement Score when model parameters are estimated and indices are computed on the same data, i.e., when predictiveness is evaluated within the sample. We chose these two indices as the first measures discrimination and the latter mostly evaluates calibration differences between the two models. We show that both sample indices are consistent for estimating their respective population parameters, and both are root-n asymptotically normal, as long as the associated population index is not null. We evaluate the actual coverage of our normal confidence intervals for both indices via extensive simulation. We also compare these to the coverage afforded by the percentile non-parametric confidence intervals. This is accomplished by simulating 500 samples and doing a Bootstrap confidence interval on each sample. We discuss our results in the context of existing work in Epidemiology, paying special attention to the zero-index case when two models cannot be distinguished by the index under discussion.