Predictive Accuracy Measures for Binary Responses: Relationships and Impact of Incidence Rate (303244)

Keywords: auc, brier score, predictive accuracy, incidence, logistic regression

Evaluating the performance of models predicting a binary outcome can be done using a variety of measures. While some measures intend to describe the model's overall fit, others more accurately describe the model's ability to discriminate between the two outcomes. If a model fits well (Brier Score or H-L test) but doesn't discriminate well (c-statistic or auc) what does that tell us? Given two models, if one discriminates well but has poor fit while the other fits well but discriminates poorly which of the two should we choose? We investigate the relationships among the measures of discrimination and overall fit using simulation studies under general conditions and also under controlled incidence rates. The measures of interest in our simulation studies include the area under the ROC curve (auc), Brier Score, discrimination slope, log loss and r-squared. The results of the simulations provide an insight in to the relationships among the measures and raise concern for scenarios when the measures may provide different conclusions. We also have included a proof to explain the relationship between the log loss and Brier score.