Bias in Estimation and Inference in Assessing the Incremental Value of New Biomarkers

• Activity Number: 341
• Type: Contributed
• Date/Time: Tuesday, July 31, 2012 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistics in Epidemiology
• Author(s): Zheyu Wang*+ and Kathleen F Kerr
• Companies: University of Washington and University of Washington
• Keywords: incremental values ; risk prediction ; AUC ; IDI ; overfitting ; cross validation

Abstract

Risk prediction has long been a key component in preventative medicine. Ongoing efforts to improve prediction models and the discovery of novel markers raise interest in quantifying the improvement in risk prediction gained by adding new markers. Estimation and inference based on changes in the c-statistics or measures such as the integrated discrimination improvement (IDI) index are widely adopted in practice. In predictive modeling, issues of overfitting are widely appreciated. However, there seems to be less awareness of such issues when assessing incremental values. The variation in predicted risks is sometimes ignored. We performed simulation studies to demonstrate that useless new markers may erroneously appear to improve prediction if improper methods are used. This bias is magnified when multiple new markers are included. Biases for useful markers and factors that can affect the magnitude of the bias are also investigated. In addition, our simulations also revealed biases in variance estimates based on standard formulae, which lead to incorrect inference. Finally, methods to overcome these problems, including cross validation and bootstrap, are investigated and discussed.