A Perturbation Method for Prediction Accuracy with Regularized Regression

• Activity Number: 241
• Type: Contributed
• Date/Time: Monday, July 30, 2012 : 2:00 PM to 3:50 PM
• Sponsor: Biometrics Section
• Author(s): Jessica Minnier*+ and Tianxi Cai
• Companies: Harvard University and Harvard University
• Keywords: prediction ; high dimensional data ; statistical genetics ; perturbation-resampling ; regularized regression ; variable selection

Abstract

Analysis of massive ''omics'' data often seeks to identify a subset of genetic markers that are predictive of disease outcomes. Traditional statistical methods for variable selection often fail in the presence of high-dimensional features. Classification algorithms based on genetic and biological markers have been developed for prediction of clinical outcomes. Robust regularization methods can achieve an optimal trade-off between the complexity of the model by simultaneously performing variable selection and estimation, leading to more accurate prediction models. However, in finite samples, it remains difficult to evaluate the predictive performance of such models. Estimates of accuracy measures such as absolute prediction error and AUC statistics may be imprecise, especially in the small study setting when cross-validation procedures are used, and hence model comparison is challenging. We propose perturbation resampling based procedures to approximate the distribution of such prediction accuracy measures in the presence of regularized estimation. This method provides a simple way to estimate confidence regions for the true prediction accuracy of a model.