Variance Estimation of Crossvalidation Estimators of the Generalization Error

• Activity Number: 51
• Type: Invited
• Date/Time: Sunday, August 7, 2005 : 4:00 PM to 5:50 PM
• Sponsor: Biometrics Section
• Author(s): Hong Tian*+ and Marianthi Markatou and Shameek Biswas and George Hripcsak
• Companies: Columbia University and Columbia University and Columbia University and Columbia University
• Keywords: cross-validation ; generalization error ; moment approximation ; prediction ; variance estimation

Abstract

An important aspect of algorithmic performance is the generalization error. The estimation of the generalization error is usually done via bootstrap or crossvalidation. However, providing an estimator of the variance of the crossvalidation (CV) estimator of the generalization error is a more difficult problem, particularly various sources of variability are taken into account. We look at the problem of variance estimation of the CV estimators of generalization error as a problem in approximating the moments of a statistic. The approximation illustrates the role of training and test sets in the performance of the algorithm. It provides a unifying approach to evaluation of various methods used in obtaining training and test sets. It takes into account the variability due to different training and test sets. For the simple problem of predicting the sample mean and in the case of smooth loss functions, we derive our moment approximation estimator and compare it with the one proposed by Nadeau and Bengio (2003). We extend these results in the regression case and the case of absolute error loss. We illustrate the results through simulation.