Optimality of Training/Test Size and Resampling Effectiveness of Cross-Validation Estimators of the Generalization Error

• Activity Number: 614
• Type: Contributed
• Date/Time: Wednesday, August 12, 2015 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Georgios Afendras* and Marianthi Markatou
• Companies: SUNY Buffalo and SUNY Buffalo
• Keywords: Cross Validation Estimator ; Resampling Effectiveness ; Training sample size ; Optimality

Abstract

An important question in constructing cross validation estimators of the generalization error is whether rules could be established that allow the optimal selection of the size of the training set, for fixed sample size n. We define the resampling effectiveness of random CV estimators of the generalization error as the ratio of the limiting value of the variance of the CV estimator over the estimated from the data variance. The variance and the covariance of different average test set errors are independent of their indices, thus, the resampling effectiveness depends on the correlation and the number of repetitions used in the random CV estimator. We discuss the statistical rules to define optimality and obtain the "optimal" training sample size as the solution of an appropriately formulated optimization problem. We show that in a broad class of smooth loss functions, and in particular for the q-class of loss functions, when the decision rule is the sample mean the problem of obtaining the optimal training sample size has a general solution independent of the data distribution. The analysis offered when the decision rule is regression illustrates the complexity of the problem.