|
Activity Number:
|
361
|
|
Type:
|
Topic Contributed
|
|
Date/Time:
|
Tuesday, August 4, 2009 : 2:00 PM to 3:50 PM
|
|
Sponsor:
|
Section on Statistics in Epidemiology
|
| Abstract - #304403 |
|
Title:
|
Estimating Cross-Validation Variability
|
|
Author(s):
|
Waleed A. Yousef*+ and Weijie Chen
|
|
Companies:
|
Helwan University and FDA
|
|
Address:
|
Ain Helwan (university campus), Helwan, International, 11795, Egypt
|
|
Keywords:
|
Classifiers ; Assessment ; Cross-Validation ; K-fold Cross Validation ; Repeated Cross Validation ; Variance Estimation
|
|
Abstract:
|
K-fold Cross-Validation (KCV) is used by many practitioners for assessing the performance of classifiers. Repeating the KCV R times, by shuffling the data each time, is called ``Repeated K-fold CV" (RKCV). At best of our knowledge, the only method available in the literature for estimating the variance of the RKCV is the sample variance; where we have R different values of performance measures obtained from the R repetitions of KCV. This ignores the correlations among these R performance estimates. We propose a new method for estimating the variance of RKCV that is asymptotically unbiased. This paper is a simulation study to assess this new method by comparing it with the true variance (obtained from Monte-Carlo trials) and the method that uses the sample variance.
|
