|
Abstract:
|
In 1974 Mervyn Stone described how to use leave-one-out cross-validation (LOOCV) both to select model or classifier parameters and to produce an almost unbiased assessment of their predictive capabilities, given the selection. This method required two layers of CV, the inner layer being used for selection and the outer layer for assessment. Two-layer CV and related techniques resample the available data intensively to improve accuracy, which is particularly attractive for small samples. Unfortunately, many researchers seem unaware of these methods, instead using either the less accurate holdout (test set) method or a single layer of CV for both selection and assessment, leading to bias. I will give two examples of the benefits of the two-layer approach. The first involves the classification of gene expression data and the second aims to automate diagnosis of sleep apnea.
|
