Comparison of Estimation Approaches in Classification Analysis and Assessment of Model Stability in Variable Selection Using Bootstrap Resampling

• Activity Number: 244
• Type: Contributed
• Date/Time: Tuesday, August 5, 2003 : 10:30 AM to 12:20 PM
• Sponsor: Section on Nonparametric Statistics
• Author(s): Joni A. Nunnery*+
• Companies: Louisana State University
• Keywords: classification analysis ; variable selection ; bootstrap ; replicability ; validity ; Akaiki Information Criterion

Abstract

This study assessed the performance of seven classification approaches for 10 predictors at sample sizes 50, 100, 200, and 400. Linear, quadratic, and logistic classification approaches were reviewed; variable subsets were selected based upon classification rates. Linear analyses were rerun using the Akaiki information criterion (AIC) for variable selection. Nearest-neighbor, kernel density function, and stepwise analyses were also reviewed. For a specific approach, three sets of analyses were conducted on a single sample of fixed size: a best subset analysis; best subset analyses on 500 bootstrap samples selected from the single sample; and a best subset analysis where the single sample estimates were applied to the population and the best subset was based upon the classification rate. A modeling approach replicated if the subset selected most often in the bootstrap samples matched the best subset in the single sample; validity was established if the subset selected in the single sample analysis performed best in terms of population classification. Results showed that linear classification using the AIC consistently performed best in terms of both replicability and validity.