Efficient Cross-Validation for Tuning and Ranking Cheminformatics Models

• Activity Number: 356
• Type: Invited
• Date/Time: Wednesday, August 1, 2007 : 8:30 AM to 10:20 AM
• Sponsor: Section on Physical and Engineering Sciences
• Author(s): William J. Welch*+ and Hui Shen
• Companies: University of British Columbia and University of British Columbia
• Keywords: Cross validation ; Drug discovery ; Model selection ; Model tuning ; Prediction performance

Abstract

Cross-validation (CV) is widely used for assessing prediction performance of a model, for tuning a model with respect to user-selected parameters, and for selecting a "best" model. Our NIH Exploratory Center for Cheminformatics Research (ECCR) is currently comparing 15 statistical modeling methods and five sets of explanatory variables for modeling a number of biological assays. Among the 15 statistical methods, K-nearest neighbors requires tuning of K, the number of neighbors, for example. By exploiting the pairing that occurs in comparing one method (a given model and explanatory variable set, with specified values for all tuning parameters) with another we are able to achieve high power to discriminate between their two performances. This adaptive and computationally efficient methodology to find the best method will be demonstrated on cheminformatic data sets from PubChem.