Complex Method Selection Using Hierarchical Bayesian Models

• Activity Number: 173
• Type: Contributed
• Date/Time: Monday, August 4, 2003 : 2:00 PM to 3:50 PM
• Sponsor: Section on Bayesian Stat. Sciences
• Author(s): Roman Shapovalov*+ and James R. Whiteley
• Companies: Oklahoma State University and Oklahoma State University
• Keywords: method selection ; hierarchical models ; MCMC ; Bayesian models

Abstract

In some cases, complex data-driven models meeting several performance criteria are needed. In pattern recognition, such criteria may be some minimum required specificity and selectivity of the classifier. The common model selection criteria, e.g., the AIC, may not be suitable because they optimize only a single performance criterion, a goodness of fit, penalized by the model size. It is also assumed that there is no heterogeneity among the training and operational data due to some changes of the modeled object in the course of the model operation. Our proposed approach is based on the MCMC hierarchical Bayesian modeling of the model performance parameters as a function of different problem factors and the selected model components. The approach is designed to choose a model that has the highest chance of meeting all the performance criteria simultaneously and it takes into account the heterogeneity among the training and operational data for the selected model by including the uncertainty due to the heterogeneity into additional priors. This approach has been intended for choosing quality control and fault detection methods, but it can be used for other practical applications.