Importance Sampling: An Alternative View of Ensemble Learning

• Activity Number: 496
• Type: Invited
• Date/Time: Wednesday, August 4, 2010 : 10:30 AM to 12:20 PM
• Sponsor: Noether Award Committee
• Author(s): Jerome H. Friedman*+ and Bogdan Popescu
• Companies: Stanford University and Stanford University

Abstract

In statistical learning one has data consisting of values of an outcome variable each of which is associated with joint values of a set of predictor variables. The goal is to use these data to derive a function of the predictor variables that can be used to estimate unknown outcome values for future joint predictor values. Ensemble methods have emerged as being among the most powerful statistical learning techniques. It is shown that many of the popular ensemble methods can be viewed from the perspective of of high-dimensional numerical integration. In particular, bagging, boosting, and Bayesian model averaging are seen to correspond to Monte Carlo integration methods each based on different importance sampling strategies. This interpretation explains some of their properties and suggests modifications to them that can improve their accuracy and especially their computational performance.