An Out-of-bag Method for Regularizing Boosted Regression

• Activity Number: 96
• Type: Invited
• Date/Time: Monday, August 9, 2004 : 10:30 AM to 12:20 PM
• Sponsor: SSC
• Author(s): Greg Ridgeway*+
• Companies: RAND Corporation
• Keywords: nonparametric regression ; boosting ; out-of-bag estimation ; regularization

Abstract

Selecting the optimal number of iterations is a key step when using any of the various flavors of boosting. The standard practice for estimating the optimal number of iterations is to leave out some fraction of the dataset as a test set and iterate the boosting algorithm until the predictive performance on the test set no longer improves. This practice, however, allocates a large part of the dataset for estimating the optimal number of iterations diluting the amount of information available for building the model structure. A variation on out-of-bag estimation can provide an approximately unbiased estimate of the improvement in generalization error attributable to the current iteration without decreasing the amount of data available for learning the model structure. When the out-of-bag estimate of improvement is zero, the iterations stop. The R package "gbm" implements this technique and I demonstrate on several datasets that this procedure offers a fully automated process for fitting boosting models, requiring fewer iterations and offering improved predictive performance.