On the Margins Distribution in Boosting and Ensemble Performance

• Activity Number: 343
• Type: Contributed
• Date/Time: Tuesday, August 5, 2014 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Waldyn Martinez*+ and J. Brian Gray
• Companies: Miami University and University of Alabama
• Keywords: Generalization Error ; Ensemble Performance ; Random Forests ; Large Margin Classifiers

Abstract

Boosting refers to methods that create a sequence of classifiers that perform at least slightly better than random (weak learners) and combine them into a highly accurate ensemble model (strong learners) through weighted voting. There is sufficient empirical evidence to suggest that the performance of boosting methods is superior to that of individual classifiers. To explain the successful performance of boosting methods, Schapire et al. (1998) developed a bound based on the margins of the training data, from which they concluded that larger margins should lead to lower generalization error. Martinez and Gray (2014) showed that simply increasing all of the margins is not sufficient for improving ensemble performance. Shen and Li (2010) and Xu and Gray (2011) provide evidence suggesting that generalization error might be reduced by increasing the mean and decreasing the variance of the margins, which we refer to as "squeezing" the margins. In this article, we evaluate the role of margins in boosting and also propose several alternative techniques for squeezing the margins.