Statistic Behavior of Boosting, Support Vector Machines, and the Like

• Activity Number: 398
• Type: Topic Contributed
• Date/Time: Thursday, August 15, 2002 : 10:30 AM to 12:20 PM
• Sponsor: Section on Physical & Engineering Sciences*
• Author(s): Tong Zhang*+
• Affiliation(s): IBM T. J. Watson Research Center
• Keywords: boosting ; classification ; consistency ; support vector machines ; minimax rate

Abstract

We study how close the optimal Bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function. We show that such a classification scheme can be generally regarded as a (non-maximum-likelihood) conditional in-class probability estimate, and we use this analysis to compare various convex loss functions appeared in the literature. Furthermore, the theoretical insight allows us to design good loss functions with desirable properties.

Another aspect of our analysis is to demonstrate consistency and minimax properties of certain classification methods using convex risk minimization. This study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines.