Sophisticated vs. Naive Bayes: Is It Worth the Effort?

• Activity Number: 119
• Type: Topic Contributed
• Date/Time: Monday, August 10, 2015 : 8:30 AM to 10:20 AM
• Sponsor: Section on Nonparametric Statistics
• Author(s): Ivan Mizera*
• Companies: University of Alberta
• Keywords: Empirical Bayes ; Kiefer-Wolfowitz NPMLE ; Mixture models ; Naive Bayes ; de Finetti theorems ; Convex optimization

Abstract

The properties of the iterative algorithm for the estimation of mixing distribution, in the spirit of Kiefer and Wolfowitz nonparametric maximum likelihood estimator for mixture models, are investigated. The idea of the algorithm, as outlined in Koenker and Mizera (2014), is based on the Lagrange dual to the original infinite-dimensional formulation; this dual is a finite-dimensional problem, with infinite-dimensional constraint whose structure allows for successive finite-dimensional approximations. Various concrete schemes of the algorithm offer a possibility to overcome the curse of dimensionality that affects the original approach; in particular, it is possible to forgo usual mixing distributions that are products of their one-dimensional marginals ("naive Bayes") in favor of general alternatives ("sophisticated Bayes"). The ensuing question then is how much this undertaking is justified by the resulting improvement of the prediction accuracy - on simulated examples and on real datasets.