Online Variational Inference for Probabilistic Topic Models

• Activity Number: 5
• Type: Invited
• Date/Time: Sunday, July 31, 2011 : 2:00 PM to 3:50 PM
• Sponsor: IMS
• Author(s): David Blei*+
• Companies: Princeton University
• Keywords: Topic modeling ; Bayesian nonparametric models ; Hierarchical models ; Variational inference

Abstract

Probabilistic topic modeling provides an important suite of tools for the unsupervised analysis of large collections of documents. Topic modeling algorithms can uncover the underlying themes of a collection and decompose its documents according to those themes. This analysis can then be used for tasks like corpus exploration, document search, and a variety of prediction problems.

Traditionally, topic models are fitted with "batch" algorithms, iteratively analyzing each document of the collection and then updating the parameters that describe the model. In this talk, I will describe a faster alternative based on online variational Bayes (VB). Online VB uses online stochastic optimization on the variational objective function with a natural gradient step. With online VB, we can handily analyze massive document collections, including those arriving in a stream. We demonstrate this approach by fitting a Bayesian nonparametric topic model to 3.3 million articles from Wikipedia. We demonstrate that online VB finds topic models as good or better than those found with batch VB, and in a fraction of the time.

(Joint work with Matt Hoffman, Chong Wang and Francis Bach)