Covariance Matrices for Mean Field Variational Bayes

• Activity Number: 157
• Type: Invited
• Date/Time: Monday, August 10, 2015 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Tamara Broderick* and Ryan Giordano
• Companies: MIT and UC Berkeley
• Keywords: variational Bayes ; linear response ; mean field

Abstract

Mean Field Variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is its (sometimes severe) underestimates of the uncertainty of model variables and lack of information about model variable covariance. We develop a fast, general methodology for exponential families that augments MFVB to deliver accurate uncertainty estimates for model variables-both for individual variables and coherently across variables. MFVB for exponential families defines a fixed-point equation in the means of the approximating posterior, and our approach yields a covariance estimate by perturbing this fixed point. Inspired by linear response theory, we call our method linear response variational Bayes (LRVB). We demonstrate the accuracy and scalability of our method on both simulated and real-world data sets.