Medians in the Space of Probability Distributions and Applications to Robust and Scalable Bayesian Inference

• Activity Number: 154
• Type: Topic Contributed
• Date/Time: Monday, August 4, 2014 : 10:30 AM to 12:20 PM
• Sponsor: International Society for Bayesian Analysis (ISBA)
• Author(s): Sanvesh Srivastava*+
• Companies: SAMSI/Duke University
• Keywords: Robust ; Scalable ; Distributed ; Median Posterior ; Bayesian Inference

Abstract

Bayesian learning methods for massive data benefit from working with data subsets or with data in small mini-batches. Significant progress has been made in scalable Bayesian learning via stochastic approximation that uses data in small mini-batches. On the contrary, Bayesian learning methods in distributed computing environments are either problem or distribution specific due to the following reason: there is no general approach for combining posterior distributions obtained from different data subsets (hereafter subset posteriors). To this end, we propose a principled approach for combining subset posteriors through their median posterior (M-posterior). It is a natural extension of the median to the space of probability distributions and is robust to outliers and model misspecification. The M-posterior minimizes the sum of its distance from all subset posteriors, where the distance between posteriors is defined using the inner-product of a Reproducing Kernel Hilbert Space. We describe the theoretical properties of this M-posterior and validate these claims through experiments on simulated and real data.