Big Data, Big Models, Big Problems: Consensus Monte Carlo Methods for Distributed Bayesian Inference

• Activity Number: 154
• Type: Topic Contributed
• Date/Time: Monday, August 4, 2014 : 10:30 AM to 12:20 PM
• Sponsor: International Society for Bayesian Analysis (ISBA)
• Author(s): Alexander Blocker*+ and Steven L. Scott and Fernando Bonassi
• Companies: Google and Google and Google
• Keywords: high-performance computing ; ensemble methods ; regression ; MCMC

Abstract

We need rich statistical models to get the most out of huge datasets, but standard iterative algorithms perform poorly on distributed systems. In this world, faults are common, communication among machines is expensive, and synchronization is a four-letter word. We need algorithms that provide distributed approximate Bayesian inference with minimal communication, especially for high-dimensional problems. Consensus Monte Carlo (CMC) methods address this by running a separate posterior sampler on each machine and aggregating the individual draws. CMC fits naturally within MapReduce models of computation and scales well with the size of parameters involved. For many models of interest, simple linear combining rules can produce draws nearly indistinguishable from those produced by serial Monte Carlo samplers. We discuss the uses and limitations of CMC methods, focusing on GLMMs and related models.