Validation of Software for Bayesian Models Using Posterior Quantiles

• Activity Number: 16
• Type: Topic Contributed
• Date/Time: Sunday, August 7, 2005 : 2:00 PM to 3:50 PM
• Sponsor: Section on Bayesian Statistical Science
• Author(s): Samantha R. Cook*+ and Andrew Gelman and Donald B. Rubin
• Companies: Columbia University and Columbia University and Harvard University
• Keywords: Markov chain Monte Carlo ; Posterior distribution ; Hierarchical models ; Gibbs sampler

Abstract

We present a simulation-based method designed to establish that software developed to fit a specific Bayesian model works properly, capitalizing on properties of Bayesian posterior distributions. The validation method involves repeatedly generating parameters and data from the model to be fit and then fitting the same model to these simulated data (i.e., generating a sample from the posterior distribution). For all scalar parameters, the quantile of the "true" parameter value with respect to its posterior distribution should follow a uniform distribution if the software is written correctly. Testing that the software works amounts to testing that these quantiles are uniformly distributed. We illustrate that the validation method finds errors in software when they exist and, moreover, the validation output can be informative about the nature and location of such errors.