An Empirical Bayes Method for Fitting Semiparametric Random Effect Models to Large Datasets

• Activity Number: 56
• Type: Topic Contributed
• Date/Time: Sunday, August 7, 2005 : 4:00 PM to 5:50 PM
• Sponsor: Section on Bayesian Statistical Science
• Author(s): Michael Pennell*+ and
• Companies: University of North Carolina, Chapel Hill/NIEHS and National Institute of Environmental Health Sciences
• Keywords: Cluster analysis ; Dirichlet process ; Empirical Bayes ; Hierarchical model ; Latent variables ; Longitudinal data

Abstract

For large datasets, it can be difficult or impossible to fit models with random effects using standard methods due to convergence and memory problems. In addition, it is appealing to take advantage of the abundant information in the data to avoid parametric assumptions, such as normality of the random effects. A Bayesian approach to this problem is to choose a Dirichlet process prior for the unknown random effects distribution, which clusters the subjects into K < N groups. Unfortunately, the number of groups increases with sample size, making computation impractical for very large N. To address this problem, we propose a two-stage clustering procedure. In the first stage, we use a sorting algorithm to assign subjects to one of G groups based on outcome and predictor values. Then, in stage 2, we assign a Dirichlet process prior to the group-specific parameters, further clustering the groups from the first stage. Because the computational burden increases primarily with G, this approach can be implemented efficiently even in very large samples. The methods are illustrated using simulated data and data from an epidemiologic study of 30,000 children followed longitudinally.