Model-Based Clustering Analysis of Large Climate Simulation Data Sets

• Activity Number: 174
• Type: Contributed
• Date/Time: Monday, July 30, 2012 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Computing
• Author(s): Wei-Chen Chen*+ and George Ostrouchov and David Pugmire and Mr Prabhat and Michael Wehner
• Companies: Oak Ridge National Laboratory and Oak Ridge National Laboratory and Oak Ridge National Laboratory and Lawrence Berkeley National Laboratory and Lawrence Berkeley National Laboratory
• Keywords: model-based clustering ; unsupervised learning ; EM ; APECM ; CAM ; SPMD

Abstract

We develop a parallel expectation-maximization (EM) algorithm for model-based clustering, utilizing high-performance computing techniques. We utilize the single program multiple data (SPMD) programming model to reduce communication between processors. Our parallel EM algorithm scales for clustering ultra-large (hundreds of terabytes) datasets. We can apply the same technique for improving the scalability of EM-alike algorithms, such as AECM and APECM. Moreover, these parallel algorithms are easily generalized for optimizing other finite mixture models. We demonstrate the performance of our parallel algorithm on a high resolution climate dataset produced by the community atmosphere model (CAM5). An accompanying R package 'pmclust', for parallel model-based clustering is released on CRAN.