Cluster Analysis via Random Partition Distributions

• Activity Number: 534
• Type: Contributed
• Date/Time: Wednesday, August 12, 2015 : 10:30 AM to 12:20 PM
• Sponsor: Section on Bayesian Statistical Science
• Author(s): David B. Dahl* and Mahlet Tadesse
• Companies: Brigham Young University and Georgetown University
• Keywords: Adjusted Rand index ; Bayesian nonparametrics ; Hierarchical clustering ; Nonexchangeable priors ; Random partitions

Abstract

We propose methodology for cluster analysis based on one of three random partition distributions indexed by pairwise distance information, the same kind of data used in hierarchical clustering. Two of the random partition distributions have been proposed as prior partition distributions in flexible Bayesian modeling and one is novel to this paper. Here, however, we are simply using these distribution for cluster analysis in direct competition to traditional hierarchical clustering. Being based on summaries of formal probability models, our cluster analysis methodology can assess uncertainty in the clustering estimates in a more intuitive manner than hierarchical clustering's dendrogram. We compare the clustering accuracy of our methods to various version of hierarchical clustering. We find empirically that our proposals are competitive with hierarchical clustering across a variety of scenarios and are superior in many cases.