Statistical Significance for Hierarchical Clustering

• Activity Number: 614
• Type: Contributed
• Date/Time: Wednesday, August 12, 2015 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Patrick Kimes* and Yufeng Liu and James Stephen Marron and D. Neil Hayes
• Companies: The University of North Carolina at Chapel Hill and The University of North Carolina and The University of North Carolina and The University of North Carolina at Chapel Hill
• Keywords: Hierarchical Clustering ; Unsupervised Analysis ; Genomics

Abstract

Hierarchical clustering has proved to be an invaluable tool for the exploratory and unsupervised analysis of high dimensional datasets in genomics and other fields. A critical and challenging question in cluster analysis is whether the identified clusters represent important underlying structure or are artifacts of natural sampling variation. Few approaches have been proposed for addressing this problem in the context of hierarchical clustering, for which the problem is further complicated by the natural tree structure of the partition, and the multiplicity of tests required to parse the layers of nested clusters. We propose a Monte Carlo based approach for testing statistical significance in hierarchical clustering which addresses these issues. The approach is implemented as a sequential testing procedure guaranteeing control of the family-wise error rate. Theoretical justification is provided for our approach, and its power to detect true clustering structure is illustrated through several simulation studies and applications to cancer gene expression datasets.