Improved Statistical Significance of Clustering

• Activity Number: 15
• Type: Topic Contributed
• Date/Time: Sunday, July 29, 2012 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Hanwen Huang*+ and Yufeng Liu and Ming Yuan and J. Steve Marron
• Companies: The University of Texas Health Science Center and The University of North Carolina at Chapel Hill and Georgia Institute of Technology and The University of North Carolina at Chapel Hill
• Keywords: Clustering ; Covariance Estimation ; High Dimension ; Invariance Principles ; Shrinkage ; Unsupervised Learning

Abstract

Statistical Significance of Clustering (SigClust) is a recently developed cluster evaluation tool which was specifically designed for testing the clustering results for high dimensional low sample size data. SigClust assesses the significance of departures from a Gaussian null distribution, using invariance properties to reduce the needed parameter estimation. The cornerstone of the SigClust analysis is the accurate estimation of the eigenvalues of the covariance matrix of the null multivariate Gaussian distribution. Empirical studies have shown that the current method used in SigClust for covariance matrix estimation needs improvement. In this paper, we introduced two such improvements. They are a likelihood based soft thresholding and a shrinkage approach. The advantages of the new methods over the previous ones are demonstrated through application to both simulated and real data.