Estimating the Number of Clusters Using Cross Validation

• Activity Number: 74
• Type: Contributed
• Date/Time: Sunday, August 9, 2015 : 4:00 PM to 5:50 PM
• Sponsor: Section on Nonparametric Statistics
• Author(s): Wei Fu* and Patrick Perry
• Companies: and New York University
• Keywords: clustering ; K-means ; cross validation ; nonparametric

Abstract

Many clustering methods, including K-means, require the user to specify the number of clusters as an input parameter. A variety of methods have been devised to choose the number of clusters automatically, but they often rely on strong modeling assumptions. We propose a data-driven approach to estimate the number of clusters based on a novel form of cross-validation. This differs from ordinary cross-validation, because clustering is fundamentally an unsupervised learning problem. Simulation and real data analysis results show that our proposed method outperforms existing methods, especially in high-dimensional settings with heavy-tailed data