Distinctness Evaluation of Unknown Clustering Structure

• Activity Number: 318
• Type: Contributed
• Date/Time: Tuesday, August 6, 2013 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Ewa Nowakowska*+
• Companies: Institute of Computer Science, PAS
• Keywords: multivariate normal mixture ; principal component analysis ; clustering

Abstract

There are numerous measures designed to evaluate quality of a given data grouping, however they all require the partition to be already determined. The objective of this paper is to present a method of preliminary analysis that operates on raw data only and evaluates the distinctness of the intrinsic yet undetermined clustering structure in data (clusterability). In the framework of heterogeneous mixture of multivariate normal distributions, we first introduce a data transformation that preserves initial distinctness of the unknown clustering structure up to a negligible error. It is designed to bring the subspace of largest overall variability close to the subspace of best between cluster separation, hence allowing for efficient dimension reduction without the knowledge of clusters. Then, in the subspace of reduced dimension we propose the clusterability coefficient and show the results of its performance assessment. The coefficient measures to what extent the data may support meaningful and efficient clustering. Among others, the information may further be used in feature selection tasks.