614 – Advances in Clustering
Noisy Data Clusters Are Hollow
François Léonard
Hydro-Quebec
A new vision in multidimensional statistics is proposed impacting several areas of applications. In these applications, a set of noisy measurements characterizing the repeatable response of a process is known as a realization and can be seen as a single point in R^N. The projections of this point on the N axes correspond to the N measurements. The contemporary vision of a diffuse cloud of realizations distributed in R^N is replaced by a cloud having the shape of a shell surrounding a topological manifold. This manifold corresponds to the process response domain observed without the measurement noise. The measurement noise, which accumulated over several dimensions, distances each realization from the manifold. The probability density function (PDF) of the realization-to-manifold distance creates the shell. Considering the central limit theorem as the number of dimensions increases, the PDF tends toward the normal distribution. In the proposed vision, the likelihood of a realization is a function of the realization-to-shell distance rather than the realization-to-manifold distance. The demonstration begins with the work of C. Shannon and ends with monitoring equipment applications.