Identification of Multivariate Outliers in Large Data Sets

• Activity Number: 475
• Type: Contributed
• Date/Time: Thursday, August 7, 2003 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Computing
• Author(s): Mark Werner*+ and Karen Kafadar
• Companies: University of Colorado, Denver and University of Colorado, Denver
• Keywords: robust estimation ; outlier identification ; Mahalanobis distance ; biweight function ; influence function

Abstract

We present a new method of detecting outliers in large data sets. This procedure uses Tukey's biweight function to assign weights to observations in separate dimensions, and obtain an initial subset presumed to be outlier-free. The sample mean and covariance can then be efficiently calculated, and a robust Mahalanobis distance assigned to each observation. We estimate the density of these robust Mahalanobis distances, and are particularly interested in the peak which indicates the range of uncontaminated robust Mahalanobis distances. As this peak ends and the density decreases, we determine a rejection value where the estimated density is sufficiently close to zero. Outliers are classified as all points whose robust Mahalanobis distance exceeds the rejection value. This procedure is very successful at finding outliers and is also computationally fast, even on non-normal data and at relatively high dimension. With the help of the influence function, we examine several numerical robustness properties of the robust estimator defined by the first phase of this algorithm, which provide insight into the estimator's performance and how it can be improved.