|
Abstract:
|
National statistical agencies are relying more heavily on administrative data sources, which are becoming increasingly larger, requiring efficient edit and imputation procedures. Outlier detection methods currently available at Statistics Canada are highly effective in settings where the variable of interest follows a unimodal distribution, either on its own, or within groups formed by a set of class variables. Often with large administrative data sources, finding a set of class variables which can be used to satisfy this assumption is a challenge, and the effectiveness of the outlier detection is subsequently reduced. This is the case for our motivating application involving international merchandise trade data. This paper explores unsupervised clustering techniques capable of handling a mixture of quantitative and qualitative variables, with the goal of applying these techniques in order to increase outlier detection efficacy. We propose a method for using cluster analysis to isolate modal distributions as a pre-treatment to outlier detection. In addition, we examine a clustering method for outlier detection directly. These methods are contrasted with a standard approach commonly used for business surveys at Statistics Canada.
|
