548 – Using Artificial Intelligence and Advanced Statistical Methods to Improve Official Statistics
Recommender Algorithms for Form Anomaly Detection
William J.J Roberts
Deloitte Consulting
Unsupervised anomaly detection is performed on forms assumed to be sparse and normally distributed. Maximum likelihood (ML) estimation is applied to estimate the parameter from a large collection of sparse forms. An expectation maximization algorithm from the literature that has been applied to sparse-matrix recommendation is used. Given the estimated parameter, constrained ML is applied to estimate anomalies. The constraints here are used to ensure that only anomalies in specific and predefined subspaces are detected. This formulation borrows from the literature of one-sided multivariate testing. The overall approach is tested and the results compared to a database of forms with known anomalies. The approach improves on the authors’ previously developed unsupervised method for anomaly detection.