Online Program Home
My Program

Abstract Details

Activity Number: 236 - SLDS Student Paper Awards
Type: Topic Contributed
Date/Time: Monday, July 30, 2018 : 2:00 PM to 3:50 PM
Sponsor: Section on Statistical Learning and Data Science
Abstract #327213
Title: Valid Inference Corrected for Outlier Removal
Author(s): Shuxiao Chen* and Jacob Bien
Companies: Cornell Univ and University of Southern California
Keywords: confidence intervals; linear regression; outlier; p-value; selective inference

Ordinary least square (OLS) estimation of a linear regression model is well-known to be highly sensitive to outliers. It is common practice to first identify and remove outliers by looking at the data then to fit OLS and form confidence intervals and p-values on the remaining data as if this were the original data collected. We show in this paper that this "detect-and-forget" approach can lead to invalid inference, and we propose a framework that properly accounts for outlier detection and removal to provide valid confidence intervals and hypothesis tests. Our inferential procedures apply to any outlier removal procedure that can be characterized by a set of quadratic constraints on the response vector, and we show that several of the most commonly used outlier detection procedures are of this form. Simulated results corroborate the theoretical results and results on data illustrate how our inferential results can differ from the traditional strategy. A companion R package, outference, implements these new procedures with an interface that matches the functions commonly used for inference with lm in R.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2018 program