Penalized Linear Regression and Influential Observations

• Activity Number: 2
• Type: Invited
• Date/Time: Sunday, July 31, 2011 : 2:00 PM to 3:50 PM
• Sponsor: IMS
• Author(s): Karen Kafadar*+ and Guilherme Rocha
• Companies: Indiana University and Indiana University
• Keywords: Lasso ; robustness ; model selection ; outliers ; leverage ; microarray experiments

Abstract

In current problems (e.g., microarrays, financial data) where the number of variables can greatly exceed the number of observations ("big p, small n"), penalized regression is successful in identifying informative variables by setting to zero a large subset of the regression coefficients. This approach to model selection aims for good fits to the data, but the resulting nonzero coefficients often are interpreted. The usual squared error loss combined with L1 penalty on the coefficients, or LASSO, results in models that can be highly sensitive to potential outliers, in either the response variable or the design space. This study examines the effect of influential points (outliers and leverage points) on L1-penalized regression estimators, when the loss function is L2, biweight, or MM. We assess sensitivity of results (bias, MSE, percent of non-zero coefficients) to 0%, 5%, 15% contamination, when the error distribution is Gaussian, Cauchy, and double-exponential. We show that a robust loss function greatly reduces the effect of outliers and influential points in the simple case of linear regression when the proportion of non-zero coefficients is less than 20%.