On the Similarity of Principal Components, Random Projections and Random Column Subsampling for Dimension Reduction in High-Dimensional Linear Regression

• Activity Number: 304 - Statistical Learning: Dimension Reduction
• Type: Contributed
• Date/Time: Tuesday, August 1, 2017 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistical Learning and Data Science
• Author(s): Martin Slawski*
• Companies: George Mason Univ
• Keywords: High-Dimensional and Large-Scale Regression ; Dimension Reduction ; Principal Components Regression ; Random Projections

Abstract

Principal Components Regression (PCR) is a traditional tool for dimension reduction in linear regression that has been both criticized and defended. One concern about PCR is that computation of the leading principal components tends to be computationally demanding for large data sets. While Random Projections (RPs) do not possess the optimality properties of the projection onto the leading principal subspace, they are computationally appealing and hence have become increasingly popular in recent years. In this paper, we present an analysis showing that the dimension reduction offered by RPs achieves a prediction error in subsequent regression close to that of PCR, at the expense of requiring a slightly large number of RPs than PCs. By drawing connections to the result for RPs, we show that this phenomenon continues hold even for the simplistic approach in which a reduced design matrix is formed by selecting columns uniformly at random