Variable Selection Methods for Big Data: A Comparative Study

• Activity Number: 312
• Type: Contributed
• Date/Time: Tuesday, August 11, 2015 : 8:30 AM to 10:20 AM
• Sponsor: Section for Statistical Programmers and Analysts
• Author(s): Jun Liu* and Xuejing Mao
• Companies: and AT&T
• Keywords: big data ; logistic regression ; variable selection

Abstract

Variable selection is an important step in statistical analysis. When the number of potential predictors is small, this step is straightforward. But with more and more predicators available in today's environment, this step becomes more and more critical and complicated. Logistic regression has many applications in business area. One of the areas logistic regression is widely used is risk management, for example, to predict the likelihood that a customer will be delinquent. In this paper, we will compare the performance of three commonly used variable selection methods in logistic regression using a large data set. This dataset is typical "Big" data as the number of records , as well as the number of variables in this dataset are very large.