Model-based and Scalable Relationship Discovery in Business Analytics

• Activity Number: 423
• Type: Contributed
• Date/Time: Tuesday, July 31, 2012 : 2:00 PM to 3:50 PM
• Sponsor: Social Statistics Section
• Author(s): Jing Shyr and Jane Chu*+ and Damir Spisic and Sier Han and Xue Ying Zhang
• Companies: IBM and IBM and IBM SPSS Predictive Analytics and IBM and
• Keywords: ANOVA test ; interaction effect ; tabular report

Abstract

This paper proposes a model-based and scalable system to produce a series of tabular reports that illustrate the important measure-dimension (or continuous target-categorical predictor) relationships and exhibit strong dimension interactions. The analysis for each report is based on a linear regression model and the interaction detection is an ANOVA test. However, only basic statistics are used to evaluate model accuracy and conduct ANOVA tests without actually fitting models. For datasets with the large number of categorical predictors, it becomes prohibitive to generate and analyze all possible tables. So a structured and scalable search process is applied: all the tables with single dimension are considered first; the tables with two and higher dimensions are considered selectively based on the analysis of the corresponding tables of lower dimension. This ensures that the computational effort needed for generating and analyzing the tables is limited. Furthermore, the top selected tables are further analyzed by detecting any cells, which correspond to the category combinations of dimensions in the table, with high contributions to the significance of the interaction effect.