Employing Data Mining to Develop a Prediction Model for the Statistical Characteristics of Test Questions

• Activity Number: 82
• Type: Contributed
• Date/Time: Sunday, August 3, 2014 : 4:00 PM to 5:50 PM
• Sponsor: Social Statistics Section
• Author(s): Peter J. Pashley*+ and Nicole E. Pashley
• Companies: Law School Admission Council and Queen's University
• Keywords: Data Mining ; Natural Language Processing ; Random Forest ; Hierarchical Modeling ; Statistical Characteristics of Test Questions

Abstract

Large-scale standardized testing programs, such as the SAT, GRE, and Law School Admission Test (LSAT), routinely pretest (or tryout) new test questions before they are given operationally. Pretesting allows for the collection of empirical data on the statistical characteristics of test questions, such as question difficulty, which are subsequently employed to assemble and equate operational test forms. Unfortunately, there are negative aspects to pretesting, including the necessity to expose test questions before they are given operationally, which could lead to test compromises. This study attempts to predict the statistical characteristics of test questions directly from question features, such as paragraph complexity and the cognitive tasks required, with the goal of eliminating, or at least reducing the need for pretesting. The very extensive LSAT question bank will be analyzed by way of data mining, natural language processing, and random forest and hierarchical modeling techniques. The resulting prediction model could impact the pretesting procedures employed by major testing programs.