A Nonparametric Divergence Approach to Detect Item Parameter Drift in Multidimensional Computerized Adaptive Testing

• Activity Number: 692
• Type: Contributed
• Date/Time: Thursday, August 13, 2015 : 10:30 AM to 12:20 PM
• Sponsor: Section on Nonparametric Statistics
• Author(s): Hyeon-Ah Kang* and Hua-Hua Chang
• Companies: University of Illinois at Urbana-Champaign and University of Illinois
• Keywords: Item parameter drift ; Multidimensional computerized adaptive testing ; informational divergence ; Mantel-Haenszel statistic

Abstract

Recognizing heterogeneous item functioning over multiple test administrations has been drawing great interests among testing organizations and psychometric research. This study presents four informational divergence measures to detect changes in item response functions (IRFs) in multidimensional computerized adaptive testing (MCAT). Significance tests based on these measures are proposed to assess the homogeneity between two IRFs. The test statistics are nonparametrically estimated using matching variables obtained from K-Nearest Neighbors algorithm based on a set of multidimensional quadrature points. Critical values are empirically obtained via bootstrap resampling. Results from simulation studies suggest that the proposed statistics are effective in detecting both linear and non-linear transformation of IRFs. They demonstrate moderate to high true positive rates while keeping false alarms at a nominal level. The present study shows that a divergence approach has great potential as an alternative to traditional parametric methods in that it can diagnose items without high calibration error, and monitor items on the fly during the course of MCAT.