Sequential Design for Computerized Adaptive Testing That Allows for Response Revision

• Activity Number: 79
• Type: Contributed
• Date/Time: Sunday, August 9, 2015 : 4:00 PM to 5:50 PM
• Sponsor: Social Statistics Section
• Author(s): Shiyu Wang* and Georgios Fellouris and Hua-Hua Chang
• Companies: University of Illinois at Urbana-Champaign and University of Illinois at Urbana-Champaign and University of Illinois
• Keywords: Computerized Adaptive Testing ; Experiment Design ; Large Sample Theory ; Item Response Theory ; Martingale Limit Theory ; Sequential Design

Abstract

In computerized adaptive testing (CAT), items (questions) are selected in real time based on the already observed responses, so that the ability of the examinee can be estimated as accurately as possible. This is formulated as a non-linear, sequential, experimental design problem with binary observations that correspond to the true or false responses. However, most items in practice are multiple-choice and dichotomous models do not make full use of the available data. Moreover, CAT has been heavily criticized for not allowing test-takers to review and revise their answers. In this work, we propose a CAT design that is based on the polytomous nominal response model and in which test-takers are allowed to revise their responses at any time during the test. We show that as the number of administered items goes to infinity, the proposed estimator is strongly consistent for any item selection and revision strategy and asymptotically normal when the items are selected to maximize the Fisher information at the current ability estimate and the number of revision is smaller than the number of items. We also present the results of a simulation study that supports our asymptotic results.