Information Accumulation, Model Selection, and Rater Behavior in Constructed Response Student Assessments

• Activity Number: 147
• Type: Topic Contributed
• Date/Time: Monday, August 9, 2004 : 2:00 PM to 3:50 PM
• Sponsor: Section on Bayesian Statistical Science
• Author(s): Louis T. Mariano*+
• Companies: RAND Corporation
• Keywords: Bayesian hierarchical models ; item response theory ; importance sampling ; Bayes factors

Abstract

Open-ended items are now commonly used in educational testing. Responses to these items are usually evaluated by human raters, often with multiple raters judging each response. When modeling such an assessment, the additional uncertainty due to the raters must be accurately captured. The IRT Facets model is contrasted with the Bayesian Hierarchical Rater Model. A rigorous treatment of the approach to dependence and uncertainty in each model shows that the Facets model is overly optimistic in the accumulation of Fisher information about examinee ability, while information under the HRM is properly bounded. The HRM is expanded to include covariates of rater behavior as a diagnostic tool for both between-rater and within-rater effects. As an example, data from a state assessment rating study is used to demonstrate the effect of modality--the design for distributing items among raters--on rater performance. The selection among multiple covariates requires a choice among models where parameters number in the thousands. Methods based in importance sampling for computing marginal densities are extended to apply to this high-dimensional setting.