Prediction of True Scores from Observed Scores and Ancillary Data

• Activity Number: 443
• Type: Contributed
• Date/Time: Tuesday, August 11, 2015 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Education
• Author(s): Lili Yao* and Shelby Haberman and Sandip Sinharay
• Companies: Educational Testing Service and Educational Testing Service and CTB
• Keywords: Composite true score ; ancillary data ; repeaters ; adjustment by minimum discriminant information ; reliability

Abstract

In educational testing, an increasingly common issue has been efficient evaluation of a skill by use of multiple sources of information. For example, a writing assessment may include human ratings of essays, electronic essay ratings, and other section scores. Because standard linear regression is not applicable due to unobserved true scores, this paper suggests applications of classical test theory to obtain best linear predictors of composite true test scores based on observed test scores and ancillary data. Such analysis often requires information on repeaters who take the assessment more than once. Because such repeaters are not a random sample of all examinees, adjustment by minimum discriminant information (Haberman, 1984) is applied to reduce the effect of selection bias. Applications are made to TOEFL iBT Writing and another large scale Writing assessment to illustrate the proposed approach. Results obtained indicate that substantial improvements are possible both in terms of reliability of scoring and in terms of assessment reliability.