Scoring Approaches for Automated Scoring of Spoken Constructed Responses

• Activity Number: 532 - Can Statistics Inform Decisions in Social, Economic, and Political Event?
• Type: Contributed
• Date/Time: Wednesday, August 1, 2018 : 10:30 AM to 12:20 PM
• Sponsor: Social Statistics Section
• Author(s): Lili Yao* and Mo Zhang and Shelby Haberman and Neil Dorans
• Companies: ETS and ETS and Edusoft and ETS
• Keywords: Spoken Constructed Responses; Speech Rater; Composite True Score; BLP

Abstract

Spoken items arise in several tests ETS administered, for example, the Speaking Test of TOEFL iBT and TOEFL Practice Online (TPO). The scoring approaches on such type of items have attracted much attention. In addition to human ratings, the speech rater engine uses natural language processing techniques to extract information from spoken responses to produce computer-generated feature scores. This study seeks to extend the application of best linear prediction (BLP) to combine the information of human scores and feature scores to predict the composite true scores across all spoken items. To implement the BLP approaches, the estimation of the covariance matrix of the measurement errors relies on agreement sample (responses graded by more than one human rater randomly chosen) and the Cronbach alpha method. To illustrate possible approaches, application is made to the TOEFL iBT Speaking test. Results indicate that BLP is a feasible approach for scoring spoken responses in terms of both scoring and assessment accuracy.