|
Effective formative assessment (FA) is indispensable for instructors to monitor students’ learning. Research has linked “write-to-learn” tasks to improved learning outcomes in mathematics, yet constructed-response methods become unwieldy for instructors with large enrollment classes.
During a previous study, a sample of 1,935 students completed six short-answer tasks. Responses were divided among four trained evaluators to measure inter- and intra-rater agreement, including 178 responses scored from the study 7 years prior. A natural language processing (NLP) algorithm scored a subset of student responses for correctness. The team has piloted cluster analysis of student responses.
Quadratic weighted kappas (QWK) between 0.74 and 0.82, and Fleiss’ kappa of 0.68, indicated substantial inter-rater agreement among four raters, including the algorithm. Also, intra-rater agreement following 7 years elapsed was QWK=0.88.
By exploiting the efficiency of technology in the form of NLP-assisted FA feedback for short-answer tasks, students in large enrollment classes can be privy to the same type of personalized feedback that enhances the learning experience in small enrollment classes.
|
