Conference Program Home
  My Program

All Times EDT

Abstract Details

Activity Number: 442 - Contributed Poster Presentations: Section on Statistics and Data Science Education
Type: Contributed
Date/Time: Wednesday, August 10, 2022 : 10:30 AM to 12:20 PM
Sponsor: Section on Statistics and Data Science Education
Abstract #323588
Title: Foundations for NLP-Assisted Formative Assessment Feedback for Short-Answer Tasks in Large-Enrollment Classes
Author(s): Susan Lloyd* and Matthew Beckman and Dennis Pearl and Rebecca Passonneau and Zhaohui Li and Zekun Wang
Companies: The Pennsylvania State University and The Pennsylvania State University and Pennsylvania State University and The Pennsylvania State University and The Pennsylvania State University and The Pennsylvania State University
Keywords: Natural Language Processing; Formative assessment; Scoring reliability; Inter- and Intra-rater agreement; Clustering; Artificial intelligence
Abstract:

Effective formative assessment (FA) is indispensable for instructors to monitor students’ learning. Research has linked “write-to-learn” tasks to improved learning outcomes in mathematics, yet constructed-response methods become unwieldy for instructors with large enrollment classes.

During a previous study, a sample of 1,935 students completed six short-answer tasks. Responses were divided among four trained evaluators to measure inter- and intra-rater agreement, including 178 responses scored from the study 7 years prior. A natural language processing (NLP) algorithm scored a subset of student responses for correctness. The team has piloted cluster analysis of student responses.

Quadratic weighted kappas (QWK) between 0.74 and 0.82, and Fleiss’ kappa of 0.68, indicated substantial inter-rater agreement among four raters, including the algorithm. Also, intra-rater agreement following 7 years elapsed was QWK=0.88.

By exploiting the efficiency of technology in the form of NLP-assisted FA feedback for short-answer tasks, students in large enrollment classes can be privy to the same type of personalized feedback that enhances the learning experience in small enrollment classes.


Authors who are presenting talks have a * after their name.

Back to the full JSM 2022 program