Challenges from Modeling Open Online Assessment Data

• Activity Number: 538 - SPEED: Predictive Analytics with Social/Behavioral Science Applications: Spatial Modeling, Education Assessment, Population Behavior, and the Use of Multiple Data Sources
• Type: Contributed
• Date/Time: Wednesday, August 1, 2018 : 10:30 AM to 11:15 AM
• Sponsor: Social Statistics Section
• Author(s): Yan Liu* and Henrike Besche and Audrey Béliveau and Xingyu Zhang and Edward Kroc and Melanie Stefan and Johanna Gutlerner and Chanmin Kim
• Companies: The University of British Columbia and Harvard Medical School and University of Waterloo and The Hong Kong University of Science and Technology and The University of British Columbia and Edinburgh Medical School and Harvard Medical School and Boston University School of Public Health
• Keywords: open online assessment; response time; generalized linear mixed model; latent class analysis; data visualization; psychometrics

Abstract

Open-book or "open" online assessments have gained popularity through Massive Open Online Courses (MOOCs). Recently, low-stakes open online quizzes have gained popularity because they are easy to implement, require little or no grading time, provide feedback promptly, and encourage higher cognitive thinking skills. The emerging open online assessment data have posted many modeling challenges to researchers. The existing psychometric models are not appropriate for this type of data because of several issues: (1) the data usually pertains to small sample size; (2) a variety of topics are included in one short quiz, which make it impossible to use the conventional multidimensional models; and (3) open online assessments bring in new information that has not been considered in the past, e.g., response time and response orders. This study is to explore the utilization of data visualization, generalized linear mixed model, and latent class analysis for analyzing open online assessment data. Our study will try to fill in the gap between the advance of online classroom assessments and the lag behind statistical models for analyzing the new type of assessment data.