475 – SPEED: Predictive Analytics with Social/Behavioral Science Applications: Spatial Modeling, Education Assessment, Population Behavior, and the Use of Multiple Data Sources
Challenges from Modeling Open Online Assessment Data
Yan Liu
Statistics of Income, Internal Revenue Service
Henrike Besche
Harvard Medical School
Audrey Béliveau
University of Waterloo
Xingyu Zhang
The Hong Kong University of Science and Technology
Edward Kroc
Melanie Stefan
Johanna Gutlerner
Chanmin Kim
Open-book or "open" online assessments have gained popularity through Massive Open Online Courses (MOOCs). Recently, low-stakes open online quizzes have gained popularity because they are easy to implement, require little or no grading time, provide feedback promptly, and encourage higher cognitive thinking skills. The emerging open online assessment data have posted many modeling challenges to researchers. The existing psychometric models are not appropriate for this type of data because of several issues: (1) the data usually pertains to small sample size; (2) a variety of topics are included in one short quiz, which make it impossible to use the conventional multidimensional models; and (3) open online assessments bring in new information that has not been considered in the past, e.g., response time and response orders. This study is to explore the utilization of data visualization, generalized linear mixed model, and latent class analysis for analyzing open online assessment data. Our study will try to fill in the gap between the advance of online classroom assessments and the lag behind statistical models for analyzing the new type of assessment data.