Determining Dimensionality of a Cognitive Process: Testing Online Reading Comprehension

• Activity Number: 296
• Type: Topic Contributed
• Date/Time: Tuesday, August 6, 2013 : 8:30 AM to 10:20 AM
• Sponsor: ENAR
• Author(s): Weiwei Cui*+ and Nell Sedransk
• Companies: National Institute of Statistical Sciences and National Institute of Statistical Sciences
• Keywords: performance-based assessment ; confirmatory factor analysis ; psychometrics ; computer_based tests ; internet learning ; reading assessment

Abstract

Internet learning, especially assessment of online learning skills, is receiving increased attention throughout the education community. Therefore performance-based Online Reading Comprehension Assessments (ORCAs) were developed to assess 7th grade students online learning skills in compliance with the national Common Core Standards. These ORCAs were designed to measure four components of online comprehension: locate, evaluate, synthesize, and communicate information. Therefore, ORCAs were built with 16 items - 4 items specifically associated with each component. Each ORCA was administered in different formats and students were randomly assigned to three groups: open internet test; closed "pseudo internet" test; and conventional computer-based multiple choice test. The key question is whether the cognitive processes are truly distinct so that the internet-based ORCAs, at least, are multidimensional. Data were analyzed by confirmatory factor analytic models to compare multidimensional to uni-dimensional models. The performance-based tests were shown to measure multiple dimensions, whereas the traditional multiple choice test is well explained by a uni-dimensional model.