New, Testable, Definitions of Reliability and Measurement Error

• Activity Number: 529
• Type: Contributed
• Date/Time: Wednesday, August 3, 2011 : 10:30 AM to 12:20 PM
• Sponsor: Social Statistics Section
• Author(s): Futoshi Yumoto and Rochelle Elaine Tractenberg*+
• Companies: American Institutes for Research and Georgetown University Medical Center
• Keywords: classical test theory ; reliability ; measurement error ; psychometrics ; cognitive testing

Abstract

"Reliability" under classical test theory (CTT) assumes that measurement error is: a) identically distributed for all respondents; and b) independent of true score. Neither is testable. A new definition of reliability and measurement error fits the Guttman model to the ability to respond to each item -demonstrated with a cognitive instrument - over time. Incorrect answers at a visit imply incorrect responses at all successive visits, representing "cognitive loss". Items with response patterns that are consistent with the Guttman model are "reliable"; "measurement error" is an item yielding an inconsistent signal about the individual's ability to respond. We modeled the first 4 annual evaluations in: a) cognitively normal (NC, N=149); b) diagnosed with possible or probable Alzheimer's disease (AD) (N=78); and c) cognitively normal with a later diagnosis of AD (converters, N=133). Of 16 test items, reliable measurement was observed for 10 items in NC, eight in converters, and two in AD. We formalized "reliability" over time and functionally defined measurement error. Measurement error was inconsistent over time and cognitive levels, violating the CTT assumptions for reliability.