321 – Methods and Applications in Biomedical Data and Clinical Trials, Part 1

Combining Paradata and Survey Responses to Identify Sources of Measurement Error in Medical Event Reporting

at Agency for Healthcare Research and Quality

The ability to observe within-subject change over time is the primary objective of most panel surveys. When characteristics of the data collection process systematically affect reporting differently at different times, it becomes difficult to differentiate true change from measurement error. The Medical Expenditure Panel Survey (MEPS) employs an overlapping panel design in which new cohorts enter the survey every January and are interviewed five times covering a cumulative two-year reference period. Underreporting is a perennial concern for household surveys and this concern may be exacerbated in panel surveys because of issues such as panel conditioning (Kalton et al 1989). In particular, a review of the literature pertaining to the accuracy of household-reported healthcare utilization data suggests that medical events tend to be underreported (Bhandari and Wagner 2006; Zuvekas and Olin 2009). Separate MEPS panels consistently exhibit a pattern of disproportionately high medical event reporting in the first round relative to all subsequent rounds and an additional decline at the final round of data collection. The fact that this pattern persists across separate panels suggests that these differences may reflect measurement error. Steps to repair this error will depend on its cause. One hypothesis is that respondents reduce their reporting in Round 2 in order to reduce burden. Alternatively, the error may be cognitive in origin, with longer reference periods in Round 2 resulting in a greater level of forgetting on the part of the respondent. In this paper we compare the plausibility of these hypotheses for explaining changes in response patterns using both paradata and survey responses. We find no support for the hypothesis that burden leads to lower reporting, however, we do find a negative association between the length of the reference period and the level of reporting.