Online Program

A Randomized Experiment Comparing Patient Survey Scores in Telephone and Mail Modes When Vendors Are Paid by the Hospitals Being Evaluated or a Third-Party Survey Vendor

Julie Brown, RAND 
Christopher Cohea, HSAG 
Mike Cui, RAND 
*Marc N. Elliott, RAND 
Laura Giordano, HSAG 
Elizabeth Goldstein, CMS 
Katrin Hambarsoomian, RAND 
William Lehrman, CMS 

Keywords: mode effect, survey methodology, patient experience

The Consumer Assessment of Healthcare Providers and Systems Hospital Survey, known as HCAHPS, is unusual among large-scale national surveys in that data are collected from dozens of certified survey vendors paid by hospitals in any of four approved survey modes and the resulting data is used for both public reporting and performance-based payment, after applying patient-mix adjustment and mode adjustment derived from a randomized survey to make results comparable across hospitals. Some might have concerns that vendors paid by hospitals whose scores have financial consequences might unconsciously encourage positive scores from patients, particularly in interactive modes such as telephone modes.

In a new randomized experiment, a neutral third-party survey vendor surveyed equivalent randomly selected patients from some of the same hospitals surveyed by their standard vendors at the same time and in the same mode (here restricting to telephone and mail modes). We used linear regression models to compare each of the 10 HCAHPS measures of patient experience and logistic regression to compare response rates. The comparisons involved 47 hospitals served by 9 survey vendors and 10,682 completed patient surveys.

Comparisons of response rates in matched hospitals found only small differences. We found no statistically significant evidence of differences in hospital patient-experience scores, reinforcing prior evidence of the validity of this data collection mechanism in the presence of standardized vendor training and oversight. Such an approach may be a model for other high-stakes applications of survey-based data collection, which increases in scope under the Patient Protection and Affordable Care Act.