Online Program

The Invalidity of the Most Common Instrumental Variable Analyses in Comparative Effectiveness Research

*Laura Faden Garabedian, Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute 
Paula Chu, Harvard University 
Sengwee Darren Toh, Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute 
Alan Zaslavsky, Department of Health Care Policy, Harvard Medical School 
Stephen Soumerai, Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute 

Keywords: Comparative Effectiveness Research, Instrumental Variables

Importance: Comparative effectiveness research (CER) relies heavily on observational methods to estimate treatment effects to guide clinical and regulatory decisions. Instrumental variable (IV) analysis is an increasingly popular CER approach that relies on identification of a variable (the IV) that affects treatment assignment but does not otherwise affect outcomes. In theory, the approach mimics random treatment assignment seen in randomized trials. In practical application, however, the analysis may be biased if the IV and the outcome are related through an unadjusted third variable (“IV-outcome confounder”), leading to questionable findings.

Objective: To evaluate trends in the use of IVs for CER and systematically identify the existence and impact of confounders of common IVs on study validity.

Design, Setting, and Participants: We conducted a systematic search in PubMed and other health/economic databases to identify published CER studies that use IV methods conducted in the US or other industrialized countries through 2011. We searched for evidence of potential confounders of the most common IV-outcome pairs.

Main Outcome Measures: Count of IV CER studies (by year, country, and outcome measure), major confounders of most common IV-outcome pairs, and proportion of IV CER studies that failed to control for these confounders.

Results: We found 187 IV CER studies meeting the selection criteria. Of these, 60.9% used one or more of the four most common IV categories – regional variation (26.2% of studies), distance to facility (20.3%), facility variation (11.8%), and physician variation (7.5%). Mortality was the most common outcome. We observed overwhelming evidence of IV-outcome confounding. Major confounders of the four most common IVs and mortality include patient’s race, socioeconomic status, clinical risk factors, health status, and urban/rural residency; and facility and procedure volume. Every IV CER study failed to control for one or more of these potentially major confounders.

Conclusions and Relevance: Many effect estimates from IV analyses in CER may be biased by the failure to adjust for major IV-outcome confounders, which can lead to overestimation, underestimation or complete reversal of the true treatment effect. While no observational method can completely eliminate confounding, we caution against over-reliance on IV studies for CER.