Going Beyond a Pre-Post Design: Propensity Score Matching in a Cost Savings Framework for Nurse Care Management Program Evaluation
John McGready, Johns Hopkins Bloomberg School of Public Health 
*Shannon Marie Elizabeth Murphy, Johns Hopkins HealthCare LLC 
Martha Sylvia, Johns Hopkins University School of Nursing 

Keywords: propensity score matching, cost effectiveness, disease management, longitudinal data analysis, observational study design

Background: There is little evidence to suggest that nurse care management (CM) programs significantly reduce healthcare costs. Results from prior studies are questionable due to the use of pre-post designs, lack of comparison groups, small sample sizes, inadequate observation lengths, and oversimplified statistical methods. Studies lacking in methodological rigor tend to report higher ROIs than studies with more rigorous designs. Previously we developed a quasi-experimental longitudinal model for estimating CM savings that accounts for temporal changes (i.e., history, maturation) and regression to the mean. The goal of the current study is to address the lingering effects of selection bias common to observational designs. Specifically we explore the impact of propensity score matching for refining the comparison group to improve counterfactual trend estimation. Study Design: We utilized monthly observations on over 24,000 study participants enrolled in a commercial health plan from 2002 through 2009. Our CM program began in 2005 and consisted of high- and low-intensity interventions. Two comparison groups were identified from the pool of members eligible for CM who never enrolled, comprised of: 1) all eligible members, and 2) matched comparison members using propensity scores developed at plan enrollment. We used linear regression, accounting for temporal dependency in participant costs utilizing GEE with a Toeplitz-10 correlation structure, to compare cost trajectories for the intervention and comparison groups. Savings were realized when CM participant costs were lower than counterfactual expectations.

Results: In comparison to the pool of all possible members, the sample of matched comparison members exhibited greater health risks to coincide with the elevated morbidity burden of CM participants. The counterfactual cost trend as estimated by the matched comparison group was higher than found using all possible comparison members. Thus, the savings rate estimated using the matched sample was also higher.

Conclusion: When assessing CM program cost savings, propensity score matching can help to limit the threat of selection bias when identifying comparison group members. Using this improved method for evaluating CM cost savings provides national, state, and private payers with a more confident basis on which to make funding decisions.