Fixing the Sample and the Sample Weights When Problems Arise in the Frame: Experiences with a Survey of Physicians

• Activity Number: 477
• Type: Contributed
• Date/Time: Wednesday, August 1, 2012 : 8:30 AM to 10:20 AM
• Sponsor: Section on Survey Research Methods
• Author(s): Eric Grau*+
• Companies: Mathematica Policy Research
• Keywords: sample frame ; sampling weights ; duplicates ; multiplicity ; replacement sample ; physician survey

Abstract

Probability samples are selected under the assumption that the sample frame closely resembles the target population. Occasionally, however, the sample frame has problems that are only discovered after the sample is selected. In this paper, we discuss a physician survey that was part of an evaluation of a Medicare pay-for-performance demonstration. Several problems with the sample frame were discovered after sample selection. In particular, the frame included duplicate physicians within and across practices. Moreover, for practices that participated in the demonstration, the frame identified all physicians in those practices as also participating in the demonstration, when this was not the case. This paper describes the process of calculating sampling weights that properly account for the original selected sample, the replacement sample, and the multiplicity adjustment for duplicates. We compare estimates obtained using the appropriately calculated weights (adjusted for nonresponse) with those obtained using a simple-minded weighting procedure.