Design Effect Anomalies in the American Community Survey

• Activity Number: 25
• Type: Topic Contributed
• Date/Time: Sunday, July 29, 2012 : 2:00 PM to 3:50 PM
• Sponsor: Section on Survey Research Methods
• Author(s): Michel Boudreaux*+ and Michael Davern and Peter Graven
• Companies: University of Minnesota and NORC and University of Minnesota
• Keywords: American Community Survey ; complex sample design ; design effects

Abstract

The American Community Survey is a widely used survey that is based on a complex sample. This paper analyzes the design effects (DE) of housing and economic variables. DE's measure the efficiency of the sample design relative to a hypothetical SRS of the same size. We find that the national DE's for personal earnings, health insurance, poverty and the percent of people living in rental housing are 2.0, 7.0, 10.0, and 21.7, respectively. These large DE's contrast with average DE's in sub-domains. Average state-level results range between 4.4 and 1.1. A potential cause for this result are outlying MSE's (the key component of the complex sample variance estimator). After excluding the top 5% of the MSE's the national DE's attenuate to expected levels. The national DE for health insurance is 2.6 versus a state average of 2.2. We use regression to identify cases that are likely to have outlying MSE's. Characteristics of interest include group-quarters, mode, demographics, and geography. We discuss implications for statistical inference in the 1, 3 and 5 year ACS files and speculate on why the published design factors (a variant of the DE measure) do not reflect the patterns we observe.