|
Abstract:
|
Non-probability surveys (NPS) are already widely in use in market research. However, their adoption for official statistics is much more problematic. AAPOR (2013) identifies a "Fit for Purpose" approach where the most difficult issue to address is making point estimates that are statistically valid, that is, can be used for statistical inference. This paper describes a methodology to empirically evaluate NPS surveys selected from a panel for statistical inference. The method compares estimates from an online panel with data from a gold standard probability survey. The key aspect of the methodology include transparency through an a priori decision rule motivated by the ASPIRE system developed by Bergdahl et al. (2014). We propose a distance metric and a predetermined cutoff value for deciding whether to accept or reject NPS estimates. Our decision rule is based on comparisons of 1) overall survey and subgroup estimates 2) the cv of the variability of the post-stratification weights and 3)the ratio of response rates. We illustrate our proposed empirical method by comparing data from a NPS quota sample for the Los Angeles area with a probability health survey of the same area.
|
