Rule of Thumb Regarding the Use Weights in Survey Sampling Analyses

• Activity Number: 413
• Type: Contributed
• Date/Time: Tuesday, August 2, 2011 : 2:00 PM to 3:50 PM
• Sponsor: Section on Survey Research Methods
• Author(s): Marnie Bertolet*+
• Companies: University of Pittsburgh
• Keywords: Survey Sampling ; Inverse Probability Weights ; Survey Weights ; Multi-Level Models ; Linear Mixed-Effects Models

Abstract

When using model-based inference in survey sampling, many practitioners are unsure as to whether sampling weights should be incorporated into the analysis. Weighted estimates can protect against informative sampling bias, however they have a larger variance that increases null results. A common rule of thumb is to compare the unweighted and weighted analyses. If the estimates match then the analysis is "correct" and you can use the more optimal unweighted analysis. If the estimates do not match, then you have non-ignorable design or model misspecification. In this case, more care is needed in model building. If a model cannot be found where the weighted and unweighted analyses match, then the weighted analysis should be used. While this advice is easy to give and implement, it is not always accurate. Using a set of simulations on survey weighted mixed-effect models; I provide counter-examples to this advice. I also provide guidance on when this advice may be correct, and when it may not be correct.