The Effect of Formal Question Characteristics on Design Effects: A Cross-National Study Using the ESS (303567)

Keywords: TSE, design effects, question characteristics

The use of the Total Survey Error (TSE) framework reflects that the precision of survey estimates depends on sample sizes and nonresponse problems, but also on nonsampling errors. In practice, it is difficult to apply TSE models as individual effects can hardly be isolated given the available data. However, indirect estimations of some elements of the TSE are possible. One approach is the use of design-effects. Design effects are the increase in estimated standard errors in comparison to a simple random sample of the same size. Design-effects can be estimated relying on the interviewers' intra-class correlations. Thus, they allow us to estimate the homogeneity of responses in a set of respondents assigned to an interviewer. This cannot be achieved in a single survey, as we do not know if homogeneity is due to spatial (social) homogeneity of the PSU or the interviewers. Research indicates that variance induced by interviewers also depends on the question characteristics. Examples of different characteristics are nonfactual questions (e.g., attitudinal questions), difficult questions, i.e., those that require an increased cognitive effort (e.g., recall questions), sensitive questions (e.g., questions asking about embarrassing or socially undesirable behavior/attitudes), and open questions (e.g., absence of explicitly articulated response categories). These questions appear to be more likely to be affected by survey interviewer effects than their factual, easy, nonsensitive, or closed counterparts. We therefore developed a classification scheme for question characteristics which may make deviations from standardized interviewing more likely. We apply this classification scheme to all ESS questions asked in Rounds 1-7 and use these variables in a multi-level model predicting interviewer-homogeneity. We provide all details about the classification and the predictive power concerning interviewer related measurement error induced by question characteristics.