Behavior Coding on a Budget: An Efficient and Informative Multi-Method Interviewer-Respondent Interaction Observation System for Question Pretesting, Design, and Evaluation (303147)

Keywords: interviewer-respondent interaction, behavior coding, interaction coding, Cannell, transgender, gender identity, sensitive questions

Observing interviewer/respondent behavior and interaction is a popular question design/pretesting approach because question designers can witness interaction sequences first-hand and obtain insights into potential problems and solutions. Called “behavior coding” or “interaction coding,” these methods are time intensive. Further, they assume that a) actual and reliable problems (versus artifacts and random phenomena) will emerge through their use; b) the problems we see will be helpful in revising questions (rather than adding noise and false signals that complicate the decision); and c) the same problems could not reliably and accurately be observed with a simpler method. We discuss a streamlined mixed-methods alternative to these approaches that combines qualitative group listening with solo listening/coding and minimal quantitative analysis. We developed this method for a gender identity (i.e., transgender) question wording experiment in the California Health Interview Survey. Four question versions were randomly assigned to n=2828 respondents in a general population survey. Two versions used a two-question process (i.e., asking respondents' birth sex, then current sex/gender). The other two used a one-question process (i.e., defining "transgender" and asking respondents if they identify as transgender). Interactions were monitored as a group, and two staff held detailed solo listening sessions, coding question/answer duration (n=220, 165 English and 55 Spanish interviews). A two-question version was easiest and quickest to administer, and obtained transgender identification rates similar to benchmark estimates. Our lower-effort mixed-method approach was sufficient to assess difficulty and impact of each question in our time-sensitive context. We discuss this method within the history of interviewer-respondent interaction coding methods, focusing on continua of effort, cost, data produced, and insights gained from various methods.