Effects of Item Order Randomization in an Online National Drug Use Survey

• Activity Number: 210 - Contributed Poster Presentations: Survey Research Methods Section
• Type: Contributed
• Date/Time: Tuesday, August 4, 2020 : 10:00 AM to 2:00 PM
• Sponsor: Survey Research Methods Section
• Author(s): Elise Catherine Amioka* and Joshua Curtis Black
• Companies: Rocky Mountain Poison & Drug Safety (RMPDS) and Rocky Mountain Poison & Drug Safety (RMPDS)
• Keywords: randomization; drug abuse; online survey design; order effects

Abstract

With the wide popularity of online surveys, understanding optimal questionnaire design is crucial. While it is understood that the order of questions, order of response options, and overall length of a questionnaire affect respondent behavior, these effects are rarely quantified. The Survey of Non-Medical Use of Prescription Drugs Program (NMURx) is an online national survey on drug use that employs several levels of block randomization to minimize serial order effects. 2018-2019 data show that even with randomization, the first drug seen on the questionnaire was endorsed more often than the last (OR: 1.80, p< 0.001), regardless of drug type. A naïve approach assuming baseline prevalence as ground truth was also developed and found that estimates of lifetime drug use asymptotically approach half of what the true prevalence is as the number of survey items increases. This framework directly informs NMURx questionnaire design and could be adopted to study other surveys. Furthermore, quantifying these effects is particularly pertinent to drug abuse research since many national probabilistic surveys on drug use do not randomize the order in which questions are asked.