Applying Agile Design Methods to Enrich and Accelerate Pretesting Benefits (303653)

Keywords: pretesting, questionnaire design, evaluation, multi-mode, web, nail, expert review, cognitive testing, crowdsourcing

Pretesting methods are ideally selected for the measurement errors they are best suited to detect. Methods for new questions or instruments may begin with rich, intensive exploratory methods such as ethnographic interviews and may follow with multiple rounds of focus groups, cognitive interviews, quantitative pretests, and behavior coding. Testing efforts may even culminate in a large operational ‘dress rehearsal’ of the planned survey. A full battery of testing is rare, because few survey programs are so new or altered that this would be deemed necessary. Fewer still allow the time and resources to conduct robust pretesting. Only the largest, most influential and well-funded federal surveys generally have the resources to support ongoing survey development research or years of iterative pretesting. This paper describes a worst-case scenario---small survey, small budget, little time, and a major redesign including new modes of data collection. With agile design methods, successive pretests for a large residential energy survey were successfully staged to redevelop a CAPI questionnaire for web and mail in the span of six months. An expert appraisal of the full questionnaire reduced the problem areas for cognitive testing, which in turn revealed question series that were candidates for targeted crowdsourcing techniques within two quantitative dress rehearsals. The result was a fully operational web and mail version of the revised questionnaire in 10 months. This paper suggests that many more survey programs with limited time and resources would benefit from similar adaptive pretesting methods.