Tune-Up Time: Using Focus Groups and Cognitive Interviews to Re-Evaluate an Established Survey (303614)

Keywords: focus groups, cognitive interviewing

The last 10 years have seen major changes in the treatment of and views toward mental health. Because of the Affordable Care Act, most health insurance plans in the United States now cover mental health and substance abuse disorder services. At the same time, the use of anti-depression, anti-anxiety, and other psychotropic drugs has steadily increased. Moreover, using mental health services may not be as stigmatizing as they were in the past, leading some who might otherwise avoid treatment to seek it out---and leading us as a society to be more willing to talk about these issues. In light of these important policy and social changes, we recently decided to reassess the Experience of Care and Health Outcomes (ECHO®) Survey. ECHO® was created over a decade ago to measure the experiences of adults and children utilizing mental health or substance abuse services. We use both cognitive interviews and focus groups to re-evaluate the instrument to see whether topics of interest and concern have changed over time and whether the questions themselves are still understood consistently and enable us to accurately describe patient experience in the changed environment. Like others, we find that focus groups and cognitive interviews are complementary approaches to evaluating questions, with each having different strengths in identifying issues in the ECHO survey instrument that need to be addressed. In our testing, we have found that issues of insurance limits on mental health care, the way mental health services are provided (routinely involving multiple providers), and the modes of communication in the Internet age are among the many things that have changed that require revision of the instrument. The re-evaluation of the ECHO instrument provides an excellent opportunity to revisit the issue of the relative strengths and weaknesses of focus groups and cognitive interviews as question evaluation strategies.