A 3-Stage Approach to Code-Frame Development for Open-Ended Survey Responses: Software Assisted, Person-Ware Refined, Statistically Evaluated

• Activity Number: 403
• Type: Contributed
• Date/Time: Tuesday, August 5, 2014 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Bernard L. Dugoni*+ and Kevin Brown
• Companies: NORC at the University of Chicago and NORC at the University of Chicago
• Keywords: text analysis ; Data mining ; Survey Research

Abstract

In a study of the priorities of educators in the midwest, a three step process was used to assess open-ended responses, combining the use of text-analysis software with researcher review techniques for a tailored solution that is computer assisted but retains the human element. The first stage used the software "QDAMiner" developed by Provalis Research to conduct content analysis based on word frequency. In the second stage, related words were combined into phrases by using "keyword-in-context" analysis, which assesses whether the same words or phrases share an single concept or mean different things to different respondents. At this stage, responses were combined by the program into a matrix of co-occurrence, which allows the analysis of cases where two or more of the keywords are used in a given response. The software tools used at the initial stages do not capture nuance or ambiuity, so it is important that post-processing be done by the researcher. The third stage involved an analysis of internal consistency and review by multiple raters to achieve useful qualitative categories which were combined for further analysis with quantitative variables also collected in the project.