An Examination of Reliability of Text Mining

• Activity Number: 532
• Type: Contributed
• Date/Time: Wednesday, August 4, 2010 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Chong Ho Yu*+ and Angel Jannasch-Pennell and Samuel DiGangi
• Companies: Arizona State University and Arizona State University and Arizona State University
• Keywords: text mining ; reliability ; data mining

Abstract

Inter-rater reliability of textual analysis conducted with qualitative methods has been a concern to many researchers. While text mining is claimed to be a reliable technique for its sophisticated algorithms, research on comparing the agreement among the results using the same dataset but different text mining applications is virtually absent. To fill this vacuum, this project compares the results of several text mining packages. The same data source, which encompasses student blogs at a university, will be used for extracting common threads. Because there is no single best solution as the benchmark, the objective of this project is not to rank the accuracy of these packages. Rather, strategies for evaluating inter-coder reliability were employed. It was found that there is a high degree of inconsistency among text miners and thus triangulation for text mining is strongly recommended.