Text Classification and Big Data

• Activity Number: 338
• Type: Contributed
• Date/Time: Tuesday, July 31, 2012 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): David Afshartous*+ and George Michailidis
• Companies: Vanderbilt University and University of Michigan
• Keywords: machine learning ; distributed computing ; text analytics ; data mining ; text categorization

Abstract

The problem of text classification is central to many businesses in the information age where massive amounts of relevant data are readily available. Classic examples include e-mail spam, customer sentiment, diagnosis from electronic medical records (EMRs), and legal discovery document classification. Text classification may be viewed as a multi-step process that begins with transforming unstructured data into a structured format, e.g., a document-term matrix where rows represent documents and columns represent document features. Supervised learning methods may then be applied, where subsequent steps require many decisions and guidance for such decisions often differs between research domains. Such decisions include feature selection, dimensionality reduction, training set size, algorithm selection, and error analysis. In this paper, we consider text classification from the perspective of big data, where data size is affected by both the number of documents and the number of features employed by the learning algorithm. We assess the impact of big data on each step of text classification, and offer suggestions in the context of both open source and commercial software options.