Communication Graphs and Text Analysis of Email

• Activity Number: 129
• Type: Topic Contributed
• Date/Time: Monday, August 8, 2005 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statisticians in Defense and National Security
• Author(s): Elizabeth Leeds*+ and David Marchette
• Companies: Naval Surface Warfare Center, Dahlgren Division and Naval Surface Warfare Center, Dahlgren Division
• Keywords: text processing ; spectral graph analysis ; random forests ; Enron email dataset

Abstract

The Enron email dataset is a collection of more than a half million email messages from more than 150 Enron employees from 1999 through 2002. Using this dataset, one could consider communication graphs constructed by placing an edge between employees when they exchange an email. Alternatively, one could consider a text intersection graph where an edge exists between employees when the content of what they email is judged to be similar. We compare the communication graphs to the text intersection graphs and claim the difference in the graphs illustrates the difference between who people know and what people know. Spectral graph techniques are used to explore the data. We utilize both intersection graphs and random forests to represent the data. The dissimilarity matrices that result from these representations capture two views of the relationships between the data. Studying these dissimilarity matrices gives insight into the structure of the data.