Data Cleansing and Preparation at the Gates: A Data Streaming Perspective

• Activity Number: 64
• Type: Invited
• Date/Time: Monday, August 9, 2004 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistical Graphics
• Author(s): Don Faxon*+ and R. Duane King and John T. Rigsby and Steve Bernard
• Companies: George Mason University and George Mason University and Naval Surface Warfare Center Dahlgren Division and George Mason University
• Keywords: data-cleaning ; streaming data ; massive datasets ; data preperation ; data collection

Abstract

Collection of internet traffic data at the gates of a large enterprise necessarily involves data-cleaning, integration, selection, and transformation, especially if data-streaming strategies are employed. The huge quantities of packets that typically cross the enterprise gateway make multiple passes through the data cost-prohibitive. Data-cleansing, customarily perceived as the removal of noise and inconsistent data, is instead seen as a flagging and tagging procedure to facilitate detection of malformed or corrupted IP packets associated with malicious intrusion, or subtle reconnaissance activity as precursor to a massive attack on the enterprise computing infrastructure. Since real-time or near-real-time implementation of data analysis comprising such innovative concepts as data streaming or evolutionary graphics, fast in-line data cleansing and preparation is required. This paper discusses and illustrates the strategies we have incorporated into our data collection and analysis.