Data Mining and Data Quality: Can't Have One Without the Other

• Activity Number: 364
• Type: Roundtables
• Date/Time: Tuesday, August 5, 2014 : 12:30 PM to 1:50 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Tamraparni Dasu*+
• Companies: AT&T Labs Research
• Keywords: Big Data Analytics ; Data Stream Mining ; Data Quality ; Anomaly detection ; metrics

Abstract

In the era of Big Data, data sources are increasingly global and heterogeneous, both in content and structure. When integrated properly and understood correctly, such data provide an unprecedented opportunity to have extreme global impact, and have been used by corporations, governments, and scientists all over the world. But, here is the catch: Data quality gets in the way. As the pioneers of Big Data analytics are beginning to realize, gathering and marshalling massive amounts of data is just the beginning. Data quality is a key component in ensuring the results of mining such intricate data remain valid and do not lead to misleading inferences. Incomplete data, duplicates, anomalies, truncated or mangled data, or the infamous "Jan1, 1970" birthday phenomenon are only a handful of the more well-known examples of data glitches that occur in complex, statistically interdependent ways with spatio-temporal correlations in real life. To detect, analyze, and quantify these data quality problems, we need a new set of sophisticated, statistically rigorous methodologies. Herein lies the opportunity for statistics to have a global impact in the future.