Detecting Novel Bivariate Associations in Large Data Sets

• Activity Number: 176
• Type: Contributed
• Date/Time: Monday, July 30, 2012 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Yakir Reshef*+ and David Reshef and Hilary Kiyo Finucane and Sharon Grossman and Gilean McVean and Peter Turnbaugh and Eric Lander and Michael Mitzenmacher and Pardis Sabeti
• Companies: and Harvard/MIT Division of Health Science and Technology and Weizmann Institute of Science and Harvard/MIT Division of Health Science and Technology and Oxford University and Harvard University and Broad Institute of MIT and Harvard and Harvard University and Harvard University
• Keywords: data exploration ; dependence ; association ; data mining ; exploratory data anlysis

Abstract

Identifying interesting relationships between pairs of variables in large data sets is increasingly important. One way of doing so is to search such data sets for pairs of variables that are closely associated. This can be done by calculating some measure of dependence for each pair, ranking the pairs by their scores, and examining the top-scoring pairs. We outline two heuristic properties--generality and equitability--that the statistic we use to measure dependence should have in order for such a strategy to be effective.

We present a measure of dependence for two-variable relationships, the maximal information coefficient (MIC), that appears to have these properties. MIC captures a wide range of associations both functional and not (generality), and assigns similar scores to relationships with similar noise levels, regardless of relationship type (equitability). Finally, we show that MIC belongs to a larger class of maximal information-based nonparametric exploration (MINE) statistics for identifying and classifying relationships.