Estimating Bias-Corrected Mutual Information for Analysis of Large Complex Data Sets

• Activity Number: 361
• Type: Contributed
• Date/Time: Tuesday, August 6, 2013 : 10:30 AM to 12:20 PM
• Sponsor: IMS
• Author(s): Susan Wilson*+ and Chris Pardy
• Companies: ANU & UNSW and University of New South Wales
• Keywords: large-scale data ; exploratory data analysis ; mixed data types ; mutual information ; bias correction ; genomic data

Abstract

Exploratory analysis of high-dimensional data sets is becoming routine in many scientific fields, and is posing statistical (including computational) challenges. One of these is how to simultaneously evaluate, pairwise, data of mixed types, namely both discrete and continuous measurements in a statistically sound way. So we have developed a nonparametric approach based on mutual information. Simulations have provided guidelines for parameter choice and shown where improvements can be made (in particular reducing the mean squared error) with bias correction using the jackknife, giving the bias corrected mutual information (BCMI) estimator. Following statistical filtering, these association estimates can be read into, say, clustering and network visualization routines for further exploration. For illustrative purposes, results from part of an analysis of a moderate-sized genomic data set are given. Briefly, comparisons with other measures of association are overviewed. An R package has been developed that is multithreaded and fast, and designed to be able to run over clusters.