|
Abstract:
|
Most so called "big datasets" are also incomplete, with varying degrees of scarcity. Denoting such datasets as sparse datasets for convenience, they pose special difficulties across the board for traditional machine learning and classification algorithms. Yet, the future of big data is now, equating to some urgency in finding approaches for overcoming the challenges of analyzing sparse datasets. In this study we developed the Neighbor-Edges Based and Unbiased Leverage Algorithm (Nebula) to tackle sparse big data. The U.S. Environmental Protection Agency's (EPA) ToxCast project evaluated a diverse set of chemicals, including both environmental chemicals and drugs, using a broad panel of high-throughput in vitro assays. ToxCast data have been studied to characterize the toxicological profiles of environmental chemicals. However, the dataset has a high degree of missing elements and thus is sparse. To warrant full utilization of ToxCast dataset generated from a huge EPA investment in risk assessment of chemicals, novel and comprehensive analyzing the dataset is needed. Therefore, as a test, we applied Nebula and modularity analysis for ToxCast data. We found that the chemical-assay ne
|
