Designing Software for Statistical Analysis of Huge Collections of Sequencing Data

• Activity Number: 2
• Type: Invited
• Date/Time: Sunday, August 3, 2014 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Graphics
• Author(s): Ben Langmead*+
• Companies: Johns Hopkins University
• Keywords: genomics ; scalability ; MapReduce ; alignment ; RNA-seq

Abstract

For 4-5 years following the advent of second-generation sequencing, per-instrument throughput increased at a rate of about 4-fold per year. The challenge for computational genomics is to make it easy for genomics researchers, including statisticians, to use the very large genomics datasets that have been generated as a result. Many studies seek to distinguish a condition of interest (e.g. disease) from the background of biological variability, sequencing error, and bias. Careful study design is key, but it is also helpful to consider many datasets at once. But designing software that can easily analyze many, large sequencing datasets is a significant technical challenge. I discuss recent ideas about how to design software that can be used to study many, large sequencing datasets. I discuss how the MapReduce framework developed at Google can (or cannot) be used to scale genomics applications up to use many computers at once, in a way that confers efficiency and fault tolerance. I also describe cloud-enabled, scalable software pipelines developed by myself and others, and show how they can yield new insights into sequencing error, bias, and variability.