Cloud-Scale Alignment of NGS Short Reads

• Activity Number: 596
• Type: Topic Contributed
• Date/Time: Thursday, August 7, 2014 : 8:30 AM to 10:20 AM
• Sponsor: Biometrics Section
• Author(s): Hao Xiong*+
• Keywords: Big Data ; Next-generation sequencing ; Cloud computing ; Distributed computing ; Alignment

Abstract

As sequencing throughput continue to outpace Morris' law, scientists increasingly rely on clouding computing to process and analyze NGS data. However, the current software are mostly file based, making I/O a major bottleneck in large-scale NGS data analysis. Short-reads alignment, a crucial step in many NGS data analysis, has standardized on a compressed and indexed file format called BAM. The BAM format scales poorly in distributed computing environment. Avro and Parquet arise from high-performance distributed computing and offer data format flexibility and cloud-scale data processing. Recently Massie et al. (2014) has proposed to apply Avro and Parquet to NGS data analysis and demonstrated the advantages with a variant-calling example. In this presentation, we will demonstrate the flexibility and speed of Avro, Parquet in NGS data analysis by implementing short reads alignment. By taking advantage of next-generation cloud computing frameworks, our software can scale with data, vastly shorten processing time, and allow real-time interactive query. Sequencing technology is outpacing Morris' law; it is high time that bioinformatics software catch up.