High-Performance Computing Based on Massive Parallel Processing: Lessons Learned from the NORC Data Enclave

• Activity Number: 464
• Type: Contributed
• Date/Time: Wednesday, August 6, 2014 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistical Computing
• Author(s): Timothy Mulcahy*+ and Johannes Huessy and Scot Ausborn
• Companies: NORC at the University of Chicago and NORC at the University of Chicago and NORC at the University of Chicago
• Keywords: big data ; analytics ; data visualization ; massive parallel processing ; high performance computing

Abstract

NORC recently enhanced its secure Data Enclave to meet the challenges of big data using high performance computing based on massive parallel processing. This presentation will include a description of our Enterprise architectural and data visualization capabilities for manipulating structured and unstructured big data. We will highlight the performance gains by kicking off a SAS query on a flat file containing 50M records as a baseline measure. While this is running in the background, we will proceed with a description of the big data system in place for structured data before executing the same query using a SQL client connected to the platform. In addition, we will demonstrate how our customized implementation of Hadoop handles unstructured data by kicking off a query in Hadoop against one of our large unstructured datasets, describe the systems in place, and monitor results. Next, we will present on our data visualization capabilities using Tableau software and examine the SQL and ASP code behind a manually created online report, noting the resources required to implement it.