Analyzing Large Data Sets in Astronomy

• Activity Number: 318
• Type: Invited
• Date/Time: Tuesday, August 3, 2010 : 10:30 AM to 12:20 PM
• Sponsor: Council of Chapters
• Author(s): Alexander Sandor Szalay*+
• Companies: The Johns Hopkins University
• Keywords: statistics ; astrophysics ; big data ; algorthims

Abstract

Modern sky surveys generate unprecedented amounts of data in all wavelengths of the optical spectrum. Much of this data is accessible on-line, in a digital form. The data have large dimensionality, spatial and temporal properties. Their analysis represents new challenges, both from a simple data management perspective, and algorithmically. As data sets reach petabytes, the notion of a perfect answer is meaningless, and we need to think about randomized, incremental and robust algorithms that give 'good enough' answers in the shortest time. The emergence of novel multicore computer architectures require us to rethink how to optimally address certain statistical questions. In some cases simple O(N^2) algorithms, running 100,000 times in parallel, perform better than complex tree traversals with log(N) scaling. We will present some examples of these challenges.