Integrating R with New Algorithms, Data Structures, and Data Sources

• Activity Number: 267
• Type: Invited
• Date/Time: Tuesday, July 31, 2012 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistical Computing
• Author(s): Duncan Temple Lang*+
• Companies: University of California at Davis
• Keywords: R ; compiler ; large data ; extensible ; large data ; block algorithms

Abstract

I'll talk about 2 approaches we are exploring to allow R to better handle large volumes of data faster and more flexibly. The first is an experiment in how to make an R-based interpreter extensible so that developers of packages can introduce new first-class data types/structures that are tightly integrated with the interpreter and existing core code. This allows us to introduce alternative implementations of common data structures where the data are structured in very different ways (e.g. streaming data) or even remotely (e.g. out of memory).

The second approach analyzes entire R "scripts" and attempts to determine the flow and lifetimes of data objects. It uses this information to try to improve the organization and reuse of memory. It also uses LLVM (Low Level Virtual Machine) to compile simple R functions into machine code that can exploit context-specific information. These can generate fast code that cumulate information by working on individual or sub-blocks of observations as opposed to all-data-in-memory algorithms.

The talk is intended to stimulate discussion of future systems and how we might "get there from here".