|
Abstract:
|
While parallelization on a local machine can be accomplished using the R packages foreach and doParallel, the local machine is often not an appropriate environment for large-scale tasks. In this tutorial, we show the user how to use a cluster of two or more computers to create and process a swarm of independent jobs. Simply put, the user writes an R script with NULL parameters and a shell script that creates a list of command lines, each dependent on a set of parameters. The resulting file is executed by the batch operating system, and the command-line arguments are passed to R using the parseCommandArgs function in the batch package. A group of these commands will be run in parallel, and the resulting output will be transferred back to R and post-processed using base R. The tutorial example, which employs the central limit theorem and takes seconds to compute, was chosen so that the user can first run on a local machine before reproducing on a cluster. The user is expected to have access to a computing cluster and basic knowledge of secure file transfer and the command line. The tutorial is written using the bash shell and, as such, assumes that the target system is Unix-like.
|
