Programming with Big Data in R

From Wikipedia, the free encyclopedia
Jump to: navigation, search
pbdR
Pbdr.png
Paradigm(s) SPMD and MPMD
Designed by Wei-Chen Chen, George Ostrouchov, Pragneshkumar Patel, and Drew Schmidt
Developer pbdR Core Team
Appeared in Sep. 2012
Preview release Through GitHub at RBigData
Typing discipline Dynamic
Influenced by R, C, Fortran, and MPI
OS Cross-platform
License General Public License and Mozilla Public License
Website http://www.r-pbd.org/

Programming with Big Data in R (pbdR)[1] is a series of R packages and an environment for statistical computing with Big Data by using high-performance statistical computation.[2] The pbdR uses the same programming language as R with S3/S4 classes and methods which is used among statisticians and data miners for developing statistical software. The significant difference between pbdR and R codes is pbdR mainly focuses on distributed memory system where data are distributed across several processors and analyzed in a batch mode, while communications between processors are based on MPI which is easily used in large high-performance computing (HPC) systems. R system mainly focuses[citation needed] on single multi-core machines for data analysis via an interactive mode such as GUI interface.

Two main implementations in R using MPI are Rmpi[3] and pbdMPI of pbdR.

The idea of SPMD parallelism is to let every processors do the same works but on different parts of large data. For example, modern GPU is a large collection of slower co-processors which can simply apply the same computation on different parts of relatively smaller data, but the SPMD parallelism ends up an efficient way to obtain final solutions, i.e. time to solution is shorter.[4] It is clearly that pbdR is not only suitable for small clusters, but also is stabler for analyzing Big data and is more scalable for supercomputers.[5] In short, pbdR

  • does not like Rmpi, snow, snowfall, do-like, nor parallel packages in R,
  • does not focus on interactive computing nor master/workers,
  • but is able to use both SPMD and task parallelisms.

Package design[edit]

Programming with pbdR requires usage of various packages developed by pbdR core team. Packages developed are the following.

General I/O Computation Application Profiling
pbdDEMO pbdNCDF4 pbdDMAT pmclust pbdPROF
pbdMPI pbdBASE
pbdSLAP
The images describes how various pbdr packages are correlated.

Among these packages, pbdMPI provides wrapper functions to MPI library, and it also produces a shared library and a configuration file for MPI environments. All other packages rely on this configuration for installation and library loading that avoid difficulty of library linking and compiling. All other packages can directly use MPI functions easily.

  • pbdMPI --- an efficient interface to MPI either OpenMPI or MPICH2 with a focus on Single Program/Multiple Data (SPMD) parallel programming style
  • pbdSLAP --- bundles scalable dense linear algebra libraries in double precision for R, based on ScaLAPACK version 2.0.2 which includes several scalable linear algebra packages (namely BLACS, PBLAS, and ScaLAPACK).
  • pbdNCDF4 --- Interface to Parallel Unidata NetCDF4 format data files
  • pbdBASE --- low-level ScaLAPACK codes and wrappers
  • pbdDMAT --- distributed matrix classes and computational methods, with a focus on linear algebra and statistics
  • pbdDEMO --- set of package demonstrations and examples, and this unifying vignette
  • pmclust—parallel model-based clustering using pbdR
  • pbdPROF—profiling package for MPI codes and visualization of parsed stats

Amount those packages the pbdDEMO package is a collection of 20+ package demos which offer example uses of the various pbdR packages, and contains a vignette which offers detailed explanations for the demos and provides some mathematical or statistical insight.

Examples[edit]

Example 1[edit]

Hello World! Save the following code in a file called ``demo.r``

### Initial MPI
library(pbdMPI, quiet = TRUE)
init()
 
comm.cat("Hello World!\n")
 
### Finish
finalize()

and use the command

mpiexec -np 2 Rscript demo.r

to execute the code where Rscript is one of command line executable program.

Example 2[edit]

The following example modified from pbdMPI illustrates the basic syntax of the language of pbdR. Since pbdR is designed in SPMD, all the R scripts are stored in files and executed from the command line via mpiexec, mpirun, etc. Save the following code in a file called ``demo.r``

### Initial MPI
library(pbdMPI, quiet = TRUE)
init()
.comm.size <- comm.size()
.comm.rank <- comm.rank()
 
### Set a vector x on all processors with different values
N <- 5
x <- (1:N) + N * .comm.rank
 
### All reduce x using summation operation
y <- allreduce(as.integer(x), op = "sum")
comm.print(y)
y <- allreduce(as.double(x), op = "sum")
comm.print(y)
 
### Finish
finalize()

and use the command

mpiexec -np 4 Rscript demo.r

to execute the code where Rscript is one of command line executable program.

Example 3[edit]

The following example modified from pbdDEMO illustrates the basic ddmatrix computation of pbdR which performs singular value decomposition on a given matrix. Save the following code in a file called ``demo.r``

# Initialize process grid
library(pbdDMAT, quiet=T)
if(comm.size() != 2)
  comm.stop("Exactly 2 processors are required for this demo.")
init.grid()
 
# Setup for the remainder
comm.set.seed(diff=TRUE)
M <- N <- 16
BL <- 2 # blocking --- passing single value BL assumes BLxBL blocking
dA <- ddmatrix("rnorm", nrow=M, ncol=N, mean=100, sd=10)
 
# LA SVD
svd1 <- La.svd(dA)
comm.print(svd1$d)
 
# Finish
finalize()

and use the command

mpiexec -np 2 Rscript demo.r

to execute the code where Rscript is one of command line executable program.

Further reading[edit]

References[edit]

  1. ^ Ostrouchov, G., Chen, W.-C., Schmidt, D., Patel, P. (2012). "Programming with Big Data in R". 
  2. ^ Chen, W.-C. and Ostrouchov, G. (2011). "HPSC -- High Performance Statistical Computing for Data Intensive Research". 
  3. ^ a b Yu, H. (2002). "Rmpi: Parallel Statistical Computing in R". R News. 
  4. ^ Mike Houston. "Folding@Home - GPGPU". Retrieved 2007-10-04. 
  5. ^ Schmidt, D., Ostrouchov, G., Chen, W.-C., and Patel, P. (2012). "Tight Coupling of R and Distributed Linear Algebra for High-Level Programming with Big Data". High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion:: 811–815. 
  6. ^ "100 most read R posts in 2012 (stats from R-bloggers) – big data, visualization, data manipulation, and other languages".