Stat Bandit

Musings on statistics, computation and data research

Quick and dirty parallel processing in R

R has some powerful tools for parallel processing, which I discovered while searching for ways to fully utilize my 8-core computer at work. What surprised me is how easy it is…about 6 lines of code, if that. Given that I wasn’t allowed to install heavy duty parallel-processing systems like MPICH on the computer, I found that the library SNOW fit the bill nicely through its use of sockets. I also discovered the libraries foreach and iterators, which were released to the community by the development team at Revolution R. Using these 3 libraries, I could easily parallelize a transformation of my dataset where the transformations happened within each unique ID. The following code did the trick:

library(foreach)
library(doSNOW)
cl <- makeCluster(6, type="SOCK") # using 6 nodes
registerDoSNOW(cl)
uID <- unique(ID)
foreach(i=icount(length(uID)) %dopar% {
transformData(dat[dat$ID==uID[i],])
}
stopCluster(cl)

Note that this is for a multiprocessor single computer. Doing this on a cluster may be more complicated, but this serves my purposes quite nicely. There are other choices for this, including the multicore library and others described in the CRAN Task View

Update: I found that this strategy did not work for R 2.11 Windows versions, since snow is not properly spawning processes. However, there is a library doSMP provided by Revolution Analytics which gets around this problem. So replacing doSNOW with doSMP should do the trick. 

Update (7/25/2011): It appears that SNOW does work again in R 2.13.0, the current version, on Windows. I’ve been using the snowfall package recently on my multi-core WinXP64 computer, and it works beautifully.

 

About these ads

2 responses to “Quick and dirty parallel processing in R

  1. AVSuresh April 11, 2011 at 4:24 AM

    Hi Abhijit,
    Can you guide me how to do this on windows server 64 bit?

    Thanks,
    Suresh

    • Abhijit May 7, 2011 at 7:19 PM

      Sorry for the late reply. I’ve been having a bit of difficulty with R 2.12 onwards. I’ll follow up with you later this week.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 391 other followers

%d bloggers like this: