[R] Suggestions for poor man's parallel processing

[Luke Tierney]

I've been working on a simple interface for this sort of thing modeled
loosely on the Python CoW (Cluster of Workstations) package.  A rough
draft writeup with a link to the preliminary package is at
http://www.stat.umn.edu/~luke/R/cluster/cluster.html. The idea is to
provide a very simple front end for handling things like farming out
simulations to a bunch of machines (or a bunch of processors on one
machine) and collecting the results.  The communications back ends
that are supported are sockets or pvm via Michael Li and Tony
Rossini's rpvm; mpi via Hao Yu's Rmpi should be eventually possible as
well.  Michael and Tony's rsprng is also supported.  It's very rough,
but I won't get to cleaning it up for a week or two at least, so if
anyone wants to play with it in the mean time, go ahead.

luke

On Wed, May 08, 2002 at 08:45:47AM -0400, David Kane  <David Kane wrote:
> Almost all of the heavy crunching I do in R is like:
> 
> > for(i in long.list){
> + do.something(i)
> + }
> > collect.results()
> 
> Since all the invocations of do.something are independent of one another, there
> is no reason that I can't run them in parallel. Since my machine has four
> processors, a natural way to do this is to divide up long.list into 4 pieces
> and then start 4 jobs, each of which would process 1/4 of the items. I could
> then wait for the four jobs to finish (waiting for tag files and the like),
> collect the results, and go on my happy way. I might do this all within R
> (using system calls to fork off other R processes?) or by using Perl as a
> wrapper.
> 
> But surely there are others that have faced and solved this problem already! I
> do not *think* that I want to go into the details of RPVM since my needs are so
> limitted. Does anyone have any advice for me? Various postings to R-help have
> hinted at ideas, but I couldn't find anything definitive. I will summarize for
> the list.
> 
> To the extent that it matters:
> 
> > R.version
>          _                   
> platform sparc-sun-solaris2.6
> arch     sparc               
> os       solaris2.6          
> system   sparc, solaris2.6   
> status                       
> major    1                   
> minor    5.0                 
> year     2002                
> month    04                  
> day      29                  
> language R                   
> 
> 
> Regards,
> 
> Dave Kane
> -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
> r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
> Send "info", "help", or "[un]subscribe"
> (in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
> _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

-- 
Luke Tierney
University of Minnesota                      Phone:           612-625-7843
School of Statistics                         Fax:             612-624-8868
313 Ford Hall, 224 Church St. S.E.           email:      luke at stat.umn.edu
Minneapolis, MN 55455 USA                    WWW:  http://www.stat.umn.edu
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._