[R] cluster-lite

[Henrik Bengtsson]

The R.batch package does most of this.  It's been several years, but I
used it run batch jobs on multiple (30-40) machines with different
OS:es but sharing the same file system.  No communication needed
between hosts.  It worked like a charm and I could process 3-4 CPU
years in a few weeks for some bootstrap simulations.  It got exception
handling and much more.  See also r-help thread 'R.batch (Was: Re: [R]
Calling R from R and specifying "wait until script is finished")' on
May 22, 2005:

  https://www.stat.math.ethz.ch/pipermail/r-help/2005-May/071981.html

The installation is no longer as documented there, but instead you can
grab it from R-forge:

  http://r-forge.r-project.org/R/?group_id=428

This very moment, the r-forge server seems to be down, so you can also
install a copy via:

source("http://www.braju.com/R/hbLite.R");
hbLite("R.batch");

Then try this:

library("R.batch");
example("JobBatch");

See the FileProgressBar class in the R.utils package for reporting
progress via the file size (0-100 bytes), which allows you to use ls
-l (or ftp remotely) to check the progress of your jobs.

Feel free to do whatever you want with it.

/Henrik

On Tue, Sep 15, 2009 at 5:01 PM, ivo welch <ivo_welch at brown.edu> wrote:
> I am about to write a "cluster-lite" R solution for myself.  I wanted to
> know whether it already exists.  If not, I will probably write up how I do
> this, and I will make the code available.
>
> Background: we have various linux and OSX systems, which are networked, but
> not set up as a cluster.  I have no one here to set up a cluster, so I need
> a "hack" that facilitates parallel programming on standard networked
> machines.   I have accounts on all the machines, ssh access (of course
> password-less), and networked file directory access.
>
> what I am ultimately trying to accomplish is built around a "simple"
> function, that my master program would invoke:
>
> master.R:
>   multisystem( c("R slv.R 1 20 file1.out", "R slv.R 21 40 file2.out", "ssh
> anotherhost R slv.R 41 80 file3.out"), announce=300)
>
> multisystem() should submit all jobs simultaneously and continue only after
> all are completed.  it should also tell me every 300 seconds what jobs it is
> still waiting for, and which have completed.
>
> with basically no logic in the cluster, my master and slv programs have to
> make up for it.  master.R must have the smarts to know where it can spawn
> jobs and how big each job should be.  slv.R must have the smarts to place
> its outputs into the marked files on the networked file directory.  master.R
> needs the smarts to combine the outputs of all jobs, and to resubmit jobs
> that did not complete successfully.  again, the main reason for doing all of
> this is to avoid setting up a cluster across OSX and linux system, and still
> to make parallel processing across linux/osx as easy as possible.  I don't
> think it gets much simpler than this.
>
> now, I know how to write the multisystem() in perl, but not in R.  so, if I
> roll it myself, I will probably rely on a mixed R/perl system here.  This is
> not desirable, but it is the only way I know how to do this.  if something
> like multisystem() already exists in R native, please let me know and save
> me from reinventing the wheel.  if it does not, some perl/R combo for this
> soon will.
>
> regards,
>
> /iaw
>
>
> --
> Ivo Welch (ivo.welch at brown.edu, ivo.welch at gmail.com)
>
>        [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>