[R] It is possible to Expand RAM with HD space?

[Petri Palmu]

Hi, (closely related to the subject) 

First, I'm not an expert in programming,
so the terminology I use can be incorrect or vague.

I wonder if it's a good development to always defend from "out of memory error"
by buying more RAM because the size of analyseable data also increases at same
time. In addition (I see) a unwise development of operating systems (at least
MS) sets higher and higher requirements for the machine (which of course is not
the R's fault)

For the future, would it be possible to build a data/object file system parallel
to R workspace in which you typically could avoid loading all the data into use
memory. So to have a very scalable solution to the painful memory problem.
I am using R in a strong relationship with a Finnish statistical system
"Survo" (www.survo.fi) In the C-programmed Survo the data or matrix files are
usually never wholly loaded into memory. Minimally you can load just one element
from the data file (via C-function) into memory
(e.g. for calculating some statistics).

Of course in that way (since the data is not in fast memory) the operations are
slower compared to operating with R objects. But the other (and important) side
of the coin is that the size of free hard disk space is practically the only
limit in Survo (of course some methods (for computational reasons) require a
loading of bigger part of the data into fast memory).

I also see that a marriage with some database system or data warehouse can
sometimes cause a creative analysis work a bit clumsy.

Example: How I have used R within Survo variable transformations. In this way
you don't need to load all the data into R workspace (just the vectors of
interest).

A small example data in Survo editor where all the operations are done:

DATA _TMP1
X   Y
1   A
2   A
3   B
4   B
5   B

FILE COPY _TMP1 TO TMP1 / copy to Survo data file

FILE STATUS TMP1  / Structure of data
 Copy of data matrix _TMP1
FIELDS: (active)
   1 NA_   1 X        (#)
   2 SA_   1 Y
END
Survo data file TMP1: record=22 bytes, M1=6 L=64  M=2 N=5


Variable transformations:

CASE 1: Survo's own VAR operation

VAR X2=log(X) TO TMP1 / Activation in the editor

FILE LOAD TMP1
DATA TMP1*
 X Y          X2
 1 A       0.000
 2 A       0.693
 3 B       1.099
 4 B       1.386
 5 B       1.609


CASE 2: Identical VAR operation via R function
(using a sucro tehcnique (~macro) in Survo)

/R_VAR X2b=c(log(TMP1$X)) TO TMP1 / for your comments etc.

FILE LOAD TMP1   /  see if the equal results ;)
DATA TMP1*
 X Y          X2         X2b
 1 A       0.000       0.000
 2 A       0.693       0.693
 3 B       1.099       1.099
 4 B       1.386       1.386
 5 B       1.609       1.609


where /R_VAR is the sucro (I haven't listed the code here) carrying out the
"visit" in R workspace (log()) and then returning the output vector into Survo
data file TMP1. Perhaps any R script that results a vector of the data
row dim can be used as an argument for this simple sucro function.

I'm very excited about data processing with R. I see my approach just to
union the strengths of these two programs. Sometimes with larger data I do
the pre-processing mainly with Survo and then the final analyses (e.g. lmer())
in R perhaps in connection with a database 
(though Survo covers a wide set of statistical operations, the matrix
interpreter is great and it's easy do your C-modules for Survo).

Perhaps this all is also possible with SAS or SPSS ("I my opinion SPSS is inconvienent
and SAS I have never used but heard that it's at least overpriced" ;-)

Best regards,
Petri






-----Original Message-----
From: r-help-bounces at stat.math.ethz.ch [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Prof Brian Ripley
Sent: 17. huhtikuuta 2007 11:19
To: Uwe Ligges
Cc: r-help at stat.math.ethz.ch
Subject: Re: [R] It is possible to Expand RAM with HD space?

On Tue, 17 Apr 2007, Uwe Ligges wrote:

>
>
> Jorge Cornejo-Donoso wrote:
>> I'm using R in a winXp machine. I have problems with the size of my 
>> database (xyz), anyone know if is possible to use some kind of swap 
>> memory, in order to expand my real RAM and use HD space for the processes?
>
> This is done automatically by your operating system. You do not need 
> to care about it.

Except perhaps that on his OS it is limited to 2GB, which can be increased to 3GB in some circumstances (see the rw-FAQ Q2.9).  And anyone using less than 2GB RAM at current prices (except perhaps on a laptop) and is running out of RAM needs to buy some more: *far* cheaper than buying SAS.


>
> Uwe Ligges
>
>
>
>
>> I'm trying to make a variogram, and I don't want to sacrifice the 
>> precision of the data, e.g. using mean or median by defined pixels or 
>> something like that (now I have to do that, or use other software like SAS).
>>
>> Thanks in advance!
>>
>> ______________________________________________
>> R-help at stat.math.ethz.ch mailing list 
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide 
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide 
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

--
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.