[R] Memory Problem

[jim holtman]

Have you tried gc() to see if any memory is released?  How big was the
file that you read in?  I don't see any large objects that appear in
your workspace.  Is there some type of processing that you did after
reading in the data?  You might want to intersperse the following
command in your script so that you can track where memory utilization
is going up:

print(memory.size())

I would try this with a smaller dataset size to see what happens.
Take a set of metrics and determine what happens as the size of the
data file is increased.  It is hard to tell without the actual script
to see what, and how, the processing is done.

Again, are there other alternatives that you might want to consider:
using a database, reading in only the columns of data you need,
preprocessing the data into smaller files, etc.  Besides reading in
the data, exactly what do you want to do with it and how much of it is
actually required for the processing?  For example, I have scripts
that only read in the data and then write out the object for later
processing since it is usually the reading and initial processing that
takes a lot of time.  This is another way of partitioning the work.
Anytime I have problems with processing data, I always take a smaller
chunk (cutting it half each time) till I can at least read it in in a
reasonable time.  One of the skills that you have to learn is to how
to debug your programs; not only actual bugs in your script, but
workarounds that may have to be created due to some constraint in the
system(s) that you are using.  This is a good place to practice design
of experiments.

On Sat, Feb 6, 2010 at 8:09 AM, Meenakshi
<meenakshichidambaram at gmail.com> wrote:
>
> This is my objects size:
>
>                                   Size     Mode
> asa_Condition               912     list
> asa_GatedCommunity        9,912     list
> asa_Neighbourhood         2,872     list
> asa_Security                832     list
> asa_Storeys                 800     list
> Condition_adju              560     list
> final_Condition             672     list
> final_GatedCommunity      3,936     list
> final_Neighbourhood       1,376     list
> final_Security              608     list
> final_Storeys               616     list
> GatedCommunity_adju       3,000     list
> model_Condition             648     list
> model_GatedCommunity        648     list
> model_Neighbourhood         648     list
> model_Security              648     list
> model_Storeys               640     list
> modeling1             9,157,856     list
> mult                  3,613,576     list
> my.object.size            6,912 function
> Neighbourhood_adju        1,080     list
> Security_adju               512     list
> Storeys_adju                520     list
> **Total              12,809,784  -------
> Warning message:
> In structure(.Internal(object.size(x)), class = "object_size") :
>  Reached total allocation of 1535Mb: see help(memory.size)
> --
> View this message in context: http://n4.nabble.com/Memory-Problem-tp1459740p1471251.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>



-- 
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem that you are trying to solve?