[R] Memory management in R

[Lorenzo Isella]

On 10/08/2010 07:25 PM, Doran, Harold wrote:
> These questions are OS-specific. Please provide sessionInfo() or other details as needed
>


I see. I am running R on a 64 bit machine running Ubuntu 10.04

 > sessionInfo()
R version 2.11.1 (2010-05-31)
x86_64-pc-linux-gnu

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=C              LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
  [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base


and in case it matters, this is the output of my top command

$ top

top - 19:28:21 up  8:04,  8 users,  load average: 0.60, 0.72, 1.33
Tasks: 220 total,   1 running, 219 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.3%us,  0.6%sy,  0.0%ni, 87.2%id,  1.9%wa,  0.0%hi,  0.0%si, 
0.0%st
Mem:   6110484k total,  3847008k used,  2263476k free,    72748k buffers
Swap:  2929656k total,        0k used,  2929656k free,  2621420k cached

Cheers

Lorenzo

> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org]
> Sent: Friday, October 08, 2010 1:12 PM
> To: r-help
> Subject: [R] Memory management in R
>
> Dear All,
> I am experiencing some problems with a script of mine.
> It crashes with this message
>
> Error in grepl(fut_string, past_string) :
>     invalid regular expression
> '12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12653a6#12
> Calls: entropy_estimate_hash ->  total_entropy_lz ->  entropy_lz ->  grepl
> In addition: Warning message:
> In grepl(fut_string, past_string) : regcomp error:  'Out of memory'
> Execution halted
>
> To make a long story short, I use some functions which eventually call
> grepl on very long strings to check whether a certain substring is part
> of a longer string.
> Now, the script technically works (it never crashes when I run it on a
> smaller dataset) and the problem does not seem to be RAM memory (I have
> several GB of RAM on my machine and its consumption never shoots up so
> my machine never resorts to swap memory).
> So (though I am not an expert) it looks like the problem is some
> limitation of grepl or R memory management.
> Any idea about how I could tackle this problem or how I can profile my
> code to fix it (though it really seems to me that I have to find a way
> to allow R to process longer strings).
> Any suggestion is appreciated.
> Cheers
>
> Lorenzo
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.