[R] R: "in-place" appending to a matrix.

[Duncan Murdoch]

On 10/16/2008 10:50 AM, culpritNr1 wrote:
> Hello Gabor,
> 
> First of all, thanks for your reply.
> 
> Indeed, your strategy solves the problem of speed=). My question was,
> however, more of an R design question.
> 
> Your suggestion of pre-allocating a big and continuous portion of memory is
> probably the first manual work around to try. And it will work.
> 
> Now, modern high level languages like the continually improving R, Python
> and also Matlab and Perl come with built-in goodies that run at C/C++ speed.
> Those built-in functions are high level versions of functionalities that we
> would code "manually" in low level languages.
> 
> Being relatively new to R, my expectations are that in a high level language
> like this, I do not need to manually program a simple operation such as
> in-place appending. Manipulation of rectangular data should be extremely
> efficient. (I use it to make computation of DNA properties in organisms with
> huge genomes like human and mouse.)

I think it's unlikely that this will soon be very efficient in R, 
because the only way to make it so is to pre-allocate a lot of extra 
space (so the appends don't grow the memory footprint).  Generally 
people run into space limitations that are harder to deal with than the 
speed limitations, so inefficient allocations for the sake of speed 
aren't going to happen unless the user asks for them (and Gabor told you 
how to do that).

Duncan Murdoch

> 
> So, do you know whether or not such functions exists or is planned to be
> incorporated in the near future? An optional module perhaps?
> 
> Again, thank you,
> 
> culpritNr1
> 
> 
> 
> 
> 
> Gabor Grothendieck wrote:
>> 
>> Create an empty matrix first and then fill it in.   That
>> will avoid the overhead in repeatedly expanding it.  If
>> you don't know how many rows then make it 1000 rows
>> and remove the unused ones once finished.
>> 
>> On Wed, Oct 15, 2008 at 4:05 PM, culpritNr1 <ig2ar-saf1 at yahoo.co.uk>
>> wrote:
>>>
>>> Hello fellow R sufferers,
>>>
>>> Is there a way to perform an appending operation in place?
>>>
>>> Currently, the way my pseudo-code goes is like this
>>>
>>> for (i in 1:1000) {
>>>    if (some condition) {
>>>        newRow <- myFunction(myArguments)
>>>        X <- rbind(X, newRow)     #  <- this is the bottleneck!!
>>>    }
>>> }
>>>
>>> As you can see, it works but as the matrix X gets the size of a few
>>> million
>>> rows, the code runs very slow.
>>>
>>> I am looking for something like the natively "in place" appending python
>>> function called "append()" or the perl function "push". "In-place"
>>> operations would allow me to do (in pseudocode)
>>>
>>> for (i in 1:1000) {
>>>    if (some condition) {
>>>        newRow <- myFunction(myArguments)
>>>        append(X, newRow)
>>>    }
>>> }
>>>
>>> You see? I do not have to call and re-assign the giant X matrix every
>>> loop
>>> cycle.
>>>
>>> Any help?
>>>
>>> Thank you,
>>>
>>> Your culpritNr1
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/R%3A-%22in-place%22-appending-to-a-matrix.-tp20001258p20001258.html
>>> Sent from the R help mailing list archive at Nabble.com.
>>>
>>> ______________________________________________
>>> R-help at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>> 
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>> 
>> 
>