[R] Large loops in R

Charles Novaes de Santana charles.santana at gmail.com
Thu Dec 6 16:52:09 CET 2012


Thank you all for your help!

I used the function "big.matrix", from a package named "bigmemory", to
allocate a large matrix 50k x 50k
(http://www.inside-r.org/packages/cran/bigmemory/docs/bigmemory).

And I followed the suggestions from Sarah to do the calculations fastly!

Thank you very much!

Best!

Charles

On Tue, Dec 4, 2012 at 9:33 PM, Sarah Goslee <sarah.goslee at gmail.com> wrote:
> I don't think there's any reason for the calculation you're doing that
> you must have the whole matrix in memory, is there?
>
> Unless there's something more than what you've shown us, you're just
> taking the sum of elementwise operations. You can read the matrix in
> in manageable chunks, take the sum of that chunk and save the single
> value. Repeat, then add them all up at the end.
>
> Sarah
>
> On Tue, Dec 4, 2012 at 3:14 PM, Charles Novaes de Santana
> <charles.santana at gmail.com> wrote:
>> On Tue, Dec 4, 2012 at 8:43 PM, Peter Langfelder
>> <peter.langfelder at gmail.com> wrote:
>>> On Tue, Dec 4, 2012 at 11:27 AM, Charles Novaes de Santana
>>> <charles.santana at gmail.com> wrote:
>>>> Dear Michael,
>>>>
>>>> Thank you for your answer.
>>>>
>>>> I have 2 matrices. Each position of the matrices is a weight. And I
>>>> need to calculate the following sum of differences:
>>>>
>>>> Considering:
>>>> mat1 and mat2 - two matrices (each of them 48000 x 48000).
>>>> d1 and d2 - two constant values.
>>>>
>>>> sum<-0;
>>>> for(i in 1:nrows1){
>>>>                         for(j in 1:nrows2){
>>>>                                         sum<-sum+ ( ( (mat1(i,j)/d1) -
>>>> (mat2(i,j)/d2) )^2 )
>>>>                                 }
>>>>                         }
>>>>                 }
>>>>
>>>> I was wondering if there is a better way to do this sum.
>>>
>>> sum( (mat1/d1-mat2/d2)^2)
>>>
>>> Correct me if I'm wrong though - aren't matrices of 48x times 48k
>>> larger than what R can handle at present?
>>>
>>> HTH
>>>
>>> Peter
>>
>> hmmm I didn't know that the limitation of R was below this value. I
>> found this error message:
>>
>> "Error in matrix(0, 48000, 48000) : too many elements specified"
>>
>> but I thought it was a machine limitation (and I was asking for access
>> to a better machine in my labs...). Thanks for clarifying it.
>>
>> Well, when Sarah gave me the answer for my problem, I got a new one :)
>> Thank you, Sarah and Peter.
>>
>> So, is there any other way to "trick R" and allocate such large matrices?
>>
>> Best,
>>
>> Charles
>> --
>> Um axé! :)
>>
>
> --
> Sarah Goslee
> http://www.functionaldiversity.org



-- 
Um axé! :)

--
Charles Novaes de Santana
http://www.imedea.uib-csic.es/~charles
PhD student - Global Change
Laboratorio Internacional de Cambio Global
Department of Global Change Research
Instituto Mediterráneo de Estudios Avanzados(CSIC/UIB)
Calle Miquel Marques 21, 07190
Esporles - Islas Baleares - España

Office phone - +34 971 610 896
Cell phone - +34 660 207 940




More information about the R-help mailing list