[R] Matrix scalar operation that saves memory?

[Shunran Zhang]

Thanks for the info.

For the data type, my matrix as of now is indeed a matrix in a perfect 
square shape filled in a previous segment of code, but I believe I could 
extract one row/column at a time to do some processing. I can also 
change that previous part of code to change the data type of it to 
something else if that helps.

Saving it to a file for manipulation and reading it back seems to be 
quite IO intensive - writing 600G of data and reading 300G back from a 
hard drive would make the code extremely heavy as well as wear my disk 
quite badly.

For now I'll try the row-by-row method and hope it works...

Sincerely,
S. Zhang


On 2023/04/12 12:39, avi.e.gross using gmail.com wrote:
> The example given does not leave room for even a single copy of your matrix
> so, yes, you need alternatives.
>
> Your example was fairly trivial as all you wanted to do is subtract each
> value from 100 and replace it. Obviously something like squaring a matrix
> has no trivial way to do without multiple copies out there that won't fit.
>
> One technique that might work is a nested loop that changes one cell of the
> matrix at a time and in-place. A variant of this might be a singe loop that
> changes a single row (or column) at a time and in place.
>
> Another odd concept is to save your matrix in a file with some format you
> can read back in such as a line or row at a time, and then do the
> subtraction from 100 and write it back to disk in another file. If you need
> it again, I assume you can read it in but perhaps you should consider how to
> increase some aspects of your "memory".
>
> Is your matrix a real matrix type or something like a list of lists or a
> data.frame? You may do better with some data structures that are more
> efficient than others.
>
> Some OS allow you to use virtual memory that is mapped in and out from the
> disk that allows larger things to be done, albeit often much slower. I also
> note that you can remove some things you are not using and hope garbage
> collection happens soon enough.
>
> -----Original Message-----
> From: R-help <r-help-bounces using r-project.org> On Behalf Of Shunran Zhang
> Sent: Tuesday, April 11, 2023 10:21 PM
> To: r-help using r-project.org
> Subject: [R] Matrix scalar operation that saves memory?
>
> Hi all,
>
> I am currently working with a quite large matrix that takes 300G of
> memory. My computer only has 512G of memory. I would need to do a few
> arithmetic on it with a scalar value. My current code looks like this:
>
> mat <- 100 - mat
>
> However such code quickly uses up all of the remaining memory and got
> the R script killed by OOM killer.
>
> Are there any more memory-efficient way of doing such operation?
>
> Thanks,
>
> S. Zhang
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>