[Rd] --max-vsize

[Christophe Rhodes]

Prof Brian Ripley <ripley at stats.ox.ac.uk> writes:

> Point 1 is as documented: you have exceeded the maximum integer and it
> does say that it gives NA.  So the only 'odd' is reporting that you
> did not read the documentation.

I'm sorry; I thought that my message made it clear that I was aware that
the NA came from exceeding the maximum representable integer.  To
belatedly address the other information I failed to provide, I use R on
Linux, both 32-bit and 64-bit (with 64-bit R).

> Point 2 is R not using the correct units for --max-vsize (it used the
> number of Vcells, as was once documented), and I have fixed.

Thank you; I've read the changes and I think they meet my needs.  (I
will try to explain how/why I want to use larger-than-integer
mem.limits() below.  If there's a better or more supported way to
achieve what I want, that'd be fine too)

> But I do wonder why you are using --max-vsize: the documentation says
> it is very rarely needed, and I suspect that there are better ways to
> do this.

Here's the basic idea: I would like to be able to restrict R to a large
amount of memory (say 4GB, for the sake of argument), but in a way such
that I can increase that limit temporarily if it turns out to be
necessary for some reason.

The desire for a restriction is that I have found it fairly difficult to
predict in advance how much memory a given calculation or analysis is
going to take.  Part of that is my inexperience with R, leading to
hilarious thinkos, but I think that part of that difficulty to predict
is going to remain even as I gain experience.  I use R both on
multi-user systems and on single-user-multiple-use systems, and in both
cases it is usually bad if my R session causes the machine to swap;
usually that swapping is not the result of a desired computation -- most
often, it's from a straightforward mistake -- but it can take
substantial amounts of time for the machine to respond to aborts or kill
requests, and usually if the process grows enough to touch swap it will
continue growing beyond the swap limit too.

So, why not simply slap on an address-space ulimit instead (that being
the kind of ulimit in Linux that actually works...)?  Well, one reason
is that it then becomes necessary to estimate at the start of an R
session how much memory will be needed over the lifetime of that
session; guess too low, and at some point later (maybe days or even
weeks later) I might get a failure to allocate.  My options at that
stage would be to save the workspace and restart the session with a
higher limit, or attempt to delete enough things from the existing
workspace to allow the allocation to succeed.  (Have I missed anything?)
Saving and restarting will take substantial time (from writing ~4GB to
disk) while deleting things from the existing session involves cognitive
overhead that is irrelevant to my current investigation and may in any
case not succeed to free enough.

So, being able to raise the limit to something generally large for a
short time to perform a computation, get the results, and then lower the
limit again allows me to protect myself in general from overwhelming the
machine with mistaken computations, while also allowing in specific
cases the ability to dedicate more resources to a particular
computation.

> I don't find reporting values of several GB as bytes very useful, but
> then mem.limits() is not useful to me either ....

Ah, I'm not particularly interested in the reporting side of
mem.limits() :-); the setting side, on the other hand, very much so.

Thank you again for the fixes.

Best,

Christophe