[R] Easy method to set user-mode virtual memory space in Windows Vista and 7

[Kingsford Jones]

Thank you for the response, Prof Ripley.  Some comments below

On Fri, Oct 30, 2009 at 9:27 AM, Prof Brian Ripley
<ripley at stats.ox.ac.uk> wrote:
> On Wed, 28 Oct 2009, Kingsford Jones wrote:
>
>> I thought I'd share this with the list since it appears to provide a
>> quick fix to some memory problems, and I haven't see it discussed in
>> relation to R.
>
> Seems you didn't look in the places mentioned in the posting guide, fo this
> is an actual FAQ!
>
> See the references in rw-FAQ Q2.9 (and this has been documented there for
> many years, and the R developer made R large-address-aware so users who
> looked at the rw-FAQ could benefit from this).

I did read the rw-FAQ, including the links from 2.9.  I am, on a daily
basis, appreciative of the impressive effort that has gone into making
R what it is, including the documentation;  however, AFAICS the links
currently in 2.9 do not tell Vista or 7 users how to set the \3GB
switch.  I see discussion of changing the boot.ini file to set the
switch, but no mention of using BCDEdit.exe.  My plan was to send the
msg to r-help and if nobody indicated it was a bad idea, make a
suggestion on r-devel that the link I sent be added to rw-FAQ 2.9.


>
> See http://msdn.microsoft.com/en-us/library/bb613473%28VS.85%29.aspx for a
> more rounded view covering all the relevant OSes.

That is a more helpful link than the one sent; perhaps it can be added
to the rw-FAQ?

>
>> To reallocate virtual memory from kernel-mode to user-mode in 32-bit
>> Vista or Windows 7 one can use the increaseuserva boot option value.
>> See
>> http://msdn.microsoft.com/en-us/library/aa906211.aspx
>>
>> On my 4GB Vista machine, R is now able to use 3GB memory (previously
>> at the default value of 2GB).
>>
>> Here's the method:
>>
>> Open cmd.exe as administrator and type
>>
>> BCDEdit /set IncreaseUserVA <value>
>>
>> where <value> is between 2048 and 3072 (ie 2-3GB), then reboot.  Given
>> that you've set the --max-mem-size flag, or used the memory.limit
>> function in an R session to increase R's memory allocation, your R
>> processes should now be allowed to access up to 3GB of virtual memory.
>
> See the rw-FAQ: it is 3GB of address space and 2.5GB of memory by default.

indeed -- I confused 'virtual memory' with 'address space' -- thank you

>
>> I am not a Windows expert, so if anyone knows of disadvantages of
>> using this method *please* post a response.
>
> This is not advised in general by those who are experts (why do you think
> that it is not the default in Windows?).

But my suggestion is essentially the same as that given in the links
from 2.9, isn't it?  The difference being that it is for modern
versions of Windows.

>  As the URL above says
>
>  'For applications that are memory-intensive, such as database
>  management systems (DBMS), the use of a larger virtual address space
>  can provide considerable performance and scalability benefits.
>  However, the file cache, paged pool, and nonpaged pool are smaller,
>  which can adversely affect applications with heavy networking or
>  I/O.  Therefore, you might want to test your application under load,
>  and examine the performance counters to determine whether your
>  application benefits from the larger address space.'
>
> Remember that you are setting this for all uses and users of your machine,
> not just the R process (so the last sentence is somewhat misleading).
>
> The best advice seems to be that if you have 4GB or more of RAM is to run a
> 64-bit version of Windows, which gives a 32-bit process a 4GB user address
> space

but I assume for many it is not an option (due to budget, organization
policy, application compatiblity...)

>.  If you have a lot more memory, consider a 64-bit build of R (and an
> OS that handles memory more efficiently: one of my background tasks right
> now is to investigate why R under Windows is running a particular
> memory-intensive multiprocessor task so much more slowly than Linux on the
> same hardware).

and your skills and efforts are very much appreciated.

regards,

Kingsford Jones