[Rd] There was a problem by the use of snow.

[Prof Brian Ripley]

On Mon, 25 Jun 2007, Ei-ji Nakama wrote:

> problem of the very large memory require by the Sign extension.
>
> --- R-2.5.0.orig/src/main/serialize.c   2007-03-27 01:42:08.000000000 +0900
> +++ R-2.5.0/src/main/serialize.c        2007-06-25 00:48:58.000000000 +0900
> @@ -1866,7 +1866,7 @@
>
> static void resize_buffer(membuf_t mb, int needed)
> {
> -    int newsize = 2 * needed;
> +    size_t newsize = 2 * needed;
>     mb->buf = realloc(mb->buf, newsize);
>     if (mb->buf == NULL)
>        error(_("cannot allocate buffer"));

Yes, thanks, but the structure also needs to be changed as the next line 
is

     mb->size = newsize;

and so this would set mb->size to a negative value.

Could you please tell us where you encountered this?

As far as I can see, this code is only used via R_serialize for 
serializing to a raw vector, in which case 'size' can safely be 'int' and 
the re-allocation should be up to 2^31-1 bytes at most (and allocating 
twice what you are asked for seems undesirable).  But there is potential 
overflow at

     if (mb->count + length > mb->size)

and possibly elsewhere.


> The time-out of read and write was not set.
>
> 51:sendData.SOCKnode <- function(node, data) {
> 52:     timeout <- getClusterOption("timeout")
> 53:     old <- options(timeout = timeout);
> 54:     on.exit(options(old))
> 55:     serialize(data, node$con)
> 56: }
> 57:
> 58:recvData.SOCKnode <- function(node) {
> 59:     timeout <- getClusterOption("timeout")
> 60:     old <- options(timeout = timeout);
> 61:     on.exit(options(old))
> 62:     unserialize(node$con)
> 63: }

I don't think sock_read/sock_write is the right place to make that 
setting.  Ideally it would be set when the option is set, but as this is 
in a module that needs an extension to the interface.

Looking at the code, we read from a socket in blocks of 4096, but we write 
in a single block.  The latter is likely to be the problem, and I think 
some redesigning is necessary here.

Perhaps you and Luke Tierney can comment on exactly what the problem is 
and how best to work around it.

>
> --- R-2.5.0.orig/src/modules/internet/sockconn.c 2006-09-04 23:20:59.000000000 +0900
> +++ R-2.5.0/src/modules/internet/sockconn.c     2007-06-25
> 00:51:38.000000000 +0900
> @@ -155,14 +155,19 @@
> static size_t sock_read(void *ptr, size_t size, size_t nitems,
>                        Rconnection con)
> {
> +    int timeout = asInteger(GetOption(install("timeout"), R_BaseEnv));
> +
> +    R_SockTimeout(timeout);
>     return sock_read_helper(con, ptr, size * nitems)/size;
> }
>
> static size_t sock_write(const void *ptr, size_t size, size_t nitems,
>                         Rconnection con)
> {
> +    int timeout = asInteger(GetOption(install("timeout"), R_BaseEnv));
>     Rsockconn this = (Rsockconn)con->private;
>
> +    R_SockTimeout(timeout);
>     return R_SockWrite(this->fd, ptr, size * nitems)/size;
> }
>
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595