[Rd] [R] Problem With Repeated Use Of Load/Save/Close Commands (PR#13841)

[murdoch at stats.uwo.ca]

On 17/07/2009 7:57 PM, Marilyn & Rich Short wrote:
> Hello,
> 
> I'm having a problem in R with repeated use of the "load", "save", and 
> "close" commands. I'm getting an error message that reads, "Too many 
> open files". I'm opening files and closing them (and unlinking them), 
> but when I go through that process 509 times, the program halts and I 
> get this error message: "cannot open the connection" with warning 
> messages: "Too many open files".  I've been working on this problem for 
> a couple of weeks and have gleaned a bit of info from different internet 
> threads, but no solutions yet.
> 
> I'm using Windows XP, SP3 and R 2.9.1.
> Here is my session info:
> 
> R version 2.9.1 (2009-06-26)
> i286-pc-mingw32
> 
> locale:
> LC_COLLATE=English_United States.1252;LC_CTYPE=English_United
> States. 1252;LC_MONETARY=English_United
> States. 1252;LC_NUMERIC=C;LC_TIME=English_United States.1252
> 
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
> 
> I'm using the vanilla load of R, with nothing added after booting up.
> The problem also occurs on my Vista machine as well.
> 
> The program below will induce the problem.
> ------------------------------------------------------------
>    junk       = 1
>    PathA    = tempdir()
>    conX     = paste(PathA,"junk",sep="\\")
>    outJunk  = file(conX, open="wb")
>    save(junk, file=outJunk)
>    close(outJunk)
>    for(i in 1:4000){
>        outMIA = file(conX, open="rb")
>        load(file=outMIA)
>        close(outMIA)
>        closeAllConnections()
>        unlink(conX)
>        rm(outMIA)
>        rm(junk)
>        cat(" i = ",i,sep=" ")
>        gc()
>        zzz = showConnections(all=FALSE)
>        cat(" zzz = ",zzz,"\n",sep=" ")
>    }
> --------------------------------------------------------------------
> There is some talk on the internet that some windows systems have a 
> limit of 512 files that can be open at one time. Even though I'm closing 
> my files each time, something is keeping track of how many times I've 
> opened and closed a file in a session. I've talked to Microsoft and run 
> a test program in Visual Studio C#, and, at the moment, it looks like 
> the problem does not lie in the Microsoft arena. The C# program 
> performed a similar task 10,000 times without a problem. I'm not totally 
> convinced, but the current evidence says to look elsewhere.
> 
> I've attached a script that will induce the problem. However, be warned 
> that, if you use it, you will have to get out of R after you run it. R 
> will no longer be able to access files such as help or sessionInfo(). 
> This can be solved by getting out of the R GUI and back in.
> 
> R E-mails from as far back as 2006 ask for help on the issue. There have 
> been several suggestions, but no confirmed solution that I can find. You 
> will see my attempt at these suggestions in the script [ rm(outMIA); 
> rm(junk); closeAllConnections(); and gc(); after close(outMIA);  and 
> unlink(conX);]. For me, this becomes important because it limits the 
> total number if iterations in nested do-loops. This is where I ran into 
> the problem. The program above will allow you to reproduce the problem.
> 
> Any suggestion would be greatly appreciated.

This looks like a bug to me, and so I've posted it to the bugs list, but 
your script doesn't make sense for normal use.  Before the loop you 
create a file named according to conX containing the junk variable, and 
then within the loop you load that file, *then delete it*, using 
unlink().  However, if you examine the value of unlink(), you'll see 
that it fails every time.

Moreover, the use of connections is unnecessary:  load() can work with a 
filename.  So you can skip the file() and close() calls.

If I simplify your script by removing the unnecessary stuff, then it 
works.  That is, this works:

     junk       = 1
     PathA    = tempdir()
     conX     = paste(PathA,"junk",sep="\\")
     save(junk, file=conX)
     for(i in 1:4000){
         load(file=conX)
         rm(junk)
         cat(" i = ",i,sep=" ")
         gc()
         zzz = showConnections(all=FALSE)
         cat(" zzz = ",zzz,"\n",sep=" ")
     }

So a suggestion for a workaround is simply to follow the pattern above, 
rather than that of your demonstration code.

I'll spend a bit more time and see if I can find what is causing the 
open file leak.

Duncan Murdoch