[Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

Tue Dec 20 17:56:10 CET 2016

On 20 December 2016 at 17:40, Martin Maechler wrote:
| >>>>> Steve Bronder <sbronder at stevebronder.com>
| >>>>>     on Tue, 20 Dec 2016 01:34:31 -0500 writes:
| 
|     > Thanks Henrik this is very helpful! I will try this out on our tests and
|     > see if gcDLLs() has a positive effect.
| 
|     > mlr currently has tests broken down by learner type such as classification,
|     > regression, forecasting, clustering, etc.. There are 83 classifiers alone
|     > so even when loading and unloading across learner types we can still hit
|     > the MAX_NUM_DLLS error, meaning we'll have to break them down further (or
|     > maybe we can be clever with gcDLLs()?). I'm CC'ing Lars Kotthoff and Bernd
|     > Bischl to make sure I am representing the issue well.
| 
| This came up *here* in May 2015
| and then May 2016 ... did you not find it when googling.
| 
| Hint:  Use  
|        site:stat.ethz.ch MAX_NUM_DLLS
| as search string in Google, so it will basically only search the
| R mailing list archives
| 
| Here's the start of that thread :
| 
|   https://stat.ethz.ch/pipermail/r-devel/2016-May/072637.html
| 
| There was not a clear conclusion back then, notably as
| Prof Brian Ripley noted that 100 had already been an increase
| and that a large number of loaded DLLs decreases look up speed.
| 
| OTOH (I think others have noted that) a large number of DLLs
| only penalizes those who *do* load many, and we should probably
| increase it.
| 
| Your use case of "hyper packages" which load many others
| simultaneously is somewhat convincing to me... in so far as the
| general feeling is that memory should be cheap and limits should
| not be low.
| 
| (In spite of Brian Ripleys good reasons against it, I'd still
|  aim for a *dynamic*, i.e. automatically increased list here).

Yes.  Start with 10 or 20, add 10 as needed.  Still fast in the 'small N'
case and no longer a road block for the 'big N' case required by mlr et al.

As a C++ programmer, I am now going to hug my std::vector and quietly retreat.

Dirk

| Martin Maechler
| 
|     > Regards,
| 
|     > Steve Bronder
|     > Website: stevebronder.com
|     > Phone: 412-719-1282
|     > Email: sbronder at stevebronder.com
| 
| 
|     > On Tue, Dec 20, 2016 at 1:04 AM, Henrik Bengtsson <
|     > henrik.bengtsson at gmail.com> wrote:
| 
|     >> On reason for hitting the MAX_NUM_DLLS (= 100) limit is because some
|     >> packages don't unload their DLLs when they being unloaded themselves.
|     >> In other words, there may be left-over DLLs just sitting there doing
|     >> nothing but occupying space.  You can remove these, using:
|     >> 
|     >> R.utils::gcDLLs()
|     >> 
|     >> Maybe that will help you get through your tests (as long as you're
|     >> unloading packages).  gcDLLs() will look at base::getLoadedDLLs() and
|     >> its content and compare to loadedNamespaces() and unregister any
|     >> "stray" DLLs that remain after corresponding packages have been
|     >> unloaded.
|     >> 
|     >> I think it would be useful if R CMD check would also check that DLLs
|     >> are unregistered when a package is unloaded
|     >> (https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29), but of
|     >> course, someone needs to write the code / a patch for this to happen.
|     >> 
|     >> /Henrik
|     >> 
|     >> On Mon, Dec 19, 2016 at 6:01 PM, Steve Bronder
|     >> <sbronder at stevebronder.com> wrote:
|     >> > This is a request to increase MAX_NUM_DLLS in Rdynload.c in from 100 to
|     >> 500.
|     >> >
|     >> > On line 131 of Rdynload.c, changing
|     >> >
|     >> > #define MAX_NUM_DLLS 100
|     >> >
|     >> >  to
|     >> >
|     >> > #define MAX_NUM_DLLS 500
|     >> >
|     >> >
|     >> > In development of the mlr package, there have been several episodes in
|     >> the
|     >> > past where we have had to break up unit tests because of the "maximum
|     >> > number of DLLs reached" error. This error has been an inconvenience that
|     >> is
|     >> > going to keep happening as the package continues to grow. Is there more
|     >> > than meets the eye with this error or would everything be okay if the
|     >> above
|     >> > line changes? Would that have a larger effect in other parts of R?
|     >> >
|     >> > As R grows, we are likely to see more 'meta-packages' such as the
|     >> > Hadley-verse, caret, mlr, etc. need an increasing amount of DLLs loaded
|     >> at
|     >> > any point in time to conduct effective unit tests. If  MAX_NUM_DLLS is
|     >> set
|     >> > to 100 for a very particular reason than I apologize, but if it is
|     >> possible
|     >> > to increase MAX_NUM_DLLS it would at least make the testing at mlr much
|     >> > easier.
|     >> >
|     >> > I understand you are all very busy and thank you for your time.
|     >> >
|     >> >
|     >> > Regards,
|     >> >
|     >> > Steve Bronder
|     >> > Website: stevebronder.com
|     >> > Phone: 412-719-1282
|     >> > Email: sbronder at stevebronder.com
|     >> >
|     >> >         [[alternative HTML version deleted]]
|     >> >
|     >> > ______________________________________________
|     >> > R-devel at r-project.org mailing list
|     >> > https://stat.ethz.ch/mailman/listinfo/r-devel
|     >> 
| 
|     > [[alternative HTML version deleted]]
| 
|     > ______________________________________________
|     > R-devel at r-project.org mailing list
|     > https://stat.ethz.ch/mailman/listinfo/r-devel
| 
| ______________________________________________
| R-devel at r-project.org mailing list
| https://stat.ethz.ch/mailman/listinfo/r-devel

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org