[Rd] maximum matrix size
Therneau, Terry M., Ph.D.
therne@u @ending from m@yo@edu
Wed Oct 3 14:52:22 CEST 2018
That is indeed helpful; reading the sections around it largely answered my questions.
Rinternals.h has the definitions
#define allocMatrix Rf_allocMatrix
SEXP Rf_allocMatrix(SEXPTYPE, int, int);
#define allocVector Rf_allocVector
SEXP Rf_allocVector(SEXPTYPE, R_xlen_t);
Which answers the further question of what to expect inside C routines invoked by Call.
It looks like the internal C routines for coxph work on large matrices by pure serendipity
(nrow and ncol each less than 2^31 but with the product > 2^31), but residuals.coxph
fails with an allocation error on the same data. A slight change and it could just as
easily have led to a hard crash. Sigh... I'll need to do a complete code review.
I've been converting .C routines to .Call as convenient, this will force conversion of
many of the rest as a side effect (20 done, 23 to go). As a statsitician my overall
response is "haven't they ever heard of sampling"? But as I said earlier, it isn't just
On 10/02/2018 12:22 PM, Peter Langfelder wrote:
> Does this help a little?
> One thing I seem to remember but cannot find a reference for is that
> long vectors can only be passed to .Call calls, not C/Fortran. I
> remember rewriting .C() in my WGCNA package to .Call for this very
> reason but perhaps the restriction has been removed.
> On Tue, Oct 2, 2018 at 9:43 AM Therneau, Terry M., Ph.D. via R-devel
> <r-devel using r-project.org> wrote:
>> I am now getting the occasional complaint about survival routines that are not able to
>> handle big data. I looked in the manuals to try and update my understanding of max
>> vector size, max matrix, max data set, etc; but it is either not there or I missed it (the
>> latter more likely). Is it still .Machine$integer.max for everything? Will that
>> change? Found where?
>> I am going to need to go through the survival package and put specific checks in front
>> some or all of my .Call() statements, in order to give a sensible message whenever a
>> bounday is struck. A well meaning person just posted a suggested "bug fix" to the github
>> source of one routine where my .C call allocates a scratch vector, suggesting "resid =
>> double( as.double(n) *nvar)" to prevent a "NA produced by integer overflow" message, in
>> the code below. A fix is obvously not quite that easy :-)
>> resid <- .C(Ccoxscore, as.integer(n),
>> resid= double(n*nvar),
>> Terry T.
>> [[alternative HTML version deleted]]
>> R-devel using r-project.org mailing list
[[alternative HTML version deleted]]
More information about the R-devel