[R] Any chance R will ever get beyond the 2^31-1 vector size limit?

[Martin Morgan]

On 04/09/2010 05:36 PM, Duncan Murdoch wrote:
> On 09/04/2010 7:38 PM, Matthew Keller wrote:
>> Hi all,
>>
>> My institute will hopefully be working on cutting-edge genetic
>> sequencing data by the Fall of 2010. The datasets will be 10's of GB
>> large and growing. I'd like to use R to do primary analyses. This is
>> OK, because we can just throw $ at the problem and get lots of RAM
>> running on 64 bit R. However, we are still running up against the fact
>> that vectors in R cannot contain more than 2^31-1. I know there are

Duncan provided an answer (maybe not too encouraging!) to your question;
I'll postulate that the two large sequence-style data sets are SNPs and
'next generation' sequencing, and that for these the picture is quite
positive.

SNPs seem more likely to (or already have) hit the 2^31-1 limit, where
one wants to represent say 2 million SNPs assayed in 10,000's of
individuals. But here it's very natural to process the SNPs either
independently or per chromosome (linkage group / ...); Clayton's
snpMatrix (on Bioconductor) package and the ncdf (ncdf4, I think, is
recently released) package make for excellent handling of this data;
throw in a bit of multicore or Rmpi and you'll be very satisfied.

Next gen sequencing generates reads (character strings), but only say
20M reads / lane of an Illumina flow cell, so even a full flow cell (8
lanes) is about an order of magnitude below the 2^31-1 reads that would
overflow a character vector. And the Biostrings package in Bioconductor
has DNAStringSet, which has already (in the development version; to be
released shortly after the next R release) addressed the 2^31-1 limit
(and has diverse sequence manipulation methods available). I'd also
think that in this case too any memory limitations are likely amenable
to sequential processing (e.g., by lane of a flow cell). Further, much
of the 'interesting' analysis from R's perspective will be when the data
are reduced (e.g., 'coverage' vectors, or count data over a relatively
small (10,000's) number of regions).

Which is not to say that truly large problems (1000's of fully sequenced
human genomes) aren't just around the corner. But here the likely
solution is a clever disk-based data base-like representation, probably
one that transcends the R community (BAM files and bigWig, for which see
Bioconductor Rsamtools and rtracklayer in the 'devel' branch, represent
perhaps a first pass at this kind of approach).

Certainly limited vector lengths add challenges to managing this
information. But I don't think these challenges are difficult to
overcome, nor do I think that they are likely to outweigh the benefits
that R offers in terms of accessible advanced statistical analysis
coupled with flexible and integrated work flows.

Martin

>> "ways around" this issue, and trust me, I think I've tried them all
>> (e.g., bringing in portions of the data at a time; using large-dataset
>> packages in R; using SQL databases, etc). But all these 'solutions'
>> are, at the end of the day, much much more cumbersome,
>> programming-wise, than just doing things in native R. Maybe that's
>> just the cost of doing what I'm doing. But my questions, which  may
>> well be naive (I'm not a computer programmer), are:
>>
>> 1) Is there an *inherent* limit to vectors being < 2^31-1 long? I.e.,
>> in an alternative history of R's development, would it have been
>> feasible for R to not have had this limitation?
> 
> The problem is that we use "int" as a vector index.  On most platforms,
> that's a signed 32 bit integer, with max value 2^31-1.
> 
> 
>>
>> 2) Is there any possibility that this limit will be overcome in future
>> revisions of R?
> 
> 
> Of course, R is open source.  You could rewrite all of the internal code
> tomorrow to use 64 bit indexing.
> 
> Will someone else do it for you?  Even that is possible.  One problem
> are that this will make all of your data incompatible with older
> versions of R.  And back to the original question:  are you willing to
> pay for the development?  Then go ahead, you can have it tomorrow (or
> later, if your budget is limited).  Are you waiting for someone else to
> do it for free?  Then you need to wait for someone who knows how to do
> it to want to do it.
> 
> 
>> I'm very very grateful to the people who have spent important parts of
>> their professional lives developing R. I don't think anyone back in,
>> say, 1995, could have foreseen that datasets would be >>2^32-1 in
>> size. For better or worse, however, in many fields of science, that is
>> routinely the case today. *If* it's possible to get around this limit,
>> then I'd like to know whether the R Development Team takes seriously
>> the needs of large data users, or if they feel that (perhaps not
>> mutually exclusively) developing such capacity is best left up to ad
>> hoc R packages and alternative analysis programs.
> 
> There are many ways around the limit today.  Put your data in a
> dataframe with many columns each of length 2^31-1 or less.  Put your
> data in a database, and process it a block at a time.  Etc.
> 
> Duncan Murdoch
> 
>>
>> Best,
>>
>> Matt
>>
>>
>>
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


-- 
Martin Morgan
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793