[R] Error because of large dimension
wdunlap at tibco.com
Sun Jan 24 23:45:02 CET 2016
> 28 PiB. Storing such a large matrix even on file is not possible.
The ads for Amazon Red Shift say it is possible. E.g.,
Amazon Redshift is a fast, fully managed, petabyte-scale data
warehouse that makes it simple and cost-effective to analyze
all your data using your existing business intelligence tools.
Start small for $0.25 per hour with no commitments and scale
to petabytes for $1,000 per terabyte per year, less than a tenth
the cost of traditional solutions. Customers typically see 3x
compression, reducing their costs to $333 per uncompressed
terabyte per year.
Cost may be an issue:
28 petabytes * 1024 petabytes/terabyte * $333 terabyte/year ~= $9.5
or $26 thousand/day.
On Sun, Jan 24, 2016 at 1:29 PM, Henrik Bengtsson <
henrik.bengtsson at gmail.com> wrote:
> FYI, the matrix you tried to allocate would hold
> (3195*1290*495*35*35*35*15) * 3 = 3.936248e+15 values. Each value
> would occupy 8 bytes of memory (for the double data type). In other
> words, in order to keep this data matrix in memory you would require a
> computer with at least 3.148998e+16 bytes of RAM, i.e. 29327331 GiB =
> 28640 TiB = 28 PiB. Storing such a large matrix even on file is not
> In other words, you need to figure out how to approach your original
> problem in a different way.
> On Sun, Jan 24, 2016 at 8:46 AM, li li <hannah.hlx at gmail.com> wrote:
> > Hi all,
> > I am doing some calculation with very large dimension. I need to
> create a
> > matrix
> > with three columns and a very large number of rows
> > (3195*1290*495*35*35*35*15=1.312083e+15) i
> > n order to allocate calculation result from a for loop.
> > R does not allow me to create such a matrix because of the large
> > (see below). Is there a way to go around this?
> > Thanks very much!!
> > Hanna
> >> matrix(0, 3195*1290*495*35*35*35*15, 3)
> > Error in matrix(0, 3195 * 1290 * 495 * 35 * 35 * 35 * 15, 3) :
> > invalid 'nrow' value (too large or NA)
> > In addition: Warning message:
> > In matrix(0, 3195 * 1290 * 495 * 35 * 35 * 35 * 15, 3) :
> > NAs introduced by coercion
> > [[alternative HTML version deleted]]
> > ______________________________________________
> > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> > and provide commented, minimal, self-contained, reproducible code.
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> PLEASE do read the posting guide
> and provide commented, minimal, self-contained, reproducible code.
[[alternative HTML version deleted]]
More information about the R-help