[R] Performance & capacity characteristics of R?

[Prof Brian D Ripley]

On Tue, 3 Aug 1999, Thomas Vogels wrote:

> 
> "Brian" == Prof Brian D Ripley <ripley at stats.ox.ac.uk> writes:
> 
> Brian> Can you tell us what statistical procedures need 1 million to 100s of
> Brian> millions or rows (observations)?  Some of us have doubted that there are
> Brian> even datasets of 100,000 examples that are homogeneous and for which a
> Brian> small subsample would not give all the statistical information. (If they
> Brian> are not homogeneous, one could/should analyse homogeneous subsets and do a
> Brian> meta-analysis.)
> 
> What if your problem is to find the outliers in a dataset?  It would
> be nice to examine the (homogeneous part of) the dataset and then
> search for the data entries "that don't quite fit in" without having
> to leave R and going to Perl or other home-grown software.

But you can do that in smaller batches: you don't need all the data at
once. A small subset will train a test for typicality, and then
the rest of the cases can be examined say 10,000 at a time.

(The big datasets I have at present are fMRI brain images, and the task is
to find the voxels that do not fit in. There are ca 40,000 voxels, and
a time series of a few hundred points on each. I process the voxels one
slice (ca 4,000) at a time.)

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._