[BioC] Maximum number of CEL files for ReadAffy() in Affy package.

[James MacDonald]

Hailong Cui wrote:
> Dear all,
> 
> First, I apologize for the mass email. I've been reading manuals, googling,
> searching the archive of the mailing list, but still cannot find an exact
> answer to my problem.
> 
> (1) Question: Can a large number of CEL files cause an overflow for the
> function ReadAffy() in the affy packages? Is there any way to fix this?
> Other options seem to be other software RMAExpress and dChip in WindowsXP.
> Any suggestions?

Well, the usual prescription is to get more RAM. However, it appears you 
already have more RAM.

> 
> (2) Background: What I am trying to do is to read in all the CEL files in
> the directory to create an AffyBatch object, so that I can use functions in
> the affy package. To be more specific, I want to do RMA, dChip normalization
> and get MAplots. In my workstation (48 64-bit CPUs, 500Gb memory),
> ReadAffy() worked fine for 241 CEL files, but when I moved on to 2,035 CEL
> files, it failed and kept showing the error message below. The number of
> rows for the CEL file is roughly 50k. On the bright side, I tried justRMA()
> and got the expression values in the text format.

Dude. Really? 500Gb RAM? Yowza.

If you want to be able to have an AffyBatch-type object to play around 
with, you might try the oligo package. This package writes the data to 
the hard drive and uses the BufferedMatrix package to speed up the I/O. 
And it seems you might have already tried that, as I see you have that 
package installed.

Best,

Jim


> 
>> R
>> library(affy)
>> Data <- ReadAffy()
> Error in read.affybatch(filenames = l$filenames, phenoData
> = l$phenoData,  :
>   allocMatrix: too many elements specified
> 
> 
> FYI, below is the session information on my workstation.
> 
>> sessionInfo()
> R version 2.7.1 (2008-06-23)
> ia64-unknown-linux-gnu
> 
> locale:
> LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=C;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C
> 
> attached base packages:
> [1] tools     stats     graphics  grDevices utils     datasets  methods
> [8] base
> 
> other attached packages:
>  [1] geneplotter_1.18.0          annotate_1.18.0
>  [3] xtable_1.5-2                AnnotationDbi_1.2.2
>  [5] RSQLite_0.6-9               DBI_0.2-4
>  [7] lattice_0.17-8              BufferedMatrixMethods_1.4.0
>  [9] BufferedMatrix_1.4.0        affy_1.18.2
> [11] preprocessCore_1.2.0        affyio_1.8.0
> [13] Biobase_2.0.1
> 
> loaded via a namespace (and not attached):
> [1] grid_2.7.1         KernSmooth_2.22-22 RColorBrewer_1.0-2
> 
> 
> 
> 
> Thank you so much for reading this and I would appreciate your reply.
> 
> Hailong
> 
> 

-- 
James W. MacDonald, MS
Biostatistician
UMCCC cDNA and Affymetrix Core
University of Michigan
1500 E Medical Center Drive
7410 CCGC
Ann Arbor MI 48109
734-647-5623