[BioC] Re: Try to run a lots of chips: 32 bits vs 64 bits

[Nicholas Lewin-Koh]

Hi,
I have nothing to add on the 32-64bit issue, However for your memory
problem to add on to what i think Laurent was suggesting is that you
have to break up your problem into smaller parts. The main problem will
be quantile normalization. The other methods work pairwise, so that you
could script it in a sequential fashion. maybe you could do the quantile
normalization on blocks of oligos across chips?

What I do for jobs that hit memory boundaries is 
1) work out the procedure on a subset of the data.
2) write a script to do the processing, delete as you go and figure out
where you can write output to be loaded back later This will slow things
down because of writing to disk.

If you can break up the processing you should be able to get the
analysis done.

Another alternative might be to try using Tony Rossini's lib PVM
interface if you have access to multiple processors on a network. Though
setting up libpvm for that is a bitch.

Nicholas