[BioC] RdbiPgSQL: slowness in dbWriteTable

Jeff Gentry jgentry at jimmy.harvard.edu
Tue Sep 4 20:56:28 CEST 2007


Hi there ...

In the past I've had some difficulty inserting some larger tables into
postgres using dbWriteTable() from RdbiPgSQL, in terms of speed
performance but now I've run into some data.frames which are just
too large (although they don't seem to be all that large in practical
terms).

As an example, I was attempting to insert the assayData from two
SnpCallSets into a DB over the weekend, 6 tables each of ~350 columns and
250K rows.  When I came back this morning (~3.5 days), it hadn't even
completed one of these tables, with both CPU and RAM maxxed (on a machine
w/ 16GB of RAM and a pretty decent CPU).

Anyone have ideas on a faster method to get these tables inserted?  One
possibility that I've found in the past to be quicker is to manually loop
INSERT statements, but the problem there is detection of data types (which
I believe is the real slowdown from the RdbiPgSQL package in the first
place) and proper quoting/non-quoting.



More information about the Bioconductor mailing list