[Rd] RData File Specification?

Cook, Ian Ian.Cook at amd.com
Thu Aug 23 22:21:04 CEST 2007


I am developing a tool for converting a large data frame stored in an uncompressed binary (XDR) RData file to a delimited text file.  The data frame is too large to load() and extract rows from on a typical PC.  I'm looking to parse through the file and extract individual entries without loading the whole thing into memory.

In terms of some C source functions, instead of doing RestoreToEnv(R_Unserialize(connection)) which is essentially what load() does, I'm looking to get the documentation I would need to build a function "SaveToCSV()" so that I could do SaveToCSV(R_Unserialize(connection)).

Where can I get documentation on the RData file format?  Does a spec document exist?

See details below.


Ian Cook | Advanced Micro Devices, Inc. | ian.cook at amd.com


Additional details:

I've browsed through the relevant source code (saveload.c, serialize.c) for ideas.

Here's a demo of the problem I'm looking to solve:

# create a sample data frame
ds <- data.frame(row1=c(1,2,3),row2=c('a','b','c'))
# save into an uncompressed binary R dataset

# Then load() can be simulated like this:

# create and open a file connection
con <- file("ds.rdata",open="rb")
# read the first 5 characters
# unserialize the remainder and restore to the environment
ds <- unserialize(con,NULL)[["ds"]]

But this takes up too much memory if the data set is too big.  I can read in the file character-by-character, i.e. using readChar(), but it's obvious that the file format is not trivial.  readChar(con,10000) for this demo yields:


This would be parse-able if I had a file spec.  Thanks.

Ian Cook | Advanced Micro Devices, Inc. | ian.cook at amd.com

More information about the R-devel mailing list