[R] Saving multiple rda-files as one rda-file

David Winsemius dwinsemius at comcast.net
Thu Jul 25 18:47:35 CEST 2013

On Jul 22, 2013, at 4:18 AM, Dark wrote:

> Hi all,
> For a project we have to process some very large CSV files (up to 40 gig)
> To reduce them in size and increase operating performance I wanted to store
> them as RData files.
> Since it was to big I decided to split the csv and saving those parts as
> separate .RDA files.
> So far so good. Now I want to bind them all together to save as one RDA file
> again and this is supprisingly difficult.
> First I load my rda files into my environment:
> load(paste(rdaoutputdir, "file1.rda", sep=""))
> load(paste(rdaoutputdir, "file2.rda", sep=""))
> load(paste(rdaoutputdir, "file3.rda", sep=""))
> etc
> Then I try to combine them into one object.
> Using rbind like this gives memory allocation problems ('Error: cannot
> allocate vector of size')
> objectToSave <- rbind(object1, object2, object3)
> using pre-allocation gives me a factor level error. I used this code:
> 	nextrow <- nrow(object1)+1
> 	object1[nextrow:(nextrow+nrow(object2)-1),] <- object2
> 	# we need to assure unique row names
>        row.names(object1) = 1:nrow(object1)
> 	rm(object2)
>        gc()
> 15! warning messages:
> 1: In `[<-.factor`(`*tmp*`, iseq, value = structure(c(1L,  ... :
>  invalid factor level, NA generated
> 2: In `[<-.factor`(`*tmp*`, iseq, value = structure(c(1L,  ... :
>  invalid factor level, NA generated

The warning messages suggests that the factor levels in object1, object2, object3 in corresponding columns are not the same.

> What can I do?

You can identify which columns are factors and make the corresponding columns have levels that span the values.


Depending on the contents of that factor you could convert to character before the rbind operation. If the levels are not particularly long (in character length), that procedure might not expand the memory footprint very much.

> Regards Derk

David Winsemius
Alameda, CA, USA

More information about the R-help mailing list