[R] "Best" way to merge 300+ .5MB dataframes?

[Prof Brian Ripley]

On 12/08/2014 07:07, David Winsemius wrote:
>
> On Aug 11, 2014, at 8:01 PM, John McKown wrote:
>
>> On Mon, Aug 11, 2014 at 9:43 PM, Thomas Adams <tea3rd at gmail.com> wrote:
>>> Grant,
>>>
>>> Assuming all your filenames are something like file1.txt,
>>> file2.txt,file3.txt... And using the Mac OSX terminal app (after you cd to
>>> the directory where your files are located...
>>>
>>> This will strip off the 1st lines, that is, your header lines:
>>>
>>> for file in *.txt;do
>>> sed -i '1d'${file};
>>> done
>>>
>>> Then, do this:
>>>
>>> cat *.txt > newfilename.txt
>>>
>>> Doing both should only take a few seconds, depending on your file sizes.
>>>
>>> Cheers!
>>> Tom
>>>
>>
>> Using sed hadn't occurred to me. I guess I'm just "awk-ward" <grin/>.
>> A slightly different way would be:
>>
>> for file in *.txt;do
>>   sed '1d' ${file}
>> done >newfilename.txt
>>
>> that way the original files are not modified.  But it strips out the
>> header on the 1st file as well. Not a big deal, but the read.table
>> will need to be changed to accommodate that. Also, it creates an
>> otherwise unnecessary intermediate file "newfilename.txt". To get the
>> 1st file's header, the script could:
>>
>> head -1 >newfilename.txt
>> for file in *.txt;do
>>    sed '1d' ${file}
>> done >>newfilename.txt
>>
>> I really like having multiple answers to a given problem. Especially
>> since I have a poorly implemented version of "awk" on one of my
>> systems. It is the vendor's "awk" and conforms exactly to the POSIX
>> definition with no additions. So I don't have the FNR built-in
>> variable. Your implementation would work well on that system. Well, if
>> there were a version of R for it. It is a branded UNIX system which
>> was designed to be totally __and only__ POSIX compliant, with few
>> (maybe no) extensions at all. IOW, it stinks. No, it can't be
>> replaced. It is the z/OS system from IBM which is EBCDIC based and
>> runs on the "big iron" mainframe, system z.
>>
>> --
>
> On the Mac the awk equivalent is gawk. Within R you would use `system()` possibly using paste0() to construct a string to send.

For historical reasons this is actually part of R's configuration: see 
the AWK entry in R_HOME/etc/Makeconf.  (There is an SED entry too: not 
all sed's in current OSes are POSIX-compliant.)

Using system2() rather than system() is recommended for new code.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Emeritus Professor of Applied Statistics, University of Oxford
1 South Parks Road, Oxford OX1 3TG, UK