[R] Making tapply code more efficient
Doran, Harold
HDoran at air.org
Mon Mar 9 15:24:32 CET 2009
Thierry and Jim:
Thank you both for your reply. I remain a bit baffled over something.
Here is the sample data generated by jim and code by Thierry, which
works exactly as expected.
x <- cbind(sample(326397, 800967, TRUE), sample(20, 800967, TRUE))
x <- data.frame(x)
names(x)[1:2] <- c('student_unique_id', 'teacher_unique_id')
tab <- table(x$student_unique_id, x$teacher_unique_id)
result <- data.frame(Student = rownames(tab), Freq = rowSums(tab), tch =
rowSums(tab > 0) == 1)
Now, here is what happens when I run this on my data (called dats3)
> tab <- table(dats3$student_unique_id, dats3$teacher_unique_id)
Error: cannot allocate vector of size 942.8 Mb
So, let's take a look at a couple of things:
> object.size(dats3) < object.size(x)
[1] TRUE
> str(x)
'data.frame': 800967 obs. of 2 variables:
$ student_unique_id: int 121914 89142 127790 61350 54684 28018 313428
27595 316285 173571 ...
$ teacher_unique_id: int 17 1 19 20 3 18 15 1 14 15 ...
> str(dats3)
'data.frame': 56204 obs. of 2 variables:
$ student_unique_id: int 20504 26172 20504 3609 4313 5058 5363 5669
6429 6560 ...
$ teacher_unique_id: int 35078 41029 35078 41437 41476 41456 41486
35415 41508 35413 ...
The sample data are smaller in size than my actual data and the
structure is exactly the same. Do you see any other reason why the
memory issue would arise here?
Harold
> -----Original Message-----
> From: ONKELINX, Thierry [mailto:Thierry.ONKELINX at inbo.be]
> Sent: Friday, February 27, 2009 10:24 AM
> To: Doran, Harold; r-help at r-project.org
> Subject: RE: [R] Making tapply code more efficient
>
> Hi Harold,
>
> What about this? You one have to make the crosstabulation once.
>
> > qq <- data.frame(student = factor(c(1,1,2,2,2)), teacher =
> factor(c(10,10,20,20,25)))
> > tab <- table(qq$student, qq$teacher)
> > data.frame(Student = rownames(tab), Freq = rowSums(tab), tch =
> rowSums(tab > 0) == 1)
> Student Freq tch
> 1 1 2 TRUE
> 2 2 3 FALSE
>
> HTH,
>
> Thierry
>
>
> --------------------------------------------------------------
> ----------
> ----
> ir. Thierry Onkelinx
> Instituut voor natuur- en bosonderzoek / Research Institute
> for Nature and Forest Cel biometrie, methodologie en
> kwaliteitszorg / Section biometrics, methodology and quality
> assurance Gaverstraat 4 9500 Geraardsbergen Belgium tel. + 32
> 54/436 185 Thierry.Onkelinx at inbo.be www.inbo.be
>
> To call in the statistician after the experiment is done may
> be no more than asking him to perform a post-mortem
> examination: he may be able to say what the experiment died of.
> ~ Sir Ronald Aylmer Fisher
>
> The plural of anecdote is not data.
> ~ Roger Brinner
>
> The combination of some data and an aching desire for an
> answer does not ensure that a reasonable answer can be
> extracted from a given body of data.
> ~ John Tukey
>
> -----Oorspronkelijk bericht-----
> Van: r-help-bounces at r-project.org
> [mailto:r-help-bounces at r-project.org]
> Namens Doran, Harold
> Verzonden: vrijdag 27 februari 2009 15:47
> Aan: r-help at r-project.org
> Onderwerp: [R] Making tapply code more efficient
>
> Previously, I posed the question pasted down below to the
> list and received some very helpful responses. While the code
> suggestions provided in response indeed work, they seem to
> only work with *very* small data sets and so I wanted to
> follow up and see if anyone had ideas for better efficiency.
> I was quite embarrased on this as our SAS programmers cranked
> out programs that did this in the blink of an eye (with a few
> variables), but R was spinning for days on my Ubuntu machine
> and ultimately I saw a message that R was "killed".
>
> The data I am working with has 800967 total rows and 31 total columns.
> The ID variable I use as the index variable in tapply() has
> 326397 unique cases.
>
> > length(unique(qq$student_unique_id))
> [1] 326397
>
> To give a sense of what my data look like and the actual
> problem, consider the following:
>
> qq <- data.frame(student_unique_id = factor(c(1,1,2,2,2)),
> teacher_unique_id = factor(c(10,10,20,20,25)))
>
> This is a student achievement database where students occupy
> multiple rows in the data and the variable teacher_unique_id
> denotes the class the student was in. What I am doing is
> looking to see if the teacher is the same for each instance
> of the unique student ID. So, if I implement the following:
>
> same <- function(x) length( unique(x) ) == 1 results <- data.frame(
> freq = tapply(qq$student_unique_id,
> qq$student_unique_id, length),
> tch = tapply(qq$teacher_unique_id, qq$student_unique_id, same)
> )
>
> I get the following results. I can see that student 1 appears
> in the data twice and the teacher is always the same.
> However, student 2 appears three times and the teacher is not
> always the same.
>
> > results
> freq tch
> 1 2 TRUE
> 2 3 FALSE
>
> Now, implementing this same procedure to a large data set
> with the characteristics described above seems to be
> problematic in this implementation.
>
> Does anyone have reactions on how this could be more
> efficient such that it can run with large data as I described?
>
> Harold
>
> > sessionInfo()
> R version 2.8.1 (2008-12-22)
> x86_64-pc-linux-gnu
>
> locale:
> LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLA
> TE=en_US.U
> TF-8;LC_MONETARY=C;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-
> 8;LC_NAME=
> C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_ID
> ENTIFICATI
> ON=C
>
> attached base packages:
> [1] stats graphics grDevices utils datasets methods base
>
>
>
>
> ##### Original question posted on 1/13/09 Suppose I have a
> dataframe as follows:
>
> dat <- data.frame(id = c(1,1,2,2,2), var1 =
> c(10,10,20,20,25), var2 = c('foo', 'foo', 'foo', 'foobar', 'foo'))
>
> Now, if I were to subset by id, such as:
>
> > subset(dat, id==1)
> id var1 var2
> 1 1 10 foo
> 2 1 10 foo
>
> I can see that the elements in var1 are exactly the same and
> the elements in var2 are exactly the same. However,
>
> > subset(dat, id==2)
> id var1 var2
> 3 2 20 foo
> 4 2 20 foobar
> 5 2 25 foo
>
> Shows the elements are not the same for either variable in
> this instance. So, what I am looking to create is a data
> frame that would be like this
>
> id freq var1 var2
> 1 2 TRUE TRUE
> 2 3 FALSE FALSE
>
> Where freq is the number of times the ID is repeated in the
> dataframe. A TRUE appears in the cell if all elements in the
> column are the same for the ID and FALSE otherwise. It is
> insignificant which values differ for my problem.
>
> The way I am thinking about tackling this is to loop through
> the ID variable and compare the values in the various columns
> of the dataframe.
> The problem I am encountering is that I don't think all.equal
> or identical are the right functions in this case.
>
> So, say I was wanting to compare the elements of var1 for id
> ==1. I would have
>
> x <- c(10,10)
>
> Of course, the following works
>
> > all.equal(x[1], x[2])
> [1] TRUE
>
> As would a similar call to identical. However, what if I only
> have a vector of values (or if the column consists of names)
> that I want to assess for equality when I am trying to
> automate a process over thousands of cases? As in the example
> above, the vector may contain only two values or it may
> contain many more. The number of values in the vector differ by id.
>
> Any thoughts?
>
> Harold
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
> Dit bericht en eventuele bijlagen geven enkel de visie van de
> schrijver weer en binden het INBO onder geen enkel beding,
> zolang dit bericht niet bevestigd is door een geldig
> ondertekend document. The views expressed in this message
> and any annex are purely those of the writer and may not be
> regarded as stating an official position of INBO, as long as
> the message is not confirmed by a duly signed document.
>
More information about the R-help
mailing list