[R] logistic discrimination: which chance performance??
Bruno L. Giordano
bruno.giordano at music.mcgill.ca
Fri Aug 11 05:07:41 CEST 2006
If posting a possible solution to one's own problem is not part of the
netiquette of this list please correct me.
Following Titus et al. (1984) one might use Cohen's kappa to have a
chance-corrected measure of agreement between the original and reproduced
Kappa() in library vcd
kappa2() in library irr
ckappa() in library psy
cohen.kappa() in library concord......
Kimberly Titus; James A. Mosher; Byron K. Williams (1984), Chance-corrected
Classification for Use in Discriminant Analysis: Ecological Applications,
American Midland Naturalist, 111(1),1-7.
----- Original Message -----
From: "Bruno L. Giordano" <bruno.giordano at music.mcgill.ca>
To: <r-help at stat.math.ethz.ch>
Sent: Thursday, August 10, 2006 6:18 PM
Subject: [R] logistic discrimination: which chance performance??
> I am using logistic discriminant analysis to check whether a known
> classification Yobs can be predicted by few continuous variables X.
> What I do is to predict class probabilities with multinom() in nnet(),
> obtaining a predicted classification Ypred and then compute the percentage
> P(obs) of objects classified the same in Yobs and Ypred.
> My problem now is to figure out whether P(obs) is significantly higher
> I opted for a crude permutation approach: compute P(perm) over 10000
> permutations of Yobs (i.e., refit the multinom() model 10000 times
> permuting Yobs) and consider P(obs) as significantly higher than chance if
> higher than the 95th percentile of the P(perm) distribution.
> Now, the problem is that the mode of P(perm) is always really close to
> P(obs), e.g., if P(obs)=1 (perfect discrimination) also the most likely
> P(perm) value is 1!!!
> I figured out that this is due to the fact that, with my data, randomly
> permuted classifications are highly likely to strongly agree with the
> observed classification Yobs, but, probably since my machine learning
> background is almost 0, I am kind of lost about how to proceed at this
> I would greatly appreciate a comment on this.
> Bruno L. Giordano, Ph.D.
> Schulich School of Music, McGill University
> 555 Sherbrooke Street West
> Montréal, QC H3A 1E3
> R-help at stat.math.ethz.ch mailing list
> PLEASE do read the posting guide
> and provide commented, minimal, self-contained, reproducible code.
More information about the R-help