[R] statistical significance of accuracy increase in classification
stefan.evert at uos.de
Thu Feb 26 15:18:32 CET 2009
On 26 Feb 2009, at 14:14, Max Kuhn wrote:
>> Do you know about any good reference that discusses kappa for
>> classification and maybe CI for kappa???
You might also want to take a look at this survey article on kappa and
Artstein, Ron and Poesio, Massimo (2008). Survey article: Inter-coder
agreement for computational linguistics. Computational Linguistics,
which you can download from
Alternatives to the standard Fleiss-Cohen asymptotic confidence
intervals in the elementary 2x2 case are discussed in
Lee, J.J., Tu, Z. N.:"A Better Confidence for Kappa on Measuring
Agreement Between Two Raters with Binary Outcomes" Journal of
Computational and Graphical Statistics, 3:301-321, 1994.
which is available from JSTOR:
An S implementation of their approximations can be downloaded here:
I've started to evaluate the accuracy of these approximations with
simulation experiments some time ago, but haven't found the time to
follow up on it.
Hope this helps,
More information about the R-help