[R] Characterizing overall agreement and the p values associated with the agreement. Kappa statistic

[John Sorkin]

R 2.10.0
Windows XP

Please forgive a question that is not strictly speaking R centric

I have 24 raters, each of who characterizes 100 pictures into one of three categories, A, B, or C. Each rater rates each picture twice in random order with sufficient time between ratings so that the rater is unlikely to remember her, or her prior rating.

data1<-matrix(nrow=24,ncol=2)
data1[,1]<-rater1Firstvalue
data1[,2]<-rater1Secondtvalue

data2<-matrix(nrow=24,ncol=2)
data2[,1]<-rater2Firstvalue
data2[,2]<-rater2Secondtvalue
.
.
.
data24<-matrix(nrow=24,ncol=2)
data24[,1]<-rater24Firstvalue
data24[,2]<-rater24Secondtvalue

I would like to determine the within-observer agreement. For each of the 24 raters, I am using kappam.fleiss to assess agreement:

Kappa[1,"Kappa"]<-kappam.fleiss(data1,   exact = TRUE, detail = TRUE)$value
Kappa[2,"Kappa"]<-kappam.fleiss(data2,   exact = TRUE, detail = TRUE)$value
.
.
.
Kappa[24,"Kappa"]<-kappam.fleiss(data24,   exact = TRUE, detail = TRUE)$value

Similarly I isolate each of the 24 p values:
Kappa[1,"Kappa"]<-kappam.fleiss(data1,   exact = TRUE, detail = TRUE)$p.value
Kappa[2,"Kappa"]<-kappam.fleiss(data2,   exact = TRUE, detail = TRUE)$p.value
.
.
.
Kappa[24,"Kappa"]<-kappam.fleiss(data24,   exact = TRUE, detail = TRUE)$p.value

Can characterize the overall agreement and p value as the mean of the 24 kappa values and 24 p values, i.e.

mean(kappa)

Thanks,
John 


John David Sorkin M.D., Ph.D.
Chief, Biostatistics and Informatics
University of Maryland School of Medicine Division of Gerontology
Baltimore VA Medical Center
10 North Greene Street
GRECC (BT/18/GR)
Baltimore, MD 21201-1524
(Phone) 410-605-7119
(Fax) 410-605-7913 (Please call phone number above prior to faxing)

Confidentiality Statement:
This email message, including any attachments, is for th...{{dropped:6}}