Kappa ScoreGood morning,
Despite my research I couldn't find an answer to my problem so here it is:
I need to calculate a kappa score on a contingency table. I have the same categories for the 2 values but JMP 15 does not want to calculate the Kappa for me. However, the Kappa formula has no restriction given that it is the ratio of the difference in probability between actual and expected to the inverse of the expected probability.
Could you help me please?
Thanks in advance
Quality and Process EngineeringRe: Kappa ScoreI would not really interpret kappa in this case. The issue is that value 1 is ALWAYS 3. There is no variability with that value 1. There are several reasons that could happen, but that is the issue. Statistics work with variability and there is no variability there.
Re: Kappa ScoreActually, JMP did calculate Kappa for you: it is 0.
To see this, let's walk through the calculation.
Pr(a) = Probability of agreement = Agreeing on value 1 + Agreeing on value 2 + Agreeing on value 3 = 0/30 + 0/30 + 29/30 = 0.967.
Pr(e) = Probability expected (you use the marginal totals here) = Expected value 1 + Expected value 2 + Expected value 3 = 0/30 * 1/30 + 0/30 * 0/30 + 29/30 * 30/30 = 0.967.
kappa = 0 / (1-.967) = 0.
Re: Kappa ScoreGood morning,
Thank you for that answer. Yes indeed if you look better the Kappa is indeed 0.
But then I find it strange. Does this mean that my 97% agreement is just due to chance?
At first glance I would say that having the same answer 29 times out of 30 is rather superior to chance, right?
So is there a better concordance test to do for this data?
Thank you again for your help.
Marie
Re: Kappa ScoreOne additional thought: You could say that Value 2 will give a result of 3 29/30 times. Just ignore Value 1 since it is always a 3.
Re: Kappa ScoreThanks a lot Dan,
Usually scores are dispatched from 0 to 3 it is why I decided to applied Kappa, but in this case I guess I will do as you said and just talk about overall concordance as Kappa is not applicable here.
Best
