How to calculate agreement using attribute gauge when there are two sets of raters

KN93

We conducted a study with 20 cases of a rare disease and had experts rate whether they thought the disease was or was not present (Y/N). Because the cases were long, we had one set of raters (6 individuals) rate the first 10 cases and one set of raters (7 individuals) rate the last 10 cases. We want to evaluate the overall diagnostic agreement between all reviewers across all cases rather than analyzing separately for the first 10 and last 10 cases, since we just want to know the extent of agreement between experts in making the diagnosis. Is there a way to run an attribute gauge report to calculate agreement % and kappa across all cases even though there were two different sets of reviewers, i.e. for the cases a rater didn't review, they have "missing data". Thanks

statman

First, welcome to the community.  There are a couple of issues with the information you present:

1. You don't have an estimate of the within rater variation, intrarater reliability (e.g., what if you were to give the same case to the same rater without their knowledge and had them rate the case again.  Would they rate it the same?)

2 The variation between raters, interrater reliability. Since the cases are different for the different raters, you won't be able to discriminate between whether their agreement (or lack there of) was due to rater or case.  The variation due to rater and case are confounded.

 

You might find this an interesting read:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900052/