Interrater reliability (IRR) (also known as “kappa” ( $\kappa$)) statistics, are used to measure agreement between two raters or coders classifying items into mutually exclusive categories. $\kappa$ statistics are widely used in fields such as psychology and medicine to evaluate the reliability or consistency of expert judgments [1] .

Simply calculating the percentage of cases where raters agree does not account for the possibility that some agreement occurs by chance. $\kappa$ is designed to measure the the degree of agreement between raters beyond what is expected by chance. Assume two raters independently classify N cases into categories + and −, and denote by $N_a$ the number of cases on which they agree. Assume $N_c$ agreements occur by chance, and the rest $N_k$ are due to knowledge (not due to chance), so that $N_a=N_c+N_k$. The number of cases remaining after subtracting chance agreements is $N-N_c$. Thus the percentage of the observed agreement $N_a$ in excess of chance agreement is:

$\kappa =\frac{N_a-N_c}{N-N_c}=\frac{N_k}{N-N_c}=\frac{P_a-P_c}{1-P_c},$

where $P_a=N_a/N$ denotes the observed percent agreement, and $P_c=N_c/N$ is the percent agreement due to chance. $P_a$ is observed, whereas $P_c$ must be estimated.

Several approaches have been proposed to estimate the probability of chance agreement. The approach used most commonly in the past (Cohen’s $\kappa$) has recently fallen under criticism [2] [3] , leading to a new approach (Gwet’s $\kappa$) which has been gained popularity over the past several years [1] [4] - [7] . However, we show that the new approach is biased. We demonstrate an unbiased approach to estimating $\kappa$ based on maximum likelihood estimation.

Historically, the most commonly used $\kappa$ statistic has been Cohen’s $\kappa$ [8] [9] , which quantifies interrater reliability for two raters applying binary ratings. Other approaches are discussed at length in [10] - [12] .

Cohen proposed calculating the probability of chance agreement $P_c$ based on an ‘always guess’ model. Suppose two raters A and B independently assign N items to two categories, + and −. Let the numbers of items assigned to each category be $N_A^{+}$, $N_A^{-}$, $N_B^{+}$, $N_B^{-}$, and the number of items on which they agree be $N_a$. Now consider what percentage of cases raters A and B would be expected to agree on if they assigned the same numbers of items to each category as they do in the observed data, but made the assignments at random (“guessing”). Under this model, A and B classify items as + with probabilities $p_A^{+}=N_A^{+}/N$, $p_B^{+}=N_B^{+}/N$, and as − with probabilities $p_A^{-}=N_A^{-}/N$, $p_B^{-}=N_B^{-}/N$. Any agreements under this model occur by chance, with probability

$P_c=p_A^{+}{p}_B^{+}+p_A^{-}{p}_B^{-}.$

Gwet proposed an alternative to Cohen’s $\kappa$, which we call Gwet’s $\kappa$ (also known as AC1 (Agreement Coefficient 1)) that addresses the limitations discussed above [12] . Gwet’s key contribution was a more realistic model of chance agreement, $P_c$, which we call the “occasional guessing” model. Because this model addresses the limitations of Cohen’s $\kappa$, Gwet’s $\kappa$ has been increasingly adopted in studies of IRR [1] [4] - [7] . However, as we show below, Gwet’s $\kappa$ also has important limitations. Specifically, the formula Gwet proposed for estimating $\kappa$ is biased.

Here we present a direct approach to estimating $P\left(R\right)=r$ in Gwet’s occasional guessing model. Unlike Gwet’s $\kappa$, the ML $\kappa$ is not based on a heuristic approximation. Rather, we derive ${\kappa }_{ML}$ by writing down the likelihood of the observed data under the occasional guessing model and then solving for the r that maximizes that likelihood.

Let $X=\left[X_1,X_2,\cdots ,X_N\right]$ represent the agreement and disagreements for the N cases, where $X_i=0$ indicates disagreement and $X_i=1$ indicates agreement. When event R occurs (random guessing), we have $P\left(X_i=0|R\right)=P\left(X_i=1|R\right)=1/2$. For easy cases, raters are not guessing (i.e. $\bar{R}$ occurs), and we have $P\left(X_i=0|\bar{R}\right)=0$, $P\left(X_i=1|\bar{R}\right)=1$. The probability that raters guess is $P\left(R\right)=r$. The probabilities for $X_i$ conditional on r are

$P\left(X_i=0|r\right)=P\left(R\right)P\left(X_i=0|R\right)+P\left(\bar{R}\right)P\left(X_i=0|\bar{R}\right)=r/2$

$P\left(X_i=1|r\right)=P\left(R\right)P\left(X_i=1|R\right)+P\left(\bar{R}\right)P\left(X_i=1|\bar{R}\right)=1-r/2$

The likelihood function for the data is: $P\left(X|r\right)={\displaystyle {\prod }_{i=1}^{N}P\left(X_i|r\right)}$, so the log-likelihood is $L\left(X|r\right)={\displaystyle {\sum }_{i=1}^N\log P\left(X_i|r\right)}$. Splitting the sum into $N_d$ terms in which they disagree ( $X_i=0$) and $N_a$ terms in which they agree ( $X_i=1$), we get

$\begin{array}{c}L\left(X|r\right)=N_d\log P\left(X=0|r\right)+N_a\log P\left(X=1|r\right)\\ =N_d\log \frac{r}{2}+N_a\log \left(1-r/2\right)\end{array}$

Taking the derivative of $L\left(X|r\right)$ with respect to r, setting it equal to zero, and solving, we get:

$\frac{\partial }{\partial r}L\left(X,r\right)=N_d/r_{ML}-\frac{1}{2}{N}_a/\left(1-r_{ML}/2\right)=0$

$\Rightarrow r_{ML}=\frac{2N_d}{N},$

where $N=N_d+N_a$. Note that $N_d/N=P_d$ the probability of disagreement.

This result makes sense: Given that the probability of agreement when raters guess is 1/2, the best estimate from the data of the number of times at least one rater was in fact guessing is twice the number of observed disagreements.

From the above calculation it follows that the estimated probability of agreement due to chance is

$P_c=P\left(R\right)P\left(A|R\right)=r_{ML}/2=N_d/N.$

We now compute the variance of our estimate of $r_{ML}$. The key computation is computing the second moment of $N_d$.

$\begin{array}{c}\text{E}\left[N_d^2\right]=\text{E}\left[{\left({\displaystyle {\sum }_i\overline{X_i}}\right)}^2\right]\\ ={\displaystyle {\sum }_{i\ne j}P\left(\overline{X_i}\overline{X_j}\right)}+{\displaystyle {\sum }_{i}P\left(\overline{X_i}\right)}\\ =\left(N^2-N\right)r^2/4+Nr/2.\end{array}$

Thus,

$\begin{array}{c}\text{Var}\left[r_{ML}\right]=\frac{4}{N^2}\left(\text{E}\left[N_d^2\right]-\text{E}{\left[N_d\right]}^2\right)\\ =\frac{4}{N^2}\left(\left(N^2-N\right)r^2/4+Nr/2-{\left(Nr/2\right)}^2\right)\\ =\frac{r\left(2-r\right)}{N}.\end{array}$

Let $f\left(r\right)=\frac{1-r}{1-r/2}$. The maximum derivative of f over $r\in \left[0,1\right]$ is 2. Thus, we have for all $ϵ>0$, $r\in \left[ϵ,1\right]$:

$\left|f\left(r\right)-f\left(r-ϵ\right)\right|\le 2ϵ.$

In other words, a confidence interval for $r\in \left[r_0-\delta ,r_0+\delta \right]$ translates into a confidence interval for ${\kappa }_{ML}$ which is ${\kappa }_{ML}\in \left[f\left(r_0\right)-2\delta ,f\left(r_0\right)+2\delta \right]$. Confidence intervals can also be calculated numerically using bootstrapping, as shown in Figure 1 .

The preceding sections have dealt with the case of classifying into 2 categories. We can analogously derive ${\kappa }_{ML}$ and r in the case where there is instead an arbitrary number, n, of classes. To do this, we generalize the “occasional guessing” model so that, for hard cases, raters guess all n classes with equal probability. Under this model, the probability of agreement by guessing is

$P_c=P\left(A,R\right)=P\left(R\right)P\left(A|R\right)=r/n,$

and the overall probability of agreement is

$\begin{array}{c}{P}_a=P\left(A,R\right)+P\left(A,\bar{R}\right)\\ =P\left(R\right)P\left(A|R\right)+P\left(\bar{R}\right)P\left(A|\bar{R}\right)\\ =r/n+\left(1-r\right)\cdot 1\\ =1+\frac{r}{n}\left(1-n\right).\end{array}$

Now, to find the theoretical ${\kappa }^{\text{*}}$ in terms of r,

$\begin{array}{c}{\kappa }^{\text{*}}=\frac{P_a-P_c}{1-P_c}\\ =\frac{1+\frac{r}{n}\left(1-n\right)-r/n}{1-r/n}\\ =\frac{1-r}{1-r/n}.\end{array}$

Next we derive the ML estimator of r. Let $X=\left[X_1,X_2,\cdots ,X_N\right]$ represent the agreement and disagreements for the N cases, where $X_i=0$ indicates disagreement and $X_i=1$ indicates agreement. When event R occurs (random guessing), we have $P\left(X_i=0|R,r\right)=P\left(X_i=1|R,r\right)=1/2$. When neither rater guesses (i.e. event $\bar{R}$ occurs), we have $P\left(X_i=0|\bar{R},r\right)=0$, $P\left(X_i=1|\bar{R},r\right)=1$. The probability that raters guess randomly is $P\left(R\right)=r$. The probabilities for $X_i$ conditional on r are

$P\left(X_i=0|r\right)=1-P_a=\frac{r}{n}\left(n-1\right)$

$P\left(X_i=1|r\right)=P_a=1+\frac{r}{n}\left(1-n\right)$

Now, to find ${\kappa }_{ML}$, we maximize the likelihood function for the data $P\left(X|r\right)={\displaystyle {\prod }_{i=1}^{N}P\left(X_i|r\right)}$, or the log-likelihood $L\left(X|r\right)={\displaystyle {\sum }_{i=1}^N\log P\left(X_i|r\right)}$. Splitting the sum into $N_d$ terms with $X_i=0$ and $N_a$ terms with $X_i=1$, we get

$\begin{array}{c}L\left(X|r\right)=N_d\log P\left(X=0|r\right)+N_a\log P\left(X=1|r\right)\\ =N_d\log \left(\frac{r}{n}\left(n-1\right)\right)+N_a\log \left(1+\frac{r}{n}\left(1-n\right)\right).\end{array}$

Taking the derivative with respect to r, setting it equal to zero, and solving, we get:

$\begin{array}{c}\frac{\partial }{\partial r}L\left(X,r\right)=N_d\cdot \frac{n}{r\left(n-1\right)}\cdot \frac{n-1}{n}+N_a\frac{1}{1+\frac{r}{n}\left(1-n\right)}\cdot \frac{n}{1-n}\\ =\frac{N_d}{r}+\frac{N_a}{\frac{n}{1-n}+r}=0\end{array}$

$\Rightarrow r_{ML}=\frac{N_d}{N}\frac{n}{n-1}$

where $N=N_d+N_a$.

We have presented a maximum likelihood approach to estimating the chance agreement probability $P_c$ in Gwet’s “occasional guessing” model of interrater agreement. Our estimator, ${\kappa }_{ML}$, is derived directly from the likelihood function of the data under this model, rather than relying on heuristic approximations as in Gwet’s $\kappa$.

We have shown that the maximum likelihood estimator $r_{ML}$ for the probability of guessing r is simply twice the observed disagreement rate between raters. Consequently, the chance agreement probability estimate $P_c$ used in ${\kappa }_{ML}$ is the observed disagreement rate. We have also generalized this result to the case of raters scoring cases that can belong to multiple classes.

A key advantage of ${\kappa }_{ML}$ is that it is an unbiased estimator of the true value of $\kappa$ predicted by the occasional guessing model. In contrast, we have demonstrated that Gwet’s formula for $P_c$, while overcoming certain limitations of Cohen’s $\kappa$, is itself biased for intermediate levels of agreement.

We have also provided the variance of the ${\kappa }_{ML}$ estimator, which can be used to construct confidence intervals. The variance depends on both the true value of r and the sample size N, decreasing as N increases as expected for a consistent estimator.

In summary, ${\kappa }_{ML}$ provides a principled approach to estimating chance agreement in the occasional guessing model, addressing limitations of previous $\kappa$ statistics. As the use of interrater reliability measures continues to grow across fields, having an unbiased estimator is important for obtaining reliable inferences from data.

Conceptualization, A.M.W., T.M.W. and M.B.W.; methodology, A.M.W., T.M.W. and M.B.W.; writing—original draft preparation, A.M.W., T.M.W. and M.B.W.; writing—review and editing, A.M.W., T.M.W. and M.B.W.; supervision, M.B.W. All authors have read and agreed to the published version of the manuscript.

This research received no external funding.

Not applicable.

Not applicable.

The code that supports the findings of this study is available from the corresponding author upon request.

The following abbreviations are used in this manuscript: