Difference of Inter-annotator Consistency (DIC)

• Difference of Inter-annotator Consistency (DIC) is a quantitative measure that compares structured pairwise annotation agreements using metrics like Cohen’s kappa and Frobenius norms.
• It evaluates how well model predictions preserve the original annotator consistency by contrasting prediction-side and ground-truth agreement matrices, often normalized for scale invariance.
• DIC is adaptable across diverse tasks, operationalized with different agreement measures (e.g., Dice, Krippendorff’s alpha) to reflect structural fidelity rather than mere consensus.

Difference of Inter-annotator Consistency (DIC) denotes a family of quantitative measures for comparing inter-annotator consistency across annotators, annotator groups, conditions, or model predictions. In the multi-annotator tendency learning literature, DIC is explicitly introduced as a model-evaluation metric that compares the inter-annotator similarity structure implied by predicted labels with the corresponding structure in the original annotations, typically through pairwise Cohen’s kappa matrices and a Frobenius-norm discrepancy (Zhang et al., 19 Mar 2025). A later unified evaluation framework retains the same matrix-comparison idea but normalizes the discrepancy by the norm of the ground-truth matrix, yielding a scale-invariant version of DIC (Zhang et al., 14 Aug 2025). In adjacent literatures, the same term is often absent, but analogous constructions appear as differences in mean inter-annotator agreement, differences in Krippendorff’s alpha, or differences of task-specific consistency indices across conditions (Abhishek et al., 12 Aug 2025, Pustu-Iren et al., 2019, Braylan et al., 2022, Tschirschwitz et al., 28 Mar 2026).

1. Canonical definitions

In Multi-annotator Tendency Learning (MATL) and Individual Tendency Learning (ITL), DIC is defined on the pairwise consistency structure among annotators. Let $M \in \mathbb{R}^{n \times n}$ be the ground-truth inter-annotator consistency matrix with entries

$m_{kl} = \kappa(Y_k, Y_l),$

and let $M' \in \mathbb{R}^{n \times n}$ be the corresponding prediction-side matrix with entries

$m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$

In "QuMATL: Query-based Multi-annotator Tendency Learning" (Zhang et al., 19 Mar 2025), DIC is defined as

$\mathrm{DIC} = \|M - M'\|_F,$

where $\|\cdot\|_F$ is the Frobenius norm. The intended interpretation is that smaller values indicate better preservation of annotator tendency information, because the predicted pairwise agreement pattern more closely matches the ground-truth pattern (Zhang et al., 19 Mar 2025).

"A Unified Evaluation Framework for Multi-Annotator Tendency Learning" (Zhang et al., 14 Aug 2025) uses the same matrix construction but defines a normalized variant:

$$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$

The normalization makes the score scale-invariant across datasets with different absolute levels of annotator consistency. In that formulation, DIC is nonnegative, $\mathrm{DIC}=0$ indicates perfect structural match, and lower remains better (Zhang et al., 14 Aug 2025).

Beyond MATL/ITL, the term can be operationalized more broadly as a difference between two consistency summaries, provided the underlying inter-annotator consistency statistic has been specified. This broader usage is explicit in several syntheses even when the source papers themselves do not introduce the term DIC (Abhishek et al., 12 Aug 2025, Pustu-Iren et al., 2019, Braylan et al., 2022, Tschirschwitz et al., 28 Mar 2026).

Context	Consistency substrate	DIC expression
MATL	Pairwise Cohen’s $\kappa$ matrix	$\|M-M'\|_F$
Unified ITL evaluation	Pairwise Cohen’s $m_{kl} = \kappa(Y_k, Y_l),$0 matrix	$m_{kl} = \kappa(Y_k, Y_l),$1
Skin lesion segmentation	Mean pairwise Dice	$m_{kl} = \kappa(Y_k, Y_l),$2
Historical video annotation	Krippendorff’s $m_{kl} = \kappa(Y_k, Y_l),$3	$m_{kl} = \kappa(Y_k, Y_l),$4
Complex-task consistency	$m_{kl} = \kappa(Y_k, Y_l),$5, $m_{kl} = \kappa(Y_k, Y_l),$6, or $m_{kl} = \kappa(Y_k, Y_l),$7	$m_{kl} = \kappa(Y_k, Y_l),$8
KALOS diagnostics	$m_{kl} = \kappa(Y_k, Y_l),$9 or pairwise $M' \in \mathbb{R}^{n \times n}$0	Difference of vitality, pairwise, or condition scores

The table makes clear that DIC is not a single universally fixed statistic. Its mathematical form depends on whether the objective is structural fidelity of a model, comparison of annotator cohorts, or contrast between annotation conditions.

2. Agreement substrate and metric dependence

DIC is only as meaningful as the inter-annotator consistency statistic on which it is built. For nominal categorical labeling, the dominant substrate in MATL/ITL is Cohen’s kappa, chosen because it is chance-corrected and suitable for pairwise comparison of annotator label sequences (Zhang et al., 19 Mar 2025, Zhang et al., 14 Aug 2025). In broader annotation methodology, metric choice is task-dependent: Cohen’s kappa is standard for two annotators; Fleiss’s kappa, Scott’s pi, or Krippendorff’s alpha are used for multiple annotators; weighted kappa and Krippendorff’s alpha with ordinal or interval distance are used for ordered scales; ICC is used for continuous ratings; and segmentation or boundary tasks may instead rely on boundary $M' \in \mathbb{R}^{n \times n}$1, $M' \in \mathbb{R}^{n \times n}$2, WindowDiff, Dice, or task-specific distance functions (James, 6 Mar 2026).

For complex structured tasks, "Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks" (Braylan et al., 2022) emphasizes a distance-based view of inter-annotator consistency. It defines observed within-item distances $M' \in \mathbb{R}^{n \times n}$3, expected inter-item distances $M' \in \mathbb{R}^{n \times n}$4, and then computes consistency via Krippendorff’s alpha,

$M' \in \mathbb{R}^{n \times n}$5

or via two distribution-sensitive alternatives: a KS-based score $M' \in \mathbb{R}^{n \times n}$6 and a $M' \in \mathbb{R}^{n \times n}$7 score representing the fraction of observed distances that are significantly better than chance. Within that framework, DIC becomes the difference of whichever consistency functional is chosen:

$M' \in \mathbb{R}^{n \times n}$8

This makes DIC explicitly contingent on the measurement model for agreement (Braylan et al., 2022).

A similar dependence appears in image segmentation. In "What Can We Learn from Inter-Annotator Variability in Skin Lesion Segmentation?" (Abhishek et al., 12 Aug 2025), inter-annotator agreement for image $M' \in \mathbb{R}^{n \times n}$9 is defined as mean pairwise Dice,

$m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$0

with

$m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$1

A groupwise DIC then arises naturally as

$m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$2

In "Investigating Correlations of Inter-coder Agreement and Machine Annotation Performance for Historical Video Data" (Pustu-Iren et al., 2019), the analogous quantity is the difference in Krippendorff’s alpha between expert and non-expert annotators. These examples show that DIC is a derived contrast, not an agreement coefficient in its own right.

3. Computation in multi-annotator tendency learning

The matrix-based DIC used in MATL/ITL begins with annotator-indexed label sequences. Let annotator $m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$3 label the item subset $m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$4, and let the pairwise overlap be $m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$5. In the unified framework, pairwise consistency is computed only when $m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$6, where $m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$7 is a minimum overlap threshold introduced to stabilize the estimate (Zhang et al., 14 Aug 2025). Ground-truth labels are denoted

$m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$8

and model predictions are

$m'_{kl} = \kappa(\hat{Y}_k, \hat{Y}_l).$9

For each valid pair $\mathrm{DIC} = \|M - M'\|_F,$0, one computes

$\mathrm{DIC} = \|M - M'\|_F,$1

The matrices $\mathrm{DIC} = \|M - M'\|_F,$2 and $\mathrm{DIC} = \|M - M'\|_F,$3 are then assembled and compared by Frobenius norm, either directly or in normalized form (Zhang et al., 14 Aug 2025).

When some annotator pairs do not satisfy the overlap criterion, the unified framework introduces a binary mask $\mathrm{DIC} = \|M - M'\|_F,$4 and computes masked Frobenius norms:

$\mathrm{DIC} = \|M - M'\|_F,$5

This preserves comparability while avoiding unstable pairwise kappas from tiny overlaps (Zhang et al., 14 Aug 2025).

"QuMATL: Query-based Multi-annotator Tendency Learning" (Zhang et al., 19 Mar 2025) describes the corresponding practical pipeline in simpler terms. For each annotator pair, the item intersection is collected; model probabilities, if present, are converted to hard labels via argmax; pairwise kappas are computed on aligned item subsets; and the resulting matrices are compared by

$\mathrm{DIC} = \|M - M'\|_F,$6

The reported computational complexity is $\mathrm{DIC} = \|M - M'\|_F,$7 for pairwise kappa computation, where $\mathrm{DIC} = \|M - M'\|_F,$8 is the average intersection size per annotator pair, and $\mathrm{DIC} = \|M - M'\|_F,$9 memory to store the two matrices (Zhang et al., 19 Mar 2025).

This workflow makes DIC a structural rather than per-item metric. It does not ask whether a model predicts the most common label well; it asks whether the model reconstructs the network of agreements and disagreements among annotators.

4. Empirical behavior in MATL and ITL

The empirical role of DIC is clearest in MATL/ITL benchmarks. "QuMATL: Query-based Multi-annotator Tendency Learning" (Zhang et al., 19 Mar 2025) reports DIC on STREET, which has 10 annotators across five perspectives, and on AMER, which has 13 annotators. The paper reports the following values, with lower being better: STREET-Happiness, D-LEMA 0.62, PADL 0.48, MaDL 0.45, Ours 0.43; STREET-Healthiness, 0.59, 0.52, 0.45, 0.38; STREET-Safety, 0.36, 0.32, 0.29, 0.24; STREET-Liveliness, 0.51, 0.43, 0.39, 0.27; STREET-Orderliness, 0.61, 0.57, 0.59, 0.54; and AMER, 0.42, 0.36, 0.31, 0.23 (Zhang et al., 19 Mar 2025). These results are used to argue that shared, learnable query tokens capture inter-annotator correlations and thereby better preserve tendency structure.

The later unified evaluation framework evaluates DIC together with Behavior Alignment Explainability (BAE) and reports that DIC distinguishes methods more clearly than conventional metrics such as ACC, Fleiss’ kappa, and PCC (Zhang et al., 14 Aug 2025). In that study, random assignment yields very high DIC values in the range 0.86–0.93, a consensus baseline yields 0.51–0.61, and the best method, QuMAB, achieves substantially lower values such as 0.23 on AMER, 0.24 on STREET Safety, and 0.27 on STREET Liveliness (Zhang et al., 14 Aug 2025). The same paper states that reported DIC values are typically in $\|\cdot\|_F$0 depending on method quality and dataset, and that means $\|\cdot\|_F$1 standard deviations are reported across repeated runs (Zhang et al., 14 Aug 2025).

The interpretive convention is stable across both papers: smaller DIC means that the predicted inter-annotator consistency structure more faithfully matches the ground truth. This suggests that DIC is particularly informative when the objective is not consensus accuracy but structural fidelity to annotator-specific behavior.

5. Operationalizations beyond MATL

Outside annotator-tendency learning, DIC appears less as a named metric than as a natural contrast between consistency summaries. In skin lesion segmentation, "What Can We Learn from Inter-Annotator Variability in Skin Lesion Segmentation?" (Abhishek et al., 12 Aug 2025) defines per-image inter-annotator agreement by mean pairwise Dice and then induces two DIC-style quantities. The first is groupwise:

$\|\cdot\|_F$2

The second is within-image spread:

$\|\cdot\|_F$3

On IMA++, benign lesions have $\|\cdot\|_F$4 and malignant lesions have $\|\cdot\|_F$5, giving $\|\cdot\|_F$6; the benign distribution is significantly higher by Mann–Whitney $\|\cdot\|_F$7 test ($\|\cdot\|_F$8), and first-order stochastic dominance testing supports benign $\|\cdot\|_F$9 malignant (Abhishek et al., 12 Aug 2025).

In historical video annotation, the relevant base measure is Krippendorff’s alpha. "Investigating Correlations of Inter-coder Agreement and Machine Annotation Performance for Historical Video Data" (Pustu-Iren et al., 2019) reports concept-level aggregated alpha values of 0.80 for experts, 0.79 for non-experts, and 0.78 jointly, and person-recognition values of 0.83, 0.71, and 0.76, respectively. From these, a natural task-level DIC is

$$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$0

which equals 0.01 for concepts and 0.12 for persons (Pustu-Iren et al., 2019). The paper does not report confidence intervals or significance tests for these alpha differences, so the contrasts are descriptive rather than inferential (Pustu-Iren et al., 2019).

For complex structured annotation tasks, "Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks" (Braylan et al., 2022) generalizes DIC to any chosen consistency functional:

$$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$1

Here, DIC may compare tasks, distance functions, annotator groups, or guideline variants. In KALOS, a meta-algorithm for complex vision tasks, DIC is treated as a derived diagnostic from agreement indices computed after resolving instance correspondence. Examples include per-annotator vitality contrasts,

$$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$2

pairwise contrasts,

$$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$3

and threshold-based contrasts,

$$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$4

all defined on a nominal reliability matrix produced by the Localization First pipeline (Tschirschwitz et al., 28 Mar 2026).

These operationalizations are not interchangeable, but they share one core principle: DIC quantifies a difference between consistency structures, rather than consistency itself.

6. Interpretation, limitations, and reporting

DIC inherits the statistical properties and pathologies of its underlying agreement measure. In the matrix-based MATL/ITL formulation, Cohen’s kappa is chance-corrected and more robust to class imbalance than raw agreement, but it can still be unstable when annotator overlaps are small, which is why the unified framework introduces a minimum overlap threshold $$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$5 and optional masking (Zhang et al., 14 Aug 2025). The same framework notes that DIC becomes more sensitive when $$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$6 is small, because very low overall ground-truth consistency shrinks the normalization denominator (Zhang et al., 14 Aug 2025).

General agreement methodology imposes additional constraints. "Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation" (James, 6 Mar 2026) emphasizes that metric choice must match the task type, that label imbalance and missing data can distort reliability estimates, and that transparent reporting should include confidence intervals, disagreement diagnostics, label distributions, and explicit description of the weighting or distance function. The same paper also states that agreement demonstrates reliability, not validity, and that disagreement can be informative rather than merely noisy (James, 6 Mar 2026). These observations apply directly to DIC, since a DIC score summarizes differences between agreement structures but does not itself establish correctness of either the annotation scheme or the model.

For ordinal data, the situation is particularly delicate. QuMATL applies standard Cohen’s kappa to STREET’s 6-point ordinal scale, while explicitly noting that weighted kappa could be used in principle but is not part of the paper’s DIC definition (Zhang et al., 19 Mar 2025). For complex structured tasks, the dependence is even stronger: DIC values built from $$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$7, $$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$8, or $$\mathrm{DIC} = \frac{\| \mathbf{M} - \mathbf{M}' \|_F}{\|\mathbf{M}\|_F}.$$9 can change materially with the chosen distance function, and the complex-task literature recommends using $\mathrm{DIC}=0$0 to rank distance functions and $\mathrm{DIC}=0$1 for interpretable “good enough” judgments (Braylan et al., 2022).

A further complication is that consistency differences interact with downstream evaluation protocols. An earlier computer vision study on annotator variance reports that detector rank is highly dependent upon the method used to form the ground truth and that confidence bounds are needed when comparing algorithms (Lampert et al., 2013). This suggests that DIC-style contrasts are most informative when reported alongside uncertainty quantification, overlap diagnostics, and explicit description of the aggregation or correspondence procedure.

As a result, DIC is best understood not as a standalone universal score, but as a second-order diagnostic. It is most valuable when the underlying agreement statistic is well matched to the annotation task, overlaps are adequate, uncertainty is reported, and the substantive question concerns the preservation, comparison, or perturbation of inter-annotator consistency structures rather than consensus alone.