Decompose the sources of annotation disagreement

Determine the relative contributions of differences in task interpretation, response-frame assignment, and annotation error to the observed disagreement in operative understanding-rate measurements.

Background

The paper evaluates operative understanding rate and attack success rate using human and machine-assisted annotations that distinguish task recognition, response framing, and harmful assistance. Independent annotation agreement varies across models, and the authors note that disagreement may arise from multiple sources, including differences in interpretation, frame assignment, and labeling errors.

The unresolved issue is to disentangle these possible sources rather than treating the aggregate disagreement statistics as if they had a single known cause. Resolving it would clarify how much of the uncertainty in operative recovery estimates reflects the evaluation subjects’ responses and how much reflects the measurement procedure.

References

Differences in task interpretation, frame assignment, and annotation error all contribute to the observed disagreement, with their relative contributions unresolved.

— Same Outcome, Different Evidence: Intent Recovery in LLM Safety Evaluation  (2610.11766 - Jiang et al., 8 Oct 2026) in Appendix, Section “Reliability and Reader Sensitivity,” subsection “Headline-Metric Agreement”