Establish reliability for reconstruction and non-Toki-Pona interfaces

Establish the reliability of operative understanding-rate annotations for the T1–T3 controlled English reconstructions and for the Zulu, Yoruba, CaesarCipher, FormalLogic, ArtPrompt, and other evaluated interfaces through independent double annotation.

Background

Independent double annotation is reported for the DR-TOK condition, where operative understanding-rate agreement is quantified using Cohen’s kappa and raw agreement. However, the controlled English reconstruction conditions T1–T3 and the other interfaces are not covered by the same comprehensive independent reliability assessment.

Because operative understanding rate depends on joint judgments about task recognition and response framing, extending independent annotation to these conditions is necessary to determine whether the reported recovery patterns are robust to reader variation across reconstruction levels and interfaces.

References

Independent double annotation covers DR-TOK; reliability for T1--T3 and the other interfaces remains to be established.

— Same Outcome, Different Evidence: Intent Recovery in LLM Safety Evaluation  (2610.11766 - Jiang et al., 8 Oct 2026) in Appendix, Section “Reliability and Reader Sensitivity,” subsection “Headline-Metric Agreement”