Mechanism underlying cross-model criterion-alignment differences

Identify the mechanism underlying the differences in criterion alignment across GPT-5.4 and Claude Opus 4.7 when evaluating hallucinations under the same nominal faithfulness-style prompt.

Background

The paper finds that frontier judges can behave substantially differently under the same nominal faithfulness-oriented prompt: GPT-5.4 exhibits strong false-positive dominance, whereas Claude Opus 4.7 is much closer to the factual-correctness region. Both models move toward more similar behavior when prompted explicitly for factual correctness, suggesting that differences in default interpretation of the evaluation criterion are not explained by model capability alone. The paper reports the empirical contrast but does not determine what mechanisms produce these cross-model differences, leaving their explanation for future work.

References

Our experiments do not identify the mechanism underlying these cross-model differences, which we leave for future work.

— The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation  (2610.08026 - Valjakka et al., 6 Oct 2026) in Section 4.3, “Judge Capability Is Not Enough”

These observations are consistent with same-family preference bias, but the two-judge design cannot identify which judge deviates, in which direction, or through what mechanism; systematically different standards for response features remain an alternative explanation.

— Before Bringing It Up: When and How AI Companions Should Use Memor  (2610.09470 - Guo et al., 7 Oct 2026) in Section 3.4, Evaluator Validity Analysis