Characterize the difficulty of entropy-class prediction for humans

Characterize how difficult it is for human annotators to predict the entropy class of an instance’s human judgment distribution.

Background

NAPHA routes predicted soft-label distributions to specialized alignment models according to low, medium, or high entropy classes. The paper shows that oracle entropy labels substantially improve alignment, but reports that standard prompting does not enable LLMs to predict those classes reliably. The authors leave unresolved whether human annotators would find entropy-class prediction easy or difficult.

References

Note that it is unclear how hard the task of predicting the entropy class is for humans; studying this is an interesting direction for future work.

— Post-hoc Alignment of LLM-judges to Human Judgment Distribution  (2609.01073 - Steindl et al., 1 Sep 2026) in Section 5, subsection “Performance per entropy class,” footnote following the discussion of entropy-class assignment