Determine whether CBI patterns generalize beyond financial question answering

Determine whether the high-consistency/incorrect quadrant mass and its concentration patterns observed in financial question answering also occur in other specialized domains, including legal, biomedical, and engineering question answering.

Background

The study evaluates consistency–accuracy dissociation exclusively in financial question answering, using professionally validated financial examination and numerical-reasoning datasets. It finds that a substantial share of incorrect answers lies in the high-consistency/incorrect quadrant, where agreement-based evaluation signals incorrectly indicate reliability.

The authors explicitly leave unresolved whether the same quadrant mass and concentration patterns appear in other specialized domains, naming legal, biomedical, and engineering question answering as examples. Cross-model convergence in the financial experiments suggests a potentially broader mechanism, but the paper does not establish its domain generality.

References

Whether the same quadrant mass and the same concentration patterns hold in other specialized domains, such as legal, biomedical, or engineering QA, remains an empirical question, although the cross-model convergence we observe suggests that the mechanism is not specific to finance.

— When Judges Agree and Are Wrong: Consistency-Accuracy Dissociation as a Blind Spot for Label-Free Evaluation  (2610.21128 - Zhang et al., 15 Oct 2026) in Section 5, subsection “Limitations”