Position-Invariant Evaluation Frameworks for LLM-as-a-Judge

Design a position-invariant evaluation framework for LLM-as-a-Judge that eliminates dependence on the ordering of candidate responses in pairwise and listwise judgments.

Background

LLM judges can exhibit positional bias, favoring candidates based on their placement in the prompt rather than intrinsic quality. Such bias can be exploited by adversaries who manipulate ordering to gain advantage.

The paper identifies the need for frameworks that neutralize ordering effects to ensure fair and consistent judgments.

References

The open research problems in this context are: Design a positional-invariant evaluation framework.

— Security in LLM-as-a-Judge: A Comprehensive SoK  (2603.29403 - Masoud et al., 31 Mar 2026) in Section 7.2, Positional Bias and Evaluation Manipulation (Challenges and Open Problems)

Whether the option order or the criteria wording still moves the returned probabilities is a separate question, and this study does not test it, so the claim made here is about output validity rather than about the absence of order or wording effects.

— Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)  (2609.24052 - Rafe et al., 21 Sep 2026) in Section 2.3, subsection “Decision models without text generation and the calibration of language-model confidence”

The corpus therefore cannot support any ``identify the disclosing reference among $K$'' evaluation without re-randomising slot order and holding the examiner gold out.

— Verifiable, Articulable, and Tacit Components of Preference  (2610.03025 - Spangher et al., 2 Oct 2026) in Appendix, Section “Claim-level disclosure cohort and pipeline,” paragraph ending “cannot support”

While pairwise evaluation is currently the most prevalent paradigm, a natural extension of this work would be to investigate whether our findings regarding verdict instability and bias transfer to list-wise ranking or multi-model grading frameworks.

— All Verdicts are Not Equal: Rethinking LLM Judge Reliability  (2610.12083 - Kumar et al., 8 Oct 2026) in Limitations section

In the perception probe the optimised image is rated slightly more premium than the standard image, so the manipulation is perceptible; yet its price-equivalent cannot be identified, because primacy saturates the response channel in the small-premium regime that matters for elicitation. This is a boundary case of our counterbalanced ($50/50$) policy estimand (Section~\ref{sec:est})---when option-order choice is near-deterministic, the policy average is pinned at $0.50$ and PE is not identified---not evidence of a null; a position-free paradigm (single-product valuation, or an image-only 2AFC) is required to estimate an image premium.

— A Cue's Hidden Price: Auditing Seller Influence on AI Shoppers  (2610.25843 - Buecheler et al., 19 Oct 2026) in Appendix, Section "The presentation (image) cue: full results" (Appendix~\ref{app:image})