Robust Ranking Mechanisms for Candidate Evaluation

Develop robust ranking mechanisms for LLM-as-a-Judge that evaluate candidate responses accurately and resist manipulation, bias, and ordering effects.

Background

Current ranking procedures used by LLM judges can be sensitive to contextual artifacts such as position, leading to unfair or unstable outcomes.

The paper calls for ranking methods that maintain reliability under adversarial or biased conditions, improving trust in comparative evaluations.

References

The open research problems in this context are: Develop a robust ranking mechanism for evaluating candidate responses.

— Security in LLM-as-a-Judge: A Comprehensive SoK  (2603.29403 - Masoud et al., 31 Mar 2026) in Section 7.2, Positional Bias and Evaluation Manipulation (Challenges and Open Problems)

Judge reliability remains an open challenge that may bias measured ASR and the resulting defense rankings. Our jailbreak evaluation uses the HarmBench Judge and the JBB-Behaviors dataset, both widely adopted in prior work; the HarmBench Judge additionally ranks best overall on the reliability-efficiency trade-off in our judge comparison (Appendix~\ref{sec:appendix:c}). Improving judge reliability and the fairness of empirical comparisons remains an important future direction.

— CASCADE Against Jailbreaks: Combination Across Stages with Controlled Attack-Defense Evaluation  (2609.21793 - Luo et al., 18 Sep 2026) in Section 5, Limitations, paragraph “Judge reliability”; see also Appendix C, Section “Jailbreak Evaluation”

We believe there is still room for improvement on applying advanced coordination techniques for listwise selection, other than just using sliding windows, such as TourRank, or Set-based ranking. We leave the exploration of more advanced coordination strategies as future work.

— Replacing Training with Memory: Listwise Selection for Text-to-SQL  (2609.00834 - Jeong et al., 1 Sep 2026) in Limitations section

Its agreement with five annotators is only near a noisy human ceiling (Krippendorff's $\alpha=0.511$, MAE${}=2.111$), so only aggregate, distributional shifts are reliable; confirming the ranking with an independent judge and a human study remains future work.

— AlcaTRAz - Anchored Tree-Rule Defense Against Jailbreaks  (2609.03693 - Reš et al., 3 Sep 2026) in Section 6, Limitations

Relating the conditions under which manipulation of preferences fails to those under which manipulation of assessments succeeds is an open question.

— Epistemic Disturbance in the Graph Model for Conflict Resolution: State-Preserving Actions, Four-Valued Assessments, and the Distinction between Capability and Intention  (2610.11690 - Kato, 8 Oct 2026) in Discussion, paragraph “Limitations and further work”