Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering

Abstract: LLMs are increasingly used as judges to evaluate code artifacts when exhaustive human review or executable test coverage is unavailable. LLM-judge is increasingly relevant in agentic software engineering workflows, where it can help rank candidate solutions and guide patch selection. While attractive for scale, current practice lacks a principled account of reliability and bias: repeated evaluations of the same case can disagree; small prompt edits can swing outcomes; and seemingly semantics-preserving, human-equivalent perturbations may elicit divergent verdicts. This paper studies LLM-as-a-Judge for code through a measurement-first lens. We analyze two pointwise judging regimes across code generation, code repair task, and test generation, and we systematically probe prompt-induced biases. Our study considers difficulty levels for repeated runs and controlled prompt interventions that isolate one presentation cue at a time, and it evaluates judges using consistency and sensitivity to bias. We find that judge decisions are highly sensitive to prompt biases even when the underlying code snippet is unchanged. Across all three tasks, several biases systematically shift preferences toward the option favored by the prompt, improving accuracy when that option aligns with the gold answer but substantially reducing it otherwise. In some settings, these effects are large enough to change task-level conclusions and alter relative model rankings. These findings show that reported judge performance may reflect prompt artifacts rather than stable assessment ability, posing a direct threat to the validity and reproducibility of code evaluation. We therefore argue that LLM-as-a-Judge studies should report bias sensitivity alongside accuracy and incorporate explicit controls to support more trustworthy model comparison in software engineering.

• The paper reveals that prompt-induced biases significantly affect LLM judges' evaluations in software engineering tasks using controlled pairwise comparisons.
• It demonstrates that superficial cue alignment can either inflate accuracy or lead to drastic failures, depending on whether the cue corresponds to the gold standard.
• The authors recommend debiasing protocols and ensemble evaluations to ensure robust, reliable software engineering assessments.

Critical Analysis of "Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering" (2604.16790)

Overview

"Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering" presents a systematic, measurement-driven investigation into the reliability and bias sensitivity of LLMs acting as judges for code-related evaluation tasks. The work critically examines how prompt-induced superficial cues affect LLM-judge decisions in software engineering (SE) endpoints, spanning code generation, program repair, and test case generation. The central thesis is that the accuracy and consistency of LLM-judge verdicts are substantially, and often predictably, influenced by prompt artifacts rather than solely by substantive code semantics, introducing a considerable threat to the reproducibility and validity of model comparison and automated evaluation in SE pipelines.

Methodological Rigor and Experimental Protocol

The authors implement a robust, controlled experimental framework based on fixed-evidence pairwise evaluation, leveraging the CODEJUDGEBENCH dataset that encompasses three representative tasks: code generation, code repair, and test generation. Judges (Qwen3-4B, Qwen2.5-Coder-3B, GPT-4) are tasked with pairwise comparisons under various prompt-induced bias conditions, derived from known artifacts in LLM evaluations, but carefully adapted for code contexts (e.g., authority citation, verbosity, chain-of-thought, model-name provenance, sentiment, etc.).

Two core research questions are addressed:

1. RQ1: Quantifying the magnitude and directionality of explicit prompt biases in LLM-judge outcomes.
2. RQ2: Assessing test-retest reliability (consistency) when the same prompt and candidates are presented multiple times.

Their protocol explicitly isolates superficial presentation effects (e.g., A/B order swaps) from semantic differences, ensuring bias measurements reflect only changes in prompt structure.

Key Empirical Findings

Prompt-Induced Biases and Systematic Decision Shifts

Across all models and tasks, LLM-judges display marked sensitivity to prompt engineering. Certain biases — particularly Authority, Refined, Chain-of-Thought (CoT), Sentiment — consistently and robustly increase the selection of the favored option (A), regardless of whether A contains the gold-standard output. This induces:

• Accuracy Improvement: If the bias aligns with the correct answer, accuracy approaches ceiling.
• Catastrophic Accuracy Reduction: If the bias conflicts with the gold answer (i.e., the cue is attached to the decoy), accuracy collapses, especially on hard instances and for TestGen.

Other biases (e.g., Verbosity, Diversity) produce opposing effects, sometimes reversing the selection tendency toward B. The impact is symmetrical and polarity-flipping under candidate order swaps, underscoring that the observed effects are not attributable to code content or inherent candidate quality.

Notably, the direction and magnitude of delta-accuracy (from the unbiased prompt) are consistent across open-source and closed-source models, with Qwen2.5-Coder-3B sometimes amplifying these effects due to stronger coupling between prompt cues and final decisions.

Consistency and Reliability under Controlled Conditions

Test-retest analysis reinforces these findings. Consistency Rates (CR) are high for easier and more canonical SE tasks (CodeRepair), but degrade considerably for more ambiguous or underspecified settings (TestGen hard). Intriguingly, powerful biases such as Sentiment or Authority increase repeatability, but this "stability" simply reflects deterministic anchoring on the injected cue rather than robust evidence-based judging. Thus, high CR under biased prompting indicates systematic error, not genuine reliability.

Baseline models differ: GPT displays higher intrinsic CR without bias, but all models show inflated CR under the strongest biases on ambiguous (hard) data, revealing that LLMs can reliably repeat biased errors.

Answer Rate and Model Format Adherence

A critical but often-overlooked failure mode is the model's failure to output a well-formed verdict. For instance, Qwen2.5-Coder-3B (code-specialized) responds in format >99% of the time, while generic Qwen3-4B may produce valid A/B verdicts in less than half the cases, often drifting to unrelated completions or free-form text. This result indicates that not all LLMs are suitable as plug-and-play judges, and answer format reliability must be reported alongside correctness.

Implications for Software Engineering and AI Evaluation

The results have direct theoretical and practical consequences:

SE and LLM-as-a-Judge Benchmarking

Reported aggregate accuracy for model A vs. model B under a single prompt and ordering is highly unstable; small, semantics-preserving perturbations (e.g., altering candidate order, adding minor prompt adjectives) can change system rankings and invalidate statistical claims. As a result, conclusions about model superiority or pipeline efficacy may be artifacts of prompt design rather than real differences in underlying model or candidate quality.

AI Evaluation Practice

The findings generalize to any pipeline relying on LLM-based evaluation proxies in contexts with ambiguous or subjective rubrics. In SE, agentic pipelines — where LLMs arbitrate patch ranking, feedback signals, or workflow routing — must implement explicit controls for order and prompt-induced bias. Reporting should systematically include:

• Results under order/A/B swaps.
• Central tendency and variance of accuracy over repeated runs and prompt perturbations.
• Breakdown by task difficulty strata.

Crucially, downstream toolchains must treat LLM-judge outputs as conditional measurements with a quantifiable and significant uncertainty, not as ground-truth oracles.

Directions for AI Debiasing and Reliable Automation

The study highlights the need for:

• Prompt and rubric sanitization to remove extraneous cues.
• Aggregation across multiple prompts and orders to estimate robust, invariant ranking.
• Adaptive calibration and escalation pipelines: when LLM verdicts are ambiguous or prompt-sensitive, defer to human adjudication or executable/runnable evidence.
• Development of judge models explicitly trained to minimize sensitivity to non-semantic features.

Potential Risks and Future Directions

The exposure of prompt-level vulnerabilities in LLM-judges signals risks not only for reproducibility and scientific validity but also for adversarial manipulation when these systems scale to critical evaluation tasks, peer review, or agentic orchestration. The case for strictly neutral, rigorously controlled evaluation protocols is unambiguous. Unmitigated, these biases threaten both fair benchmarking and deployment safety.

Future research should address:

• Multi-prompt ensemble methods for more robust judging.
• Cross-lingual and multi-modal extension of bias analysis (including code with embedded NL or docstrings).
• Task-specific debiasing/training objectives.
• Standardized reporting checklists for judge-based SE evaluation following best practices from metrology.

Conclusion

This work establishes, in quantifiable terms, that LLM-as-a-Judge for code evaluation is highly susceptible to prompt-induced, non-semantic bias, leading to large and systematic instability in outcome accuracy and reliability. These effects are persistent across model architectures, training regimes, and SE tasks, with significant methodological and deployment ramifications. The main recommendation is for the SE and AI research communities to treat judge design and prompt engineering as first-class, measurable factors, enforcing rigorous, exposure-controlled evaluation and transparent reporting. Through such protocols, it will become possible to distinguish meaningful progress in code intelligence and SE automation from fragile gains driven by prompt artifacts.

Citation: "Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering" (2604.16790)