AdvJudge-Zero Vulnerability

• AdvJudge-Zero is a vulnerability framework showing that adversarial low-perplexity control tokens can reliably flip binary verdicts in judge models.
• It leverages beam search and next-token sampling to discover token suffixes that perturb shallow decision boundaries, causing high false positive rates.
• Defense via LoRA-based adversarial fine-tuning effectively reduces false positives while maintaining robust evaluation performance.

AdvJudge-Zero refers to a vulnerability and corresponding attack and defense framework in modern LLM-as-a-Judge architectures, particularly those used in post-training reward modeling for pipelines such as RLHF (Reinforcement Learning from Human Feedback), DPO (Direct Preference Optimization), and RLAIF (Reinforcement Learning with AI Feedback). The core insight is that short, low-perplexity control-token suffixes may be found efficiently, often via beam search and next-token sampling, which reliably induce high false positive rates by flipping binary “No” verdicts to “Yes” during automated evaluation, reward ranking, or model selection (Li et al., 19 Dec 2025). AdvJudge-Zero formalizes this phenomenon, characterizes its underlying “soft mode” in hidden state geometry, provides an efficient method for finding such adversarial tokens, and empirically demonstrates vulnerability in multiple judge architectures.

1. LLM-as-a-Judge Vulnerability: Problem Formulation

In the LLM-as-a-Judge paradigm, a LLM $M$ is prompted with a composite template $X$ containing a question, a candidate solution, and a reference answer, after which the model outputs a binary verdict—“Yes” or “No”—at a specific decision token position. The model’s next-token logits at this position, $\mathbf{z}(X)$, yield a logit-gap: $\Delta(X) = z_{\text{Yes}}(X) - z_{\text{No}}(X)$ Correct refusals are characterized by $\Delta(X)<0$, but appending a suffix (“control-token” sequence) $c$ yields

$\Delta(X\oplus c) > 0$

representing a binary flip from “No” to “Yes.” Critically, the attack seeks control token sequences $c$ that are short and have low perplexity under the model itself, making them likely to be generated by policy agents or as realistic continuations in reward pipelines.

A small control token suffix induces a finite perturbation to the final hidden state before verdict, $\Delta\mathbf{h}$, moving it across the linear decision boundary implemented by the verdict head. Because the decision head is a high-gain shallow map, even modest perturbations can reliably induce flips.

2. Adversarial Control Token Discovery Methodology

The AdvJudge-Zero algorithm searches for and ranks effective control token suffixes via model-internal next-token sampling and beam search, subject to a maximum length ($n$) and perplexity constraint ($\mathrm{PPL}(c) \le \tau$). The objective is to maximize the number of prompts for which appending $c$ flips the model’s verdict: $$c^* = \arg\max_{c:\; \mathrm{len}(c)\le n,\, \mathrm{PPL}(c)\le \tau} \sum_{i=1}^N \mathbf{1}\bigl[\Delta(X_i\oplus c)>0\bigr]$$ or equivalently minimize a differentiable ReLU-based surrogate.

Algorithmically, the model samples candidates from its own distribution over possible next tokens, expands promising beams up to length $n$, and evaluates flip rate and mean logit-gap increments for each $c$. Suffixes that are effective and plausible (in terms of typical decoding) are thereby discovered without reliance on labeled seeds or whitebox gradients.

Unlike prior gradient-based adversarial methods, AdvJudge-Zero’s search leverages generation probabilities, enforcing realistic surface-form constraints and better simulating reward hacking in downstream RLHF or DPO usage.

3. Geometric Characterization: The Low-Rank “Soft Mode”

Analysis of the hidden state changes induced by successful control tokens reveals that adversarial perturbations $\Delta\mathbf{h}$ are not distributed isotropically in model space. Principal component analysis (PCA) of $\Delta\mathbf{h}$ across many base-prompt/suffix pairs shows that a single direction (PC1) accounts for a large fraction ($\sim30\%$) of the total variance, substantially exceeding the null baseline ($1/d$).

Moreover, this principal direction is consistently anti-aligned (cosine similarity ≈ –0.09 to –0.13, Z-scores < –4) with the linear classifier’s “refusal direction” $\mathbf{w}_F$ (the weight difference for “No” vs. “Yes”). This empirical result establishes that the most effective control-token sequences excite a shared low-rank “soft mode” in hidden state space, directly steering the verdict output toward acceptance. The vulnerability is thus rooted in a shallow, near-linear separation at the decision head, susceptible to small, structure-aligned perturbations (Li et al., 19 Dec 2025).

4. Empirical Evaluation: Models, Datasets, and Results

AdvJudge-Zero is tested across both large open-weight models (Llama-3, Qwen-2.5/3, Gemma-3) and specialized judge models (Omni-Judge, general-verifier, Qwen2.5-7B-Instruct-RLVR, Master-RM Judge) on benchmarks spanning mathematical, logical, and scientific problem domains (AIME, MATH, GSM8K, MultiRLVR).

Direct application of AdvJudge-Zero yields false positive rates (FPR) close to 100% on most open-weight and many specialized judge models, far surpassing prior “seeded” attacks such as the Master-RM baseline. Specialized contrastive and step-by-step judges show significantly improved robustness (FPR < 1%), but most generic judge architectures present a broad low-perplexity attack surface, including in cross-model (transfer) settings.

Notably, FPR rarely improves with longer suffixes beyond 3–5 tokens; in many cases, attack effectiveness saturates or even declines, supporting the “soft mode” mechanism over a hypothesis of cumulative gradient progress.

Comparative FPR Results

Model or Method	AIME	MATH	GSM8K	MultiRLVR
Qwen3-4B, AdvJudge	100%	100%	100%	100%
Qwen2.5-7B, AdvJudge	99.3%	99.9%	99.9%	79.5%
Omni-Judge, AdvJudge	96.5%	99.4%	99.8%	49.5%
general-verifier	0%	0%	0.01%	3.8%

The data demonstrate the critical vulnerability across a broad class of judge model architectures.

5. Defense: LoRA-Based Adversarial Fine-Tuning

To mitigate AdvJudge-Zero vulnerability, targeted adversarial training was applied using LoRA adapters on judge models. The defense routine consists of fine-tuning on a balanced corpus (50% correct, 50% incorrect “No” examples now augmented with random adversarial control-token suffixes). Hyperparameters included LoRA rank $r=4$, $\alpha=16$, dropout 0.05, and a single epoch.

This approach restored the Omni-Judge model from base FPRs >95% down to <6% on all benchmarks while maintaining or improving true positive rates (TPR ≈ 100%). This finding suggests that diverse, data-efficient control-token stress tests constitute an effective and scalable adversarial training signal, hardening the decision boundary without substantial loss of evaluation quality.

6. Practical Recommendations and Broader Implications

• Integrate flip-aware diagnostics in post-training reward pipelines (RLHF/DPO/RLAIF), e.g., flagging abrupt verdict logit-gap inversions or filtering high-risk suffixes.
• Use discovered AdvJudge-Zero token sets as stress-test and guardrail lists, for both reward debiasing and inference-time detection.
• Employ adversarial training—preferably with frequent re-seeding—to improve robustness as models and threat models evolve.
• Explore hybrid objectives combining adversarial token training, contrastive preference learning, and representation regularization to thicken the verdict head decision boundary.

The AdvJudge-Zero methodology reveals an inherent geometric exposure in shallow, linear verdict heads to aligned low-perplexity perturbations, establishing a baseline for reward hacking defenses and judge model security at scale (Li et al., 19 Dec 2025).