Verifier-Free Intrinsic Gradient-Norm Reward (VIGOR)

• The paper introduces VIGOR, where teacher-forced gradient norms serve as intrinsic rewards, eliminating the need for external verifiers.
• It employs rank-based normalization and a √T length correction to stabilize training and counteract length bias in gradient computations.
• Empirical results demonstrate substantial gains in math and code tasks, validating the effectiveness of the verifier-free reinforcement approach.

Searching arXiv for the specified papers and closely related work to ground the article in current literature. Verifier-free Intrinsic Gradient-Norm Reward (VIGOR) is a verifier-free reinforcement-learning method for LLMs in which the reward is derived from the model’s own teacher-forced negative log-likelihood gradient norm rather than from gold labels, execution tests, or any external verifier (Wen et al., 11 May 2026). For a prompt, the method samples a group of completions and assigns higher within-group rewards to outputs that induce smaller $\ell_2$ norms of the teacher-forced gradients under the current parameters, interpreting lower gradient norms as an intrinsic preference signal for policy optimization (Wen et al., 11 May 2026). In a broader conceptual sense, VIGOR also names a verifier-free intrinsic gradient-norm reward principle that is closely matched by earlier teacher–student automatic curriculum learning work in deep reinforcement learning, where a teacher is rewarded by the student’s gradient norms and no separate verifier model, oracle, or success classifier is used (Campbell et al., 2023).

1. Definition and conceptual position

VIGOR was introduced as a response to a specific limitation of Reinforcement Learning with Verifiable Rewards (RLVR): RLVR depends on gold labels or domain-specific verifiers, such as exact-match answer checking for math or unit tests for code, which limits scalability to tasks where such verifiers do not exist, are expensive, or are unreliable (Wen et al., 11 May 2026). The method is also positioned against Reinforcement Learning from Internal Feedback (RLIF), which uses internal signals such as confidence, entropy, or likelihood-based pseudo-rewards, but is described as vulnerable to brittleness, exploitability during training, instability, or late-stage performance collapse (Wen et al., 11 May 2026).

Within that framing, VIGOR occupies an intermediate position. It is verifier-free, like RLIF, but its intrinsic signal is not confidence or entropy. Instead, it is the gradient norm of the model’s own teacher-forced loss (Wen et al., 11 May 2026). The stated motivation is an optimization view: if a completion yields a small gradient norm under the current parameters, then the current model is already relatively “comfortable” with it, suggesting a flatter local loss surface and smoother updates (Wen et al., 11 May 2026).

A common misconception is that VIGOR is simply a confidence-based or likelihood-based heuristic under a new name. The defining distinction in the data is that the reward is constructed in parameter space from teacher-forced negative log-likelihood gradients, and the paper emphasizes that this aggregates information across the whole model rather than relying only on next-token distribution statistics (Wen et al., 11 May 2026). Another misconception is that verifier-free methods necessarily require an auxiliary evaluator; in VIGOR, no such external checker is used.

2. Reward construction from teacher-forced gradient norms

For each prompt $x$, VIGOR samples a group of completions from the current policy,

$$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$

with $G=8$ in the experiments (Wen et al., 11 May 2026). For a sampled completion $y=(y_1,\dots,y_T)$ of length $T=|y|$, the method computes a teacher-forced negative log-likelihood averaged over tokens:

$$\ell_{\text{mean}}(x,y)=\frac{1}{T}\sum_{t=1}^{T}\ell_t(x,y),$$

where

$\ell_t(x,y)=-\log \pi_\theta(y_t \mid x, y_{<t}).$

It then computes the gradient of this scalar loss with respect to the current model parameters,

$$\mathbf{g}(x,y)=\nabla_\theta \ell_{\text{mean}}(x,y),$$

and uses the Euclidean norm $\|\mathbf{g}(x,y)\|_2$ as the intrinsic signal (Wen et al., 11 May 2026).

The reward direction is inverted relative to the raw norm: smaller gradient norms receive higher reward. The raw intrinsic score is

$x$0

The negative sign turns minimization of gradient norm into a reward-maximization signal, while the $x$1 factor corrects a systematic length bias induced by averaging token-level losses (Wen et al., 11 May 2026). The paper argues that, loosely, one expects

$x$2

so longer completions can appear artificially better if raw gradient norms are used directly (Wen et al., 11 May 2026).

The corrected scores are not used as final rewards without further transformation. For each prompt, completions are ranked by their corrected scores, and the rank-normalized reward is defined as

$x$3

where $x$4 and larger rank means better completion; thus the worst completion maps to $x$5 and the best completion maps to $x$6 (Wen et al., 11 May 2026). This produces a bounded within-group reward.

A further technical point is that the gradient used for reward construction is detached from the computation graph and is not used for backpropagation through the reward (Wen et al., 11 May 2026). This stop-gradient choice is presented as a stabilization device that preserves a first-order update.

3. Integration into GRPO-style reinforcement learning

VIGOR is implemented inside a GRPO-style optimization framework (Wen et al., 11 May 2026). For each prompt, the algorithmic sequence is: sample $x$7 completions from $x$8; compute the teacher-forced mean NLL loss for each completion; compute $x$9 and its $$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$0 norm; apply the $$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$1 correction; rank the completions within the prompt and map ranks to $$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$2; standardize rewards to obtain advantages; and update the policy with a PPO-style clipped objective and a KL penalty to a reference model (Wen et al., 11 May 2026).

The group-relative standardized advantage is

$$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$3

where $$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$4 is the mean reward in the group (Wen et al., 11 May 2026). The policy objective is written as

$$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$5

with token ratio

$$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$6

and clipped ratio

$$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$7

This makes the update depend on the relative ordering of completions for the same prompt rather than on absolute reward scale across prompts (Wen et al., 11 May 2026).

The paper highlights three stabilization mechanisms: $$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$8 length correction, rank-based shaping, and stop-gradient through reward computation (Wen et al., 11 May 2026). It also reports an LM-head-only gradient-norm variant that computes the reward using only the output head parameters, reducing memory and wall-clock cost while preserving much of the performance (Wen et al., 11 May 2026). Training uses group size $$y_i \sim \pi_\theta(\cdot \mid x), \qquad i=1,\dots,G,$$9, a KL penalty to the reference policy, and sequential gradient computation for each completion to manage memory (Wen et al., 11 May 2026).

4. Antecedents in automatic curriculum learning

Although the term VIGOR was introduced for LLM post-training in 2026, an earlier deep-reinforcement-learning paper, “Automatic Curriculum Learning with Gradient Reward Signals,” proposed a teacher–student ACL framework in which the teacher is rewarded by the student’s gradient norms rather than by an external verifier or hand-designed difficulty score (Campbell et al., 2023). The paper does not frame itself as “verifier-free VIGOR” explicitly, but the data describes it as functionally very close because there is no separate verifier model, oracle, or success classifier providing reward to the teacher (Campbell et al., 2023).

In that ACL setting, the student is a standard RL agent trained on tasks or initial states provided by the teacher, the teacher is another RL policy that learns to select better tasks or initial states for the student, and the teacher’s reward is derived from the norm of the student’s gradient updates during training (Campbell et al., 2023). The student update is written as

$G=8$0

where $G=8$1 is the dataset or task assignment generated by the teacher policy $G=8$2 (Campbell et al., 2023).

The paper studies two gradient-based learning-progress measures:

$G=8$3

and

$G=8$4

After collecting $G=8$5 student-learning episodes, the teacher dataset is formed as

$G=8$6

followed by

$G=8$7

The algorithmic loop is summarized as

$G=8$8

$G=8$9

In this sense, the paper provides a direct precursor to the verifier-free intrinsic gradient-norm reward idea, but in ACL rather than LLM post-training (Campbell et al., 2023).

This historical connection is conceptually important because it shows that “gradient norm as intrinsic learning-progress proxy” predates the LLM formulation. A plausible implication is that VIGOR is best understood not as an isolated reward-engineering trick, but as one instance of a broader design pattern in which optimization dynamics themselves become the supervisory signal.

5. Empirical behavior and reported results

The LLM paper evaluates VIGOR using Qwen2.5-3B-Base and Qwen2.5-7B-Base, with MATH for math post-training and CodeContests for a lightweight code-training setting (Wen et al., 11 May 2026). Benchmarks include GSM8K, MATH500, AMC, LiveCodeBench-v6, CRUX, MMLU-Pro, and IFEval (Wen et al., 11 May 2026).

For Qwen2.5-7B-Base trained on MATH, the reported results are: math average improves from 42.58 to 69.77; code average improves from 9.69 to 40.42; MMLU-Pro remains strong at 43.09; and IFEval improves to 37.03 (Wen et al., 11 May 2026). The paper highlights particularly large gains on GSM8K (+45.64) and AMC (+22.74) (Wen et al., 11 May 2026). For Qwen2.5-3B-Base trained on MATH, math average goes from 48.34 to 59.14 and code average from 16.98 to 27.95 (Wen et al., 11 May 2026). On CodeContests with Qwen2.5-3B, code average improves from 16.98 to 23.64 and math average from 48.34 to 56.57 (Wen et al., 11 May 2026). The paper characterizes this as cross-domain transfer: even when trained only on math, VIGOR improves code benchmarks substantially (Wen et al., 11 May 2026).

The paper compares against three baselines: the base model before RL, GT-Reward using GRPO with exact-match or verifier-based reward, and INTUITOR as a verifier-free RLIF baseline using self-certainty or likelihood-based internal feedback (Wen et al., 11 May 2026). The stated comparative finding is that VIGOR outperforms INTUITOR on math and is usually better on code and robustness, while being competitive with, or better than, verifiable reward training depending on task and model (Wen et al., 11 May 2026). It also emphasizes more stable training dynamics than INTUITOR, which is described as showing late-stage regression (Wen et al., 11 May 2026).

The earlier ACL paper reports related evidence in a different domain. Its environments are PointMaze_OpenDense-v3, AntMaze_OpenDense-v4, and AdroitHandRelocate-v1, with total steps 1,000,000, batch size 256, hidden layer size 256, number of layers 3, discount factor 0.99, and temperature 0.05 (Campbell et al., 2023). In PointMaze, both train and eval returns converge faster with a teacher than without one, and the average gradient per timestep teacher appears slightly better than the overall episode gradient teacher (Campbell et al., 2023). In AdroitHandRelocate, the average gradient per timestep teacher outperforms both the overall episode gradient teacher and the no-teacher baseline, while in AntMaze there is no significant benefit from teaching and teacher and no-teacher agents perform similarly after 1M steps (Campbell et al., 2023). The paper also reports that in AdroitHandRelocate the teacher gradually increases goal distance over time, suggesting a progressively harder curriculum (Campbell et al., 2023).

Taken together, these findings support two distinct but related empirical claims present in the data: first, gradient-norm-derived intrinsic rewards can be effective without external verifiers; second, the structure of the gradient-based reward matters, since raw or differently aggregated gradient signals can behave differently across tasks and domains.

6. Ablations, limitations, and interpretive boundaries

The strongest ablation in the LLM paper concerns the $y=(y_1,\dots,y_T)$0 correction. Removing it causes severe length hacking and performance collapse, especially on Qwen2.5-3B: GSM8K and AMC can drop to near zero, code transfer collapses, and output length inflates dramatically (Wen et al., 11 May 2026). This establishes length bias as a real failure mode rather than a minor implementation detail. Removing rank shaping slightly hurts math and code performance on 3B and hurts more substantially on 7B, including a large regression on MMLU-Pro (Wen et al., 11 May 2026). The paper also tests INTUITOR plus rank normalization, which improves INTUITOR somewhat but still remains below VIGOR, supporting the claim that the gradient-norm signal itself matters and that the gain is not only due to better normalization (Wen et al., 11 May 2026).

Another analysis reported in the data is rank–accuracy monotonicity: when the $y=(y_1,\dots,y_T)$1 completions are ranked by $y=(y_1,\dots,y_T)$2, better-ranked completions are more accurate, showing a monotonic relationship between the intrinsic signal and correctness (Wen et al., 11 May 2026). The paper further reports that, compared with INTUITOR, VIGOR avoids late-stage collapse, keeps top-25% reward accuracy more stable, does not drive the same explosive length growth, and produces outputs that are shorter than INTUITOR’s while not showing excessive repetition (Wen et al., 11 May 2026).

The limitations are explicit. Applicability to open-ended generation is unclear; the experiments focus on verifiable reasoning tasks such as math and code (Wen et al., 11 May 2026). Computational overhead is non-trivial because VIGOR requires per-sample gradient computation rather than forward-only intrinsic rewards (Wen et al., 11 May 2026). Gradient norm remains a proxy objective and may not always align with downstream quality, and reward scale instability across prompts is one reason rank shaping is needed (Wen et al., 11 May 2026). On competitive programming-like benchmarks, VIGOR sometimes underperforms INTUITOR, which the authors attribute to algorithmic correctness hinging on discrete choices not well captured by smoothness or gradient norms (Wen et al., 11 May 2026).

The ACL precursor states parallel caveats. It is described as an early prototype, does not provide a rigorous theoretical analysis of gradient norms as intrinsic rewards, controls initial states only rather than richer task generation, and does not fully specify all implementation details of how gradients are aggregated into the teacher’s observation or reward beyond the two core formulas (Campbell et al., 2023). Performance is environment-dependent, with AntMaze showing no clear gains, and smoothing is said to be important because many raw curves are noisy and the smoothed results are more interpretable (Campbell et al., 2023).

These limitations delimit the current scope of verifier-free intrinsic gradient-norm reward methods. The data supports the view that such methods can serve as practical RL signals without external verifiers, but does not support a claim that gradient norms are universally reliable proxies for correctness or usefulness. A plausible implication is that VIGOR is best treated as a parameter-space intrinsic reward mechanism whose effectiveness depends on calibration devices such as length correction, rank shaping, and task-domain fit.