Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers

Abstract: Rubrics have emerged as an alternative to RLVR in open-ended domains where a single ground-truth final answer is not available. Existing rubric-based training methods rely on an LLM verifier that scores each rollout against rubrics. This introduces substantial training-time overhead, exposes optimization to verifier-specific biases, and reduces rubric feedback to a sparse end-of-trajectory signal. We propose Rubric-Guided Self-Distillation (RGSD), a verifier-free training method in which the base policy, conditioned on the rubric, serves as the teacher for the unconditioned student. RGSD distills the rubric-conditioned teacher distribution into the student token-by-token, replacing sparse trajectory-level rewards with dense per-token learning signals and removing the LLM judge from the training loop entirely. Across Qwen-2.5 (3B, 7B) and Qwen3-Thinking (4B, 8B) models on medical and science domains, RGSD achieves rubric satisfaction comparable to judge-based GRPO while using one on-policy rollout per prompt and no training-time verifier calls. Ablations show that raw rubrics provide a stronger teacher enrichment signal than self-generated reference responses, while a stronger GRPO judge can outperform RGSD in some settings, positioning RGSD as a complementary verifier-free alternative when verifier cost or reliability is the bottleneck.

• The paper presents RGSD, which replaces sparse, verifier-based rewards with dense token-level supervision in rubric-graded LLM training.
• It demonstrates significant computational efficiency and improved rubric satisfaction compared to GRPO across medical and science domains.
• Findings reveal that RGSD mitigates verbosity and false-claim issues while enabling scalable, bias-resistant post-training workflows.

Rubric-Guided Self-Distillation: Verifier-Free Post-Training for Rubric-Graded LLMs

Motivation and Background

Rubric-guided paradigms have become central to post-training in open-ended domains where programmatic reward functions are unavailable. An established approach, Group-Relative Policy Optimization (GRPO), leverages LLM verifiers to score student rollouts against task-specific rubrics, aggregating multi-criterion judgments into scalar rewards. While effective, this protocol incurs substantial computational overhead, exposes optimization to verifier biases, and reduces multi-criterion rubric feedback to a sparse end-of-trajectory signal, impairing credit assignment.

Rubric-Guided Self-Distillation (RGSD) (2606.12507) addresses these challenges by eliminating the LLM verifier from the training loop and replacing scalar trajectory-wide rewards with dense token-level supervision. RGSD exploits the empirical conditioning gap: a model’s rubric-conditioned responses exhibit substantial lifting ($+30$--$45$ percentage points) in rubric satisfaction relative to unconditioned outputs. By distilling this privileged rubric-conditioned distribution into an unconditioned student, RGSD internalizes rubric guidance without external grading, reframing rubrics from evaluative instruments into teacher context.

Figure 1: RGSD leverages rubric-conditioned teacher supervision, whereas GRPO relies on external LLM grading per rollout.

Methodology

RGSD instantiates two copies of a base model: the student (trainable, conditioned only on prompt) and the teacher (frozen, conditioned on both prompt and rubric). For each training instance, the student samples an on-policy rollout. The teacher, conditioned on the full rubric and the rollout prefix, provides token-level target distributions. The student minimizes a clipped Jensen–Shannon divergence between its distribution and the teacher’s at every token position, yielding dense per-token learning signals.

Formally, for dataset $\mathcal{D}$ of $(q, R)$ pairs, the student rollout $y \sim \pi_S(\cdot | q)$ is distillation-targeted against $\pi_T(\cdot | q, R, y_{<t})$. The loss function interpolates between forward and reverse KL ($\beta=0.5$), clipped to stabilize gradients. For models with reasoning traces (e.g., Qwen3-Thinking), tokens within > blocks are masked to avoid rubric leakage, and only answer tokens contribute to the loss.

This pipeline converts rubric feedback from sparse trajectory-wide grading (as in GRPO) to dense per-token supervision, while completely eliminating training-time verifier calls and the associated operational complexity.

Experimental Evaluation

RGSD and GRPO were evaluated on medical and science domains using four base models: Qwen-2.5-3B/7B-Instruct and Qwen3-4B/8B-Thinking. Training utilized RubricHub datasets, with evaluation performed on RubricHub validation splits, HealthBench (medical), and ResearchQA (science), using gpt-5.4 as the held-out rubric judge.

Results demonstrate that RGSD matches or exceeds GRPO on rubric satisfaction across both domains and all base models, with domain-averaged gains of $+6.1$pp (medical) and $+4.9$pp (science), compared to GRPO’s $+5.9$pp and $45$0pp respectively. RGSD reaches these improvements with significantly lower computational cost: one rollout per prompt versus sixteen for GRPO, and zero judge calls versus over 1M per domain for GRPO.

Figure 2: RGSD training dynamics show comparable or superior rubric satisfaction to GRPO, with substantially less verbosity on Qwen-2.5 models.

RGSD is consistently more concise. On Qwen-2.5 models, RGSD’s responses are $45$1--$45$2 shorter than GRPO’s, with no reduction in rubric satisfaction. GRPO optimization induces length inflation, often associated with reward hacking exploits—producing verbose responses with high claim density but elevated false-claim rates.

Figure 3: RGSD outperforms GRPO in rubric satisfaction on science domain; GRPO again induces longer responses without consistent score advantage.

Auxiliary audits reveal that GRPO not only increases response length but also false-claim rates: on HealthBench, GRPO’s false-claim fraction rises from $45$3 to $45$4, whereas RGSD grows more modestly ($45$5 to $45$6), despite comparable rubric satisfaction.

Ablations and Analysis

RGSD’s enrichment signal and the strength of GRPO’s judge were analyzed. Conditioning the teacher on raw rubrics provides a stronger signal than using self-generated reference responses, avoiding stylistic artifacts and ensuring comprehensive criterion coverage.

Figure 4: Raw rubrics as enrichment produce higher rubric satisfaction than reference responses; length inflation is suppressed regardless of enrichment.

GRPO’s performance improves with stronger judges (e.g., gpt-oss-120b), occasionally surpassing RGSD, but retains the computational costs and length inflation. Thus, RGSD is preferable when verifier calls are costly or unreliable, and the base model’s rubric-conditioning lift is substantial. The rubric-conditioning gap is a practical diagnostic for RGSD efficacy.

Figure 5: Stronger judge boosts GRPO’s rubric satisfaction but increases computational overhead and verbosity compared to RGSD.

For reasoning-trace models, masking <think> tokens is critical to prevent rubric leakage, as teacher traces often enumerate criteria explicitly when conditioned on rubrics, causing pathological distillation if unmasked.

Figure 6: For Qwen3-Thinking, masking reasoning trace tokens enhances rubric satisfaction and controls length.

Multi-seed analysis confirms robustness, with tight variance in rubric satisfaction across multiple runs.

Figure 7: Multi-seed envelope shows tight RGSD score variance, confirming reproducibility.

Generalization to OLMo and Mistral models demonstrates RGSD’s transferability beyond Qwen families.

Figure 8: RGSD improves rubric satisfaction across OLMo and Mistral bases, demonstrating method generality.

Implications and Future Directions

RGSD establishes rubric-conditioned self-distillation as a viable alternative to sparse, judge-driven RL in open-ended generative tasks. By reframing rubrics as privileged context for dense supervision, RGSD achieves quality parity with GRPO at an order-of-magnitude reduction in compute without inheriting judge biases. These characteristics make RGSD a compelling choice in settings where verifier resources are constrained or operationally infeasible.

Theoretical implications include the demonstration that dense, context-conditioned teacher signals can substitute for external scalar rewards, and that substantial conditioning gaps can be leveraged to bootstrap quality improvements in unconditioned students. Practically, RGSD enables scalable, efficient, and bias-resistant post-training workflows for LLMs in medical, scientific, and creative domains.

Future work may refine rubric-enrichment strategies, investigate mixed enrichment signals, or integrate dynamic rubric construction during training. The efficacy of RGSD hinges on the base model’s rubric-conditioning lift; further exploration into architectural and training factors influencing this gap is warranted. Comparison with advanced verifier-based RL methods, especially those with sophisticated credit assignment and strong judges, will further delineate RGSD’s operational regime.

Conclusion

Rubric-Guided Self-Distillation (RGSD) introduces a paradigm shift in post-training for rubric-graded LLMs, replacing verifier-driven sparse rewards with dense, context-conditioned supervision. RGSD achieves competitive rubric satisfaction on open-ended tasks, reduces computation, and mitigates verbosity drift and reward hacking observed in GRPO. The method is most effective when base models exhibit large rubric-conditioning lifts and verifier resources are bottlenecked, positioning RGSD as a practical and theoretically sound alternative to existing rubric-based RL approaches (2606.12507).