Rubric-Conditioned Self-Distillation

• The paper introduces rubric-conditioned self-distillation as a novel post-training paradigm that converts coarse outcomes into dense, criterion-level signals for improved policy performance.
• It employs both response-level and token-level strategies—such as ROPD, RCSD, and RGSD—to streamline teacher-student learning dynamics and mitigate supervision issues.
• Empirical evaluations demonstrate significant gains in accuracy and efficiency, with reduced sample and time requirements across benchmarks like AIME, GPQA, and HealthBench.

Searching arXiv for papers on rubric-conditioned self-distillation and closely related methods. Search query: "Rubric-Conditioned Self-Distillation arXiv 2026" Rubric-conditioned self-distillation is a post-training paradigm in which a model is optimized on its own on-policy trajectories while a teacher—often the same model under privileged conditioning—uses rubrics, rubric-like criteria, or structured diagnostic feedback to convert coarse outcome information into denser learning signals. In the 2026 literature, the term spans several closely related regimes: response-level rubric rewards induced from teacher–student contrasts, token-level distillation from a rubric-conditioned self-teacher, and long-horizon variants in which hindsight hints, stage rubrics, or reflective critiques act as localized rubric signals (Fang et al., 8 May 2026, Gu et al., 17 Jun 2026, Rezaei et al., 10 Jun 2026). The common premise is that rubrics decompose quality into explicit criteria that are more aligned with correctness, process quality, or task structure than scalar rewards or single reference rationales.

1. Conceptual scope and historical placement

Rubric-conditioned self-distillation emerged at the intersection of on-policy distillation, reinforcement learning with verifiable rewards, and chain-of-thought distillation. Traditional on-policy distillation supplies dense supervision through teacher logits, but this generally requires white-box teacher access and aligned tokenizers. "Rubric-based On-policy Distillation" introduces ROPD precisely to remove this restriction: it replaces teacher logits with structured, prompt-specific semantic rubrics induced from teacher–student contrasts, thereby retaining on-policy optimization while requiring only teacher text outputs (Fang et al., 8 May 2026). In parallel, "Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation" frames rubrics as criterion-level feedback that can condition a privileged teacher and yield token-level guidance on the student’s own sampled trajectories, rather than forcing imitation of a single chain-of-thought rationale (Gu et al., 17 Jun 2026). "Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers" pushes the idea further by removing the training-time rubric verifier entirely: the rubric-conditioned base policy itself becomes the teacher for an unconditioned student (Rezaei et al., 10 Jun 2026).

This literature is partly motivated by a shared diagnosis of earlier methods. Off-policy rationale distillation can inherit noisy, incomplete, or path-specific rationales and suffers from student–teacher prefix mismatch. Scalar-reward RL compresses multi-criterion judgments into a single end-of-trajectory signal, creating a credit-assignment bottleneck. Several contemporaneous papers generalize the rubric idea beyond explicit checklists. In "Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information," the privileged context $c$ may include a "brief grading rubric," but the core issue is that self-distillation toward a teacher conditioned on oracle-like context can suppress deliberation tokens in math reasoning (Shen et al., 12 May 2026). In "HINT-SD," localized hindsight hints are described as acting like step-specific rubrics, while "AMR-SD" compresses verifier outcomes, peer rollouts, or reference feedback into Socratic hints and critiques that function as rubric-like guidance (Yeo et al., 18 May 2026, Wei et al., 18 May 2026).

A central consequence is that rubric-conditioned self-distillation is not a single objective but a family of post-training designs unified by privileged criterion-level supervision. Some methods use rubrics to score entire responses, some use them to condition a teacher distribution token by token, and some use them to structure stagewise or turn-local supervision in long-horizon agents.

2. Formalization and objectives

A common setup uses an input prompt $x$, a student policy $\pi_\theta(y \mid x)$, and a teacher policy that has access either to external teacher responses or to privileged context unavailable at inference time. In ROPD, the teacher is text-only: for each prompt $x$, teacher responses $\mathcal{Y}_x^T = \{y_j^T\}_{j=1}^m$ and student rollouts $\mathcal{Y}_x^S = \{y_i^S\}_{i=1}^n$ are sampled, and a rubric $R_x = \{c_k\}_{k=1}^K$ is induced, where each criterion is $c_k = (\rho_k, w_k)$ with textual criterion $\rho_k$ and weight $w_k > 0$. A verifier returns binary judgments $x$0, and the response-level score is the weighted pass rate

$x$1

ROPD then uses GRPO-style on-policy optimization with group-relative advantage

$x$2

followed by a clipped objective with KL regularization to a fixed reference policy $x$3 (Fang et al., 8 May 2026).

Token-level formulations condition the teacher directly on rubrics. In RCSD, the framework is explicitly two-stage. Stage I trains a rubric generator with

$x$4

where the teacher has privileged access to a reference answer $x$5. Stage II trains a reasoner using

$x$6

The teacher is rubric-conditioned, the student is not, and forward KL is emphasized because it preserves support over multiple valid derivations that satisfy the rubric (Gu et al., 17 Jun 2026).

RGSD keeps the same high-level teacher–student asymmetry but removes verifier calls entirely. With on-policy rollouts $x$7, it minimizes a token-wise divergence between $x$8 and $x$9. The reported implementation uses clipped Jensen–Shannon divergence with $\pi_\theta(y \mid x)$0 and per-token clip $\pi_\theta(y \mid x)$1, masking > ... spans for reasoning-trace models to avoid leakage of rubric references into the student (Rezaei et al., 10 Jun 2026).

A different mathematical diagnosis appears in AntiSD. There the student is $\pi_\theta(y \mid x)$2, the self-teacher is $\pi_\theta(y \mid x)$3, and the teacher–student log-ratio

$\pi_\theta(y \mid x)$4

is identified as conditional PMI. This shows why ordinary self-distillation can reinforce shortcut tokens and suppress deliberation tokens when $\pi_\theta(y \mid x)$5 behaves like an oracle. AntiSD therefore ascends a divergence rather than descending it, using a JSD-based, entropy-gated token-level signal (Shen et al., 12 May 2026).

3. Architectural patterns and method families

The literature clusters into a small number of recurring designs.

Framework family	Teacher or rubric role	Training signal
ROPD	Prompt-specific rubrics induced from teacher–student contrasts	Response-level weighted pass-rate reward with GRPO
RCSD / RGSD	Teacher conditioned on rubric text	Per-token KL or clipped JSD on student rollouts
AntiSD / SGSD	Privileged context validated or gated before distillation	Polarity-aware token-level weighting
HINT-SD / AMR-SD / RubricEM	Hints, reflections, or stage rubrics act as localized rubric signals	Targeted, stagewise, or modulation-based token updates

ROPD is the clearest response-level formulation. Its Rubricator induces criteria shared across the student rollouts for a prompt, with taxonomy over Task Completion, Observable Quality, and General Reasoning; the verifier then scores each rollout blindly against these criteria, including teacher responses in the pool to calibrate difficulty (Fang et al., 8 May 2026). This design is explicitly black-box compatible and does not require token alignment.

RCSD and RGSD are token-level self-distillation frameworks. In both, the student rolls out under the plain prompt while the teacher sees the rubric. RCSD additionally learns a rubric generator, so rubrics themselves become on-policy outputs of a first-stage student model, whereas RGSD treats provided rubrics as direct teacher-side privileged context and focuses on eliminating the verifier from the loop (Gu et al., 17 Jun 2026, Rezaei et al., 10 Jun 2026).

Other methods adapt the rubric idea to noisier privileged information. SGSD retrieves skill–mistake pairs, forms a multi-teacher pool, and validates teacher polarity against outcome $\pi_\theta(y \mid x)$6; its details explicitly describe how the same machinery can be adapted to rubric-conditioned self-distillation, including rubric–counterrubric contexts and confidence-weighted gating (Huang et al., 27 May 2026). HINT-SD narrows supervision to a sparse set of failure-relevant turns selected by full-trajectory hindsight, where the injected hint acts as a localized rubric for that step (Yeo et al., 18 May 2026). AMR-SD interposes a reflection bottleneck so that the teacher never conditions on raw oracle traces; instead it conditions on concise hints or critiques structured by rubric-like dimensions such as Format Check, Reference Anchor, The Fault Line, and Root Cause Analysis (Wei et al., 18 May 2026). RubricEM extends the idea from short reasoning traces to deep research agents by making rubrics the shared interface across planning, evidence gathering, review, synthesis, judging, and reflection memory (Li et al., 11 May 2026).

4. Empirical findings

The strongest response-level evidence comes from ROPD. In black-box experiments with GPT-5.2-chat as teacher and Qwen3-4B as student, ROPD ranked first across all reported non-thinking and thinking tracks, reaching, for example, 65.02 on AIME24 non-thinking and 75.41 on AIME24 thinking. On AIME25 thinking, ROPD reached 68.75, surpassing the teacher’s 67.08. In white-box experiments with Qwen3-30B-A3B as teacher and Qwen3-4B as student, ROPD outperformed the logit-based baselines LOPD and ExOPD; on AIME24 non-thinking, the reported scores were SFT 26.69, LOPD 47.92, ExOPD 50.66, and ROPD 63.33. Average across four math benchmarks was 45.87 for ROPD, compared with 35.25 for ExOPD and 32.82 for LOPD. The paper also reports that on AIME24 non-thinking, ROPD reached 48.3% with approximately $\pi_\theta(y \mid x)$7k samples versus $\pi_\theta(y \mid x)$8k for LOPD, a $\pi_\theta(y \mid x)$9 sample reduction, alongside a wall-clock speedup of about $x$0 to the same threshold, $x$1h versus $x$2h (Fang et al., 8 May 2026).

Token-level rubric-conditioned self-distillation also showed competitive or superior results. RCSD, evaluated on GPQA-Diamond, SciBench, PIQA, RaR-Science, ResearchQA, and RubricHub with a Qwen3-8B backbone, achieved an average of 70.6 versus 69.2 for GRPO and 69.7 for OPSD. The paper states that RCSD surpasses GRPO by 1.4 points and OPSD by 0.9 points on average; forward KL was the best overall loss variant, though JSD and reverse KL won on individual benchmarks (Gu et al., 17 Jun 2026). RGSD, evaluated across Qwen-2.5 and Qwen3-Thinking models on medical and science domains, reported rubric-satisfaction gains comparable to judge-based GRPO while using one on-policy rollout per prompt and no training-time verifier calls: medical improved by $x$3 percentage points over base versus $x$4 for GRPO, and science by $x$5 versus $x$6. On Qwen-2.5-7B HealthBench-300, the factual-claim audit showed claims per answer rising from 15.8 to 38.4 under GRPO and to 25.8 under RGSD, while false-claim rate rose from 30.5% to 45.1% under GRPO and to 35.1% under RGSD (Rezaei et al., 10 Jun 2026).

Methods that address failure modes of privileged-context distillation also reported substantial gains. AntiSD reached the GRPO baseline’s accuracy in $x$7 to $x$8 fewer training steps and improved final accuracy by up to 11.5 points across five models from 4B to 30B parameters; for Qwen3-4B-IT-2507, the reported average rose from 51.3 under GRPO to 62.8 under AntiSD (Shen et al., 12 May 2026). SGSD, which treats skill-conditioned self-distillation as teacher hypothesis validation, reported that on Qwen3-1.7B the average score increased from 37.4 to 43.7, outperforming GRPO by 6.2% and OPSD by 1.7% on AIME24, AIME25, and HMMT25 (Huang et al., 27 May 2026). HINT-SD improved over the dense per-turn feedback baseline by up to 18.80 percent while reducing time per training step from 84.76 s to 37.45 s and peak GPU memory from 126 GB to 85 GB (Yeo et al., 18 May 2026). RubricEM-8B improved from 49.2 average under SFT to 55.5 after RL across HealthBench, ResearchQA, DRB, and ResearchRubrics, while AMR-SD reported a full-model score of 74.3 on SciKnowEval mixed, versus 70.5 without Meta-Reflection and 70.6 without Temporal Annealing (Li et al., 11 May 2026, Wei et al., 18 May 2026).

5. Mechanistic interpretation and recurrent misconceptions

A first recurrent claim is that rubrics align more directly with correctness or task success than raw token similarity. ROPD reports that on AIME24, rubric reward achieved $x$9 with respect to ground-truth correctness, whereas teacher log-likelihood was reported as misaligned at about $\mathcal{Y}_x^T = \{y_j^T\}_{j=1}^m$0 AUC in the ROPD family, and top-24 token overlap was near random at about $\mathcal{Y}_x^T = \{y_j^T\}_{j=1}^m$1 AUC. The paper interprets this as evidence that rubric rewards suppress stochastic surface-form variation and isolate task-level reasoning principles (Fang et al., 8 May 2026). RCSD makes a related argument from a different angle: a single reference rationale may be noisy or path-specific, whereas rubrics specify what a strong response should satisfy, not a single trajectory the student must reproduce (Gu et al., 17 Jun 2026).

A second misconception is that rubric-conditioned self-distillation is always a standard KL descent toward a privileged teacher. AntiSD shows that this can fail in math reasoning. Because $\mathcal{Y}_x^T = \{y_j^T\}_{j=1}^m$2 equals conditional PMI, privileged solution-like context raises probability on structural connectives and verifiable claims while lowering it on deliberation tokens such as “Wait,” “Let,” “Maybe,” and “Alternatively.” Standard self-distillation therefore tilts learning toward “solution-shaped scaffolding” and away from multi-step search; AntiSD reverses the sign, uses JSD ascent, and disables the term when teacher entropy collapses (Shen et al., 12 May 2026). This does not negate rubric-conditioned self-distillation, but it shows that the form of privileged conditioning and the direction of the update are decisive.

A third misconception is that rubrics matter only at final-answer evaluation time. RubricEM explicitly argues that rubrics should not serve merely as final-answer evaluators but as the shared interface that structures policy execution, judge feedback, and agent memory in deep research agents (Li et al., 11 May 2026). HINT-SD and AMR-SD make the same point in more localized form: hints and critiques are valuable precisely when inserted at the action span or token region where the correction is actionable, not when broadcast over an entire trajectory (Yeo et al., 18 May 2026, Wei et al., 18 May 2026).

6. Limitations, failure modes, and open directions

The literature repeatedly identifies rubric quality as a central bottleneck. ROPD reports sensitivity to rubric induction quality: single-teacher rubrics collapse into path-matching, poor weighting can reduce discriminative power, and early training can produce superficial strategies that score well without substantive correctness (Fang et al., 8 May 2026). RCSD similarly notes that learned rubrics can be verbose, overlapping, or generic, although degraded rubrics still improved over the base model in its reported ablations (Gu et al., 17 Jun 2026). RGSD adds that long, specific rubrics provide richer guidance but can increase thinking-trace leakage, motivating explicit masking of <think> spans (Rezaei et al., 10 Jun 2026).

Another recurring limitation is domain scope. Several papers emphasize formal reasoning benchmarks—math, science, medicine, or programmatic environments—and explicitly leave subjective or creative tasks open. ROPD states that generalization to subjective or creative tasks remains to be established (Fang et al., 8 May 2026). SGSD notes that its current formulation targets mathematical reasoning with automatic verification and would need a calibrated judge or outcome signal for open-ended tasks (Huang et al., 27 May 2026). RubricEM demonstrates that rubric-guided methods can move beyond exact verifiability to long-form research, but it also reports risks from judge drift, misguiding rubrics, distillation overfitting in reflections, and infrastructure staleness in asynchronous pipelines (Li et al., 11 May 2026).

Open directions in the papers converge on three themes. One is richer rubric dynamics: evolving rubrics that co-adapt with the student, online rubric or bank maintenance, and calibration of rubric weights without supervision (Fang et al., 8 May 2026, Huang et al., 27 May 2026). A second is hybridization: combining rubric-conditioned dense guidance with occasional verifier calls or with process and outcome verification where available (Rezaei et al., 10 Jun 2026, Gu et al., 17 Jun 2026). A third is broader task generalization: subjective and creative generation, multimodal tasks, larger code suites, multi-turn agents, and heterogeneous teacher–student families (Shen et al., 12 May 2026, Yeo et al., 18 May 2026). Taken together, these directions suggest that rubric-conditioned self-distillation is best understood not as a closed algorithmic template but as a general strategy for turning structured evaluative criteria into on-policy learning signals while controlling leakage, over-conditioning, and verifier dependence.