Outcome-Guided Logit Steering (OGLS-SD)

• The paper presents OGLS-SD, a self-distillation framework that adjusts teacher logits using outcome rewards to overcome miscalibrated token-level supervision.
• It contrasts correct and incorrect on-policy trajectories by computing a logit-space steering direction, thereby mitigating teacher–student mismatch from privileged conditioning.
• Empirical results on Qwen3 models reveal that OGLS-SD yields smoother training dynamics and improved accuracy over SFT, GRPO, and standard OPSD.

Outcome-Guided Logit Steering (OGLS-SD) is an on-policy self-distillation framework for LLM reasoning in which a student model learns from a calibration-adjusted teacher distribution rather than from raw privileged teacher outputs. It was introduced in “OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning” as a response to a specific failure mode of on-policy self-distillation: self-reflected teacher responses conditioned on privileged solutions can be shifted by reflection-induced bias and response templates, producing miscalibrated token-level supervision. OGLS-SD addresses this by using verifiable outcome rewards to contrast successful and failed on-policy trajectories, constructing a steering direction in logit space that modifies a non-privileged anchor teacher, and then distilling the student toward that steered teacher only on incorrect rollouts. On Qwen3-1.7B and Qwen3-4B, the method improves over SFT, GRPO, and standard OPSD on AIME 2024 and AIME 2025, while also exhibiting smoother training and better preservation of epistemic reflection (Yang et al., 12 May 2026).

1. On-policy self-distillation as the immediate context

OGLS-SD is defined within on-policy self-distillation (OPSD), a post-training paradigm in which a single LLM acts as both student and teacher. For each problem, the student samples an on-policy trajectory

$y \sim \pi_{\theta}(\cdot \mid x),$

while the teacher is the same architecture conditioned on privileged information $s$, typically a reference solution or expert chain-of-thought available only during training. Standard OPSD distills token distributions along the student’s own trajectory by minimizing

$$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$

The student therefore learns from dense token-level supervision rather than sparse sequence-level rewards (Yang et al., 12 May 2026).

This setup uses what the paper calls privileged teacher distributions,

$\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$

and self-reflected teacher responses generated from prompts of the form “after reading the reference solution, derive the answer in your own words…”. In this formulation, the teacher has access to solution information that the deployed student will never observe. The promise of OPSD is that privileged conditioning can provide a richer supervisory signal than outcome-only reinforcement-style training, but that promise depends on the teacher distribution being a suitable target for a non-privileged student.

A useful implication is that OGLS-SD is not a generic decoding-time control method. It is specifically a training-time distillation framework built around self-generated on-policy rollouts, privileged teacher conditioning, and KL-based supervision. This distinguishes it from steering schemes whose primary role is inference-time attribute control.

2. Teacher–student mismatch and reflection-induced bias

The central problem OGLS-SD addresses is teacher–student mismatch in OPSD. Even when teacher and student share the same base model, the teacher’s distribution differs because it is conditioned on privileged solution information and a reflection template, whereas the student must solve from scratch. The ideal target inside the non-privileged student class is the marginal privileged teacher

$$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$

but standard OPSD instead uses one specific solution and one prompt template (Yang et al., 12 May 2026).

The paper identifies two sources of distortion. The first is solution-specific path bias: conditioning on one reference solution concentrates the teacher around a particular reasoning pattern, narrows the distribution, and can overwrite alternative valid strategies. The second is reflection-induced bias and overconfidence from prompting. Because the teacher is prompted after seeing a correct solution and instructed to “truly understand it” and “derive the same final answer,” its next-token distributions become sharper, exploratory or epistemic tokens are suppressed, and confidence shifts survive even after averaging over multiple privileged prompts. The result is miscalibrated token-level supervision: the student is trained to imitate a mode that is systematically “too sure” relative to its test-time operating regime (Yang et al., 12 May 2026).

Several diagnostics in the paper make this mismatch concrete. In a toy digits $0$–$9$ experiment, even a mixture over privileged prompts remains far from the non-privileged distribution, with total variation distance $> 0.5$. In a combinatorial game case study, OPSD and OGLS-SD both infer the correct modular pattern and intermediate counts, but OPSD overconfidently maps the count to the wrong player, whereas OGLS-SD revisits the Alice/Bob role assignment and reaches the correct final answer. A further indicator is the frequency of epistemic markers such as “wait”, “hold on”, and “let me rethink”: OPSD suppresses them early in training and only partially recovers, while OGLS-SD increases them throughout training.

A common misconception is that averaging privileged teacher outputs should be sufficient to resolve mismatch. The paper argues against this directly: prompt-induced marginalization does not recover the non-privileged distribution because the privileged prompt itself places the model in a different behavioral regime.

3. The OGLS-SD mechanism

OGLS-SD combines verifiable outcome rewards with logit-space steering. For each input $x_i$, the current student samples $G$ rollouts,

$s$0

which are then scored by a verifier $s$1. This partitions the on-policy trajectories into

$s$2

Outcome correctness is sequence-level, but OGLS-SD turns it into dense token-level guidance by constructing a contrastive steering direction in logit space (Yang et al., 12 May 2026).

The framework begins with a non-privileged anchor teacher,

$s$3

where $s$4 is kept fixed as the original base model. Positive and negative privileged guidance are then formed by averaging teacher logits over correct and incorrect pools: $s$5 with

$s$6

The steering direction is

$s$7

and the steered teacher logits are

$s$8

The student is trained by token-wise KL against this steered teacher,

$s$9

but only for incorrect rollouts: $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$0 This selective application is intended to reshape bad trajectories without perturbing already correct behavior (Yang et al., 12 May 2026).

The logic of the contrast is central. Positive and negative privileged teachers share the same reflection template and solution-aware prompt regime, so their difference attenuates shared overconfidence and template artifacts while preserving outcome-discriminative components. This is why the method can use $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$1 without simply amplifying privileged bias.

In the broader steering literature, this is a distinctive instantiation of logit steering. It differs from training-free logit-level interventions that modify the final pre-softmax logits at each decoding step with a constant positive logit bias $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$2 on a favored token set $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$3, and it also differs from annotator-conditioned class-logit calibration where a lightweight probe directly adjusts three-way ideological logits (An et al., 16 Jan 2026).

4. Training procedure, implementation, and operating conditions

The training loop is fully on-policy. For each sampled example $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$4, the student produces $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$5 rollouts using a vLLM backend with temperature $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$6. Each rollout is verified, correct and incorrect pools are formed, mixed positive and negative logits are computed, the non-privileged anchor teacher is evaluated on the student prefixes, and the student is updated by minimizing the steering loss on incorrect trajectories. If either guidance pool is empty, the rollout is skipped because no contrast is possible (Yang et al., 12 May 2026).

The reported implementation uses Qwen3-1.7B and Qwen3-4B as base models, OpenThoughts as the mathematical reasoning training corpus, and AIME 2024 plus AIME 2025 as evaluation benchmarks. Fine-tuning is performed with LoRA, using learning rate $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$7, rank $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$8, and scaling $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).$$9. The maximum training generation length is 8192 tokens, but distillation uses the first 1024 tokens. The steering coefficient $\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$0 is ramped up during training to a maximum of $\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$1, and symmetric pointwise KL clipping at $\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$2 is applied to avoid outlier gradients. The teacher is fixed as the original base model throughout training for stability (Yang et al., 12 May 2026).

The framework presupposes verifiable tasks. The paper lists math problems with exact numeric answers, code generation with test suites, logic puzzles with deterministic checkers, and structured QA with deterministic answer sets as suitable domains. It also presupposes privileged information, either a reference solution or the ability to obtain a small set of high-quality correct rollouts. This suggests that OGLS-SD is best understood as a calibration-aware bridge between RLVR-style outcome supervision and token-level distillation, rather than as a universal steering method for arbitrary open-ended generation.

Its operational costs are higher than single-trajectory SFT or RLVR because each prompt requires multiple rollouts and multiple teacher evaluations under guided contexts. The paper nevertheless positions the token-level KL objective as more sample-efficient and stable for reasoning than a pure RLVR objective.

5. Empirical results, stability, and ablations

On Qwen3-1.7B, mean@8 on AIME24/AIME25/Avg is 51.5/36.7/44.1 for the base model, 48.4/36.3/42.4 for SFT, 51.1/38.3/44.7 for GRPO, 56.3/40.8/48.5 for OPSD, and 59.2/45.0/52.1 for OGLS-SD. On Qwen3-4B, the corresponding values are 74.9/66.4/70.1 for the base model, 70.2/62.3/66.3 for SFT, 75.6/68.1/71.9 for GRPO, 76.3/67.9/72.1 for OPSD, and 77.9/68.8/73.4 for OGLS-SD. Across both model sizes, OPSD improves over GRPO and SFT, and OGLS-SD improves further over OPSD; SFT alone can degrade performance relative to the base model on these reasoning benchmarks (Yang et al., 12 May 2026).

Training dynamics are an equally important part of the empirical picture. OPSD peaks around $\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$3 steps on AIME24 and then degrades, whereas OGLS-SD yields a smoother and more stable curve. The epistemic-marker analysis aligns with that difference: OPSD suppresses explicit reflection early and only partially rebounds, while OGLS-SD increases reflection markers over training. The associated interpretation in the paper is that OGLS-SD corrects privileged-prompt mismatch rather than merely sharpening supervision.

The ablations are especially diagnostic. Replacing the single privileged teacher with a positive mean teacher (PMT) improves AIME24 performance from 56.3 to 57.1 on Qwen3-1.7B, but remains below OGLS-SD at 59.2. More strikingly, positive-only steering,

$\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$4

scores 55.4, worse than standard OPSD, and exhibits early peaking followed by degradation. This directly rebuts the idea that OGLS-SD is simply “stronger” privileged distillation: the benefit comes from outcome-guided contrastive steering with negative guidance, not from $\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$5 amplification alone (Yang et al., 12 May 2026).

A plausible implication is that OGLS-SD belongs to a broader class of methods in which post-training gains arise from reweighting or re-expressing latent capabilities rather than constructing entirely new ones. That interpretation is consistent with bias-only adaptation results showing that frozen backbones equipped with layer-wise additive steering vectors can recover or exceed fully tuned reasoning performance on GSM8K and MATH in several settings (Sinii et al., 24 May 2025).

6. Related formulations, misconceptions, and limitations

OGLS-SD sits inside a broader steering landscape but occupies a specific niche. In one neighboring line of work, “Ideology as a Problem: Lightweight Logit Steering for Annotator-Specific Alignment in Social Media Analysis” introduces a dual-probe module that learns a directional term $\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$6, a non-negative score term $\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$7, and a redistribution coefficient $\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$8 to calibrate three-way class logits $\pi_{\theta'}(\cdot \mid x, s, y_{<t}),$9 for annotator-specific ideological judgments, without altering the frozen LLM’s internal activations. That method is structurally outcome-guided logit steering, but it operates on class-label logits for social media ideology classification rather than on token-level teacher distributions during self-distillation (Xia et al., 8 Dec 2025).

Another neighboring line uses training-free inference-time token-score tables. “Steering LLMs Before They Speak: Logit-Level Interventions” computes z-normalized log-odds scores from labeled corpora, selects a favored subset $$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$0 inside a top-$$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$1 candidate set, and adds a constant positive logit bias $$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$2 during decoding. On OSE and WikiPol it reports up to $$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$3p accuracy and a near tripling of $$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$4, and on RealTox it reports an $$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$5 improvement of roughly $$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$6. This is logit steering in the strictest decoding-time sense, but unlike OGLS-SD it does not construct a teacher distribution, does not use privileged self-distillation, and does not rely on on-policy correct/incorrect trajectory contrast (An et al., 16 Jan 2026).

A third relevant perspective concerns when linear steering is likely to work. “Predicting Where Steering Vectors Succeed” defines the Linear Accessibility Profile, with

$$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$7

and shows that peak $$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$8 predicts steering effectiveness at $$\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],$$9 to $0$0, while layer selection correlations range from $0$1 to $0$2. This suggests a useful diagnostic lens for outcome-guided steering more broadly: linear steering succeeds when the target outcome is already output-aligned, whereas nonlinear methods are needed when the concept is present but not linearly accessible through the unembedding (Billa, 16 Apr 2026).

Several limitations of OGLS-SD follow directly from its design. It is infeasible for non-verifiable tasks because it depends on reliable correct/incorrect labels. Noisy rewards contaminate $0$3, $0$4, and the steering direction. When correct rollouts are rare, especially early in training or on very hard problems, steering opportunities are skipped because either the positive or negative pool is empty. Distillation is truncated to the first 1024 tokens, so improvements are less direct on very late reasoning steps. Compute cost is materially higher than SFT or single-trajectory RLVR. The method also does not solve the normative question of what should count as the correct privileged target; it only calibrates token-level supervision so that outcome information shapes the teacher distribution more appropriately under the deployment-time mode (Yang et al., 12 May 2026).