---
title: Outcome-Guided Logit Steering (OGLS-SD)
url: https://www.emergentmind.com/topics/outcome-guided-logit-steering-ogls-sd
type: topic
---

# Outcome-Guided Logit Steering (OGLS-SD)

Outcome-Guided Logit Steering (OGLS-SD) is an on-policy self-distillation framework for large language model reasoning in which a student model learns from a calibration-adjusted teacher distribution rather than from raw privileged teacher outputs. It was introduced in “OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning” as a response to a specific failure mode of on-policy self-distillation: self-reflected teacher responses conditioned on privileged solutions can be shifted by reflection-induced bias and response templates, producing miscalibrated token-level supervision. OGLS-SD addresses this by using verifiable outcome rewards to contrast successful and failed on-policy trajectories, constructing a steering direction in logit space that modifies a non-privileged anchor teacher, and then distilling the student toward that steered teacher only on incorrect rollouts. On Qwen3-1.7B and Qwen3-4B, the method improves over SFT, GRPO, and standard OPSD on AIME 2024 and AIME 2025, while also exhibiting smoother training and better preservation of epistemic reflection [2605.12400].

## 1. On-policy self-distillation as the immediate context

OGLS-SD is defined within on-policy self-distillation (OPSD), a post-training paradigm in which a single language model acts as both student and teacher. For each problem, the student samples an on-policy trajectory
\[
y \sim \pi_{\theta}(\cdot \mid x),
\]
while the teacher is the same architecture conditioned on privileged information \(s\), typically a reference solution or expert chain-of-thought available only during training. Standard OPSD distills token distributions along the student’s own trajectory by minimizing
\[
\mathcal{L}_{\mathrm{OPSD}}(\theta) = \sum_{t=1}^{T} \mathrm{KL}\Big( \pi_{\theta}(\cdot \mid x_i, y_{i,<t}) \;\Big\|\; \mathrm{sg}\big[ \pi_{\theta'}(\cdot \mid x_i, s_i, y_{i,<t}) \big] \Big).
\]
The student therefore learns from dense token-level supervision rather than sparse sequence-level rewards [2605.12400].

This setup uses what the paper calls privileged teacher distributions,
\[
\pi_{\theta'}(\cdot \mid x, s, y_{<t}),
\]
and self-reflected teacher responses generated from prompts of the form “after reading the reference solution, derive the answer in your own words…”. In this formulation, the teacher has access to solution information that the deployed student will never observe. The promise of OPSD is that privileged conditioning can provide a richer supervisory signal than outcome-only reinforcement-style training, but that promise depends on the teacher distribution being a suitable target for a non-privileged student.

A useful implication is that OGLS-SD is not a generic decoding-time control method. It is specifically a training-time distillation framework built around self-generated on-policy rollouts, privileged teacher conditioning, and KL-based supervision. This distinguishes it from steering schemes whose primary role is inference-time attribute control.

## 2. Teacher–student mismatch and reflection-induced bias

The central problem OGLS-SD addresses is teacher–student mismatch in OPSD. Even when teacher and student share the same base model, the teacher’s distribution differs because it is conditioned on privileged solution information and a reflection template, whereas the student must solve from scratch. The ideal target inside the non-privileged student class is the marginal privileged teacher
\[
\bar P_T(\cdot\mid x,y_{<t}) = \mathbb{E}_{s\sim P(s\mid x,y_{<t})} \left[ \pi_{\theta'}(\cdot\mid x,s,y_{<t}) \right],
\]
but standard OPSD instead uses one specific solution and one prompt template [2605.12400].

The paper identifies two sources of distortion. The first is solution-specific path bias: conditioning on one reference solution concentrates the teacher around a particular reasoning pattern, narrows the distribution, and can overwrite alternative valid strategies. The second is reflection-induced bias and overconfidence from prompting. Because the teacher is prompted after seeing a correct solution and instructed to “truly understand it” and “derive the same final answer,” its next-token distributions become sharper, exploratory or epistemic tokens are suppressed, and confidence shifts survive even after averaging over multiple privileged prompts. The result is miscalibrated token-level supervision: the student is trained to imitate a mode that is systematically “too sure” relative to its test-time operating regime [2605.12400].

Several diagnostics in the paper make this mismatch concrete. In a toy digits \(0\)–\(9\) experiment, even a mixture over privileged prompts remains far from the non-privileged distribution, with total variation distance \(> 0.5\). In a combinatorial game case study, OPSD and OGLS-SD both infer the correct modular pattern and intermediate counts, but OPSD overconfidently maps the count to the wrong player, whereas OGLS-SD revisits the Alice/Bob role assignment and reaches the correct final answer. A further indicator is the frequency of epistemic markers such as “wait”, “hold on”, and “let me rethink”: OPSD suppresses them early in training and only partially recovers, while OGLS-SD increases them throughout training.

A common misconception is that averaging privileged teacher outputs should be sufficient to resolve mismatch. The paper argues against this directly: prompt-induced marginalization does not recover the non-privileged distribution because the privileged prompt itself places the model in a different behavioral regime.

## 3. The OGLS-SD mechanism

OGLS-SD combines verifiable outcome rewards with logit-space steering. For each input \(x_i\), the current student samples \(G\) rollouts,
\[
\{y_j\}_{j=1}^G \sim \pi_{\theta}(\cdot \mid x_i),
\]
which are then scored by a verifier \(R(x_i, y_j)\in\{0,1\}\). This partitions the on-policy trajectories into
\[
\mathcal{Y}^{+}(x_i) = \{y_j : r_j = 1\}, \qquad
\mathcal{Y}^{-}(x_i) = \{y_j : r_j = 0\}.
\]
Outcome correctness is sequence-level, but OGLS-SD turns it into dense token-level guidance by constructing a contrastive steering direction in logit space [2605.12400].

The framework begins with a non-privileged anchor teacher,
\[
z_{j,t}^0 = z_{\theta'}(\cdot \mid x_i, y_{j,<t}),
\]
where \(\theta'\) is kept fixed as the original base model. Positive and negative privileged guidance are then formed by averaging teacher logits over correct and incorrect pools:
\[
\bar{z}^{\pm}_{j,t} = \frac{1}{|\mathcal{P}_{j}^{\pm}|} \sum_{g \in \mathcal{P}_{j}^{\pm}} z_{\theta'}\big( \cdot \mid x_i, g, y_{j,<t} \big),
\]
with
\[
\mathcal{P}_{j}^{+}(x_i) = \mathcal{Y}^{+}(x_i) \cup \{s_i\},
\qquad
\mathcal{P}_{j}^{-}(x_i) = \mathcal{Y}^{-}(x_i).
\]
The steering direction is
\[
\Delta_{j,t} = \bar{z}^{+}_{j,t} - \bar{z}^{-}_{j,t},
\]
and the steered teacher logits are
\[
z^{T}_{j,t} = z^{0}_{j,t} + \lambda \Delta_{j,t},
\qquad
\pi_T(\cdot \mid x_i, y_{j,<t}) = \mathrm{softmax} \big( z^{T}_{j,t} \big).
\]
The student is trained by token-wise KL against this steered teacher,
\[
L_j^{\mathrm{steer}}(\theta) = \sum_{t=1}^{T_j} \mathrm{KL}\!\Big( \pi_{\theta}(\cdot \mid x_i, y_{j,<t}) \;\Big\|\; \mathrm{sg}\!\big[ \pi_T(\cdot \mid x_i, y_{j,<t}) \big] \Big),
\]
but only for incorrect rollouts:
\[
\mathcal{L}_{\mathrm{steer}}(\theta) = \sum_{j=1}^{G} \mathbf{1}\{r_j=0\} \; L_j^{\mathrm{steer}}(\theta).
\]
This selective application is intended to reshape bad trajectories without perturbing already correct behavior [2605.12400].

The logic of the contrast is central. Positive and negative privileged teachers share the same reflection template and solution-aware prompt regime, so their difference attenuates shared overconfidence and template artifacts while preserving outcome-discriminative components. This is why the method can use \(\lambda > 1\) without simply amplifying privileged bias.

In the broader steering literature, this is a distinctive instantiation of logit steering. It differs from training-free logit-level interventions that modify the final pre-softmax logits at each decoding step with a constant positive logit bias \(\delta\) on a favored token set \(F_t\), and it also differs from annotator-conditioned class-logit calibration where a lightweight probe directly adjusts three-way ideological logits [2601.10960].

## 4. Training procedure, implementation, and operating conditions

The training loop is fully on-policy. For each sampled example \((x_i, s_i)\), the student produces \(G=8\) rollouts using a vLLM backend with temperature \(1.1\). Each rollout is verified, correct and incorrect pools are formed, mixed positive and negative logits are computed, the non-privileged anchor teacher is evaluated on the student prefixes, and the student is updated by minimizing the steering loss on incorrect trajectories. If either guidance pool is empty, the rollout is skipped because no contrast is possible [2605.12400].

The reported implementation uses Qwen3-1.7B and Qwen3-4B as base models, OpenThoughts as the mathematical reasoning training corpus, and AIME 2024 plus AIME 2025 as evaluation benchmarks. Fine-tuning is performed with LoRA, using learning rate \(5\times 10^{-6}\), rank \(r=64\), and scaling \(\alpha=128\). The maximum training generation length is 8192 tokens, but distillation uses the first 1024 tokens. The steering coefficient \(\lambda\) is ramped up during training to a maximum of \(3\), and symmetric pointwise KL clipping at \(0.05\) is applied to avoid outlier gradients. The teacher is fixed as the original base model throughout training for stability [2605.12400].

The framework presupposes verifiable tasks. The paper lists math problems with exact numeric answers, code generation with test suites, logic puzzles with deterministic checkers, and structured QA with deterministic answer sets as suitable domains. It also presupposes privileged information, either a reference solution or the ability to obtain a small set of high-quality correct rollouts. This suggests that OGLS-SD is best understood as a calibration-aware bridge between RLVR-style outcome supervision and token-level distillation, rather than as a universal steering method for arbitrary open-ended generation.

Its operational costs are higher than single-trajectory SFT or RLVR because each prompt requires multiple rollouts and multiple teacher evaluations under guided contexts. The paper nevertheless positions the token-level KL objective as more sample-efficient and stable for reasoning than a pure RLVR objective.

## 5. Empirical results, stability, and ablations

On Qwen3-1.7B, mean@8 on AIME24/AIME25/Avg is 51.5/36.7/44.1 for the base model, 48.4/36.3/42.4 for SFT, 51.1/38.3/44.7 for GRPO, 56.3/40.8/48.5 for OPSD, and 59.2/45.0/52.1 for OGLS-SD. On Qwen3-4B, the corresponding values are 74.9/66.4/70.1 for the base model, 70.2/62.3/66.3 for SFT, 75.6/68.1/71.9 for GRPO, 76.3/67.9/72.1 for OPSD, and 77.9/68.8/73.4 for OGLS-SD. Across both model sizes, OPSD improves over GRPO and SFT, and OGLS-SD improves further over OPSD; SFT alone can degrade performance relative to the base model on these reasoning benchmarks [2605.12400].

Training dynamics are an equally important part of the empirical picture. OPSD peaks around \(\sim 100\) steps on AIME24 and then degrades, whereas OGLS-SD yields a smoother and more stable curve. The epistemic-marker analysis aligns with that difference: OPSD suppresses explicit reflection early and only partially rebounds, while OGLS-SD increases reflection markers over training. The associated interpretation in the paper is that OGLS-SD corrects privileged-prompt mismatch rather than merely sharpening supervision.

The ablations are especially diagnostic. Replacing the single privileged teacher with a positive mean teacher (PMT) improves AIME24 performance from 56.3 to 57.1 on Qwen3-1.7B, but remains below OGLS-SD at 59.2. More strikingly, positive-only steering,
\[
z^{T}_{j,t}
=
z^{0}_{j,t}
+
\lambda \left(z^{+}_{j,t} - z^{0}_{j,t}\right),
\]
scores 55.4, worse than standard OPSD, and exhibits early peaking followed by degradation. This directly rebuts the idea that OGLS-SD is simply “stronger” privileged distillation: the benefit comes from outcome-guided contrastive steering with negative guidance, not from \(\lambda>1\) amplification alone [2605.12400].

A plausible implication is that OGLS-SD belongs to a broader class of methods in which post-training gains arise from reweighting or re-expressing latent capabilities rather than constructing entirely new ones. That interpretation is consistent with bias-only adaptation results showing that frozen backbones equipped with layer-wise additive steering vectors can recover or exceed fully tuned reasoning performance on GSM8K and MATH in several settings [2505.18706].

## 6. Related formulations, misconceptions, and limitations

OGLS-SD sits inside a broader steering landscape but occupies a specific niche. In one neighboring line of work, “Ideology as a Problem: Lightweight Logit Steering for Annotator-Specific Alignment in Social Media Analysis” introduces a dual-probe module that learns a directional term \(s\), a non-negative score term \(g\), and a redistribution coefficient \(\mu\) to calibrate three-way class logits \(\{L,C,R\}\) for annotator-specific ideological judgments, without altering the frozen LLM’s internal activations. That method is structurally outcome-guided logit steering, but it operates on class-label logits for social media ideology classification rather than on token-level teacher distributions during self-distillation [2601.04207].

Another neighboring line uses training-free inference-time token-score tables. “Steering Language Models Before They Speak: Logit-Level Interventions” computes z-normalized log-odds scores from labeled corpora, selects a favored subset \(F_t\) inside a top-\(K\) candidate set, and adds a constant positive logit bias \(\delta\) during decoding. On OSE and WikiPol it reports up to \(+47\%\)p accuracy and a near tripling of \(F_1\), and on RealTox it reports an \(F_1\) improvement of roughly \(50\times\). This is logit steering in the strictest decoding-time sense, but unlike OGLS-SD it does not construct a teacher distribution, does not use privileged self-distillation, and does not rely on on-policy correct/incorrect trajectory contrast [2601.10960].

A third relevant perspective concerns when linear steering is likely to work. “Predicting Where Steering Vectors Succeed” defines the Linear Accessibility Profile, with
\[
A_{\mathrm{lin}}(\ell)
=
\frac{1}{N} \sum_{i=1}^{N}
\mathbf{1}\!\left[\arg\max_v \bigl(W_U \cdot \text{LayerNorm}(h_\ell^{(i)})\bigr)_v = t_i\right],
\]
and shows that peak \(A_{\mathrm{lin}}\) predicts steering effectiveness at \(\rho= +0.86\) to \(+0.91\), while layer selection correlations range from \(\rho= +0.63\) to \(+0.92\). This suggests a useful diagnostic lens for outcome-guided steering more broadly: linear steering succeeds when the target outcome is already output-aligned, whereas nonlinear methods are needed when the concept is present but not linearly accessible through the unembedding [2604.15557].

Several limitations of OGLS-SD follow directly from its design. It is infeasible for non-verifiable tasks because it depends on reliable correct/incorrect labels. Noisy rewards contaminate \(\bar z^+\), \(\bar z^-\), and the steering direction. When correct rollouts are rare, especially early in training or on very hard problems, steering opportunities are skipped because either the positive or negative pool is empty. Distillation is truncated to the first 1024 tokens, so improvements are less direct on very late reasoning steps. Compute cost is materially higher than SFT or single-trajectory RLVR. The method also does not solve the normative question of what should count as the correct privileged target; it only calibrates token-level supervision so that outcome information shapes the teacher distribution more appropriately under the deployment-time mode [2605.12400].

Source: https://www.emergentmind.com/topics/outcome-guided-logit-steering-ogls-sd