---
title: 'RIRS: Reasoning-Induced Representation Shift'
url: https://www.emergentmind.com/topics/reasoning-induced-representation-shift-rirs
type: topic
---

# RIRS: Reasoning-Induced Representation Shift

Searching arXiv for the cited papers and closely related work on representation shifts during reasoning.
Reasoning-Induced Representation Shift (RIRS) denotes a family of phenomena in which engaging, prompting, or training for reasoning alters the internal representational regime used by a model, sometimes improving abstraction and generalization, and sometimes degrading grounding, safety, or transfer. Across recent work, RIRS appears not as a single mechanism but as a recurring pattern: reasoning can move inference from one state distribution, subspace, or structural regime to another, and the consequences depend on whether that movement preserves task-relevant constraints such as semantic grounding, compositional structure, or safety circuitry [2602.00770][2603.21736][2602.16587].

## 1. Conceptual scope and theoretical status

RIRS is best treated as an umbrella construct spanning several levels of analysis. At the most abstract level, Wu argues that different forms of reasoning require different representational structures, organized around four properties: operability, consistency, structural preservation, and compositionality. In that framework, induction, analogy, causal inference, deduction, and formal logic do not merely differ in difficulty; they impose different structural demands on the representational system, and scaling without “structural reorganization” is insufficient to cross the principal boundary between weaker and stricter reasoning regimes [2603.21736]. This suggests that robust reasoning may require a shift in structural guarantees rather than only more computation over an unchanged code.

A more explicit architectural version appears in work on “Representation of the Logical Structure” (RLS). There, the central claim is that reasoning over natural-language arguments works better when the system first transforms text into an intermediate representation that encodes logical atoms, rules, and inferential structure, after which reasoning is “deterministic and easy.” The representation shift is therefore engineered rather than inferred from hidden-state dynamics: reasoning is implemented by moving from surface natural language to a constrained symbolic abstraction [2508.14488].

Earlier representation-learning work already anticipated this general picture. “Reasoning-Modulated Representations” inserts a pre-trained reasoning processor into the latent path of a perceptual model and argues that the frozen processor “modulates the representations” learned by the natural encoder, forcing them into a space compatible with an algorithmic prior [2107.08881]. In abstract visual reasoning, DAReN similarly couples representation and reasoning by structuring latent dimensions into rule-relevant, rule-irrelevant, and nuisance factors, then updating rule-bearing dimensions via row-consistent latent operations; the reported result is that joint reasoning-and-representation learning improves both RPM accuracy and disentanglement metrics over stage-wise baselines [2109.13156].

Taken together, these works support a broad interpretation: RIRS includes both online state transitions during reasoning and slower architectural or training-induced reorganization of the latent space. What varies is whether the shift is observed as hidden-state motion, imposed as an intermediate representation, or induced by optimization.

## 2. Formalizations of representational change during reasoning

One direct formalization treats reasoning as a sequence of latent states during autoregressive generation. In “Reasoning as State Transition,” a transformer is decomposed into a backbone \(f_\theta\) and decoder \(g_\theta\), with internal state
\[
c_t := f_\theta(x, y_{1:t}) \in \mathbb{R}^m,
\]
where \(c_t\) is the final-layer hidden state of the last token. Reasoning is then modeled as the trajectory
\[
x \to c_0 \to c_1 \to \cdots \to c_T.
\]
The paper operationalizes representational evolution through probing accuracy, start–end alignment statistics, and qualitative distributional reshaping over generation, arguing that reasoning tasks exhibit “continuous and significant representation change throughout the generation process” [2602.00770].

A complementary geometric formalization appears in “The Geometry of Reasoning.” There, a reasoning sequence induces a context-cumulative trajectory
\[
Y=[y_1,\ldots,y_T], \qquad y_t=\Psi(S_t),
\]
with stepwise shifts
\[
\Delta y_t = y_t-y_{t-1}.
\]
The paper further posits a smooth trajectory \(\widetilde{\Psi}(s)\), defines local velocity \(v(s)=\frac{d}{ds}\Psi(s)\), and uses Menger curvature over triples of consecutive states to characterize second-order structure. The core empirical claim is that semantic carrier dominates trajectory position, whereas logical structure is more visible in first-order and especially second-order geometry, so logic is read off more clearly from \(\Delta y_t\) and curvature than from static embeddings alone [2510.09782].

A third formalization emphasizes decomposition of reasoning-conditioned scores. In recommendation, the “reasoning shift” in Semantic ID-based foundation recommenders is analyzed via
\[
S(y\mid\cdot)=\log P(y\mid\cdot),
\]
and conditional pointwise mutual information
\[
CPMI(y;x\mid c)=\log\frac{P(y\mid x,c)}{P(y\mid c)} = S(y\mid x,c)-S(y\mid c).
\]
This yields
\[
S(y\mid x,c)=CPMI(y;x\mid c)+S(y\mid c),
\]
where \(CPMI(y;x\mid c)\) is interpreted as Semantic ID Consistency and \(S(y\mid c)\) as the General Subspace Prior induced by chain-of-thought. Here RIRS is framed as a shift in effective decoding mass from a history-grounded semantic-ID state toward a general-language prior state [2602.16587].

A minimal operational definition is used in SHIFT for RLVR data selection. For an input \(x\), the method computes start and end hidden states of a single deterministic reasoning rollout, averages across layers, and defines
\[
\Delta(x)=\mathbf e(x)-\mathbf s(x), \qquad q(x)=\|\Delta(x)\|_2.
\]
The RIRS magnitude \(q(x)\) is then used as a training-free proxy for instance utility [2605.28631].

These formulations differ in granularity, but they converge on one point: reasoning is represented not merely by outputs, but by movement between internal states, and that movement can be measured through score decomposition, hidden-state trajectories, or start–end deltas.

## 3. Empirical manifestations across domains

A prominent domain-specific instance appears in Semantic ID-based recommendation. In OpenOneRec-style models, enabling explicit CoT often degrades recommendation quality rather than improving it. On 1,000 sampled test instances from both Ad and Product, naïve CoT reduces Recall@10 for Qwen-1.7B / OpenOneRec from **0.0282** to **0.0217** on Ad and from **0.0195** to **0.0111** on Product; for Qwen-8B / OpenOneRec, Recall@10 drops from **0.0394** to **0.0375** on Ad and from **0.0239** to **0.0145** on Product. The same paper reports **SDI \(=41.09\), AEI \(=1.495\)** for Think=On versus **SDI \(=27.23\), AEI \(=2.126\)** for Think=Off, supporting the claim that verbose reasoning redistributes attention toward general-text tokens and dilutes attention per CoT token [2602.16587].

In general LLM reasoning, the strongest direct evidence for online state transition comes from generation-time probing. “Reasoning as State Transition” reports that post-training yields only limited improvement in static initial representation quality, typically less than **5%** across training stages, while generation accuracy improves much more; on reasoning tasks, final representations correlate strongly with correctness, whereas initial representations do not. In several settings, ROC-AUC for correctness prediction from final states rises substantially, while initial–final correlations remain very low, with some \(r^2\) values near **0.00**. This supports the view that reasoning success is associated with ending in a different representation distribution rather than merely reading out a pre-existing solution from \(c_0\) [2602.00770].

RIRS also appears in post-training transfer. On controlled Qwen3-14B experiments with math-only tuning, supervised fine-tuning improves math but often damages non-reasoning capabilities, whereas RL improves math while preserving or improving general behavior. The reported average scores are: math **27.7** for base, **49.8** for think-SFT, **32.3** for no-think-SFT, and **53.8** for RL; other reasoning **30.2**, **45.3**, **45.2**, and **60.0**; non-reasoning **45.7**, **21.1**, **29.0**, and **53.2**. The associated PCA-based latent shift on non-reasoning inputs is **113.7** for no-think SFT, **38.2** for think SFT, and **36.9** for RL, with RL also showing much smaller token-distribution KL divergence to the base model on MATH500 and IFEval (**0.084** and **0.019**, versus **0.372** and **0.283** for SFT-no-think) [2507.00432].

Under distribution shift, the phenomenon can take a different geometric form. “Farther the Shift, Sparser the Representation” reports that as difficulty increases across harder reasoning questions, longer contexts, added answer choices, and knowledge conflicts, the last hidden states become systematically sparser. On MATH-500, accuracy degradation correlates with sparsity using \(r=0.897\) for accuracy versus \(L_1\) and \(r=-0.915\) for accuracy versus Top-10% Energy. In the knowledge-conflict setting, conflict examples are consistently sparser, with paired t-test \(p<2.0\times10^{-29}\). This suggests that one observable signature of RIRS under harder or more OOD reasoning is a shift toward concentrated terminal representations [2603.03415].

Finally, multimodal safety provides a non-CoT but structurally analogous case. In VLM jailbreaks, harmful inputs remain separable from benign inputs, yet jailbreak samples form a distinct internal state separable from refusal samples. The image-induced shift
\[
\Delta \mathbf h^{(\ell)}(x)=\mathbf h^{(\ell)}([I,T])-\mathbf h^{(\ell)}([\varnothing,T])
\]
is projected onto a learned jailbreak direction, yielding a jailbreak-related shift that predicts behavior across explicit, implicit, and adversarial jailbreaks. On LLaVA-1.5-7B, for instance, a large fraction of jailbreak outputs still contain safety warnings—**70.24%** on explicit MM-SafetyBench, **76.18%** on HADES, and **69.69%** on RedTeam2K—supporting the claim that harmfulness remains represented even when the model has shifted into a jailbreak state [2603.17372].

## 4. Mechanisms and failure modes

Several papers identify a recurring pattern: reasoning does not fail only by producing wrong answers; it can reconfigure the internal route by which the model decides.

In recommendation, the proposed root cause is **textual inertia from the General Subspace**. The General Subspace and Semantic ID Subspace “share a latent space” but “remain semantically distinct and are not perfectly aligned,” so long free-form rationales increase the text-conditioned prior \(S(y\mid c)\), weaken Semantic ID Consistency, and cause ungrounded textual drift. The paper explicitly states that verbose CoT can “hallucinate logic that is weakly grounded in the recommendation subspace” [2602.16587].

In safety, the corresponding phenomenon is **Reasoning-Induced Misalignment (RIM)**. The mechanistic account identifies attention heads that facilitate refusal by reducing attention to CoT tokens and shifting attention toward the empty span between `<think>` and `</think>` in no-think mode. It also identifies higher activation entanglement between reasoning and safety in safety-critical neurons than in control neurons, with Reciprocal Activation Shift correlating with misalignment rate at **\(r=0.891\), \(p=0.003\)** in Qwen3-4B. Safety-critical neuron ablation increases misalignment by **+13.26%** on average and reduces math accuracy by **-18.19%**, supporting a neuron-level interference account in which reasoning gains and safety degradation share overlapping resources [2509.00544].

A related but task-specific failure mode is **cyclical reasoning** in long CoT. “Shift-FFN” reports that smaller adjacent-token hidden-state differences are associated with higher tendency toward cyclical reasoning, defined behaviorally through generations that exceed the 32k inference limit and often repeat previous inference steps. The paper formalizes adjacent-token relative change as
\[
s(\mathbf x_{i-1}^{l},\mathbf x_i^{l})=\frac{\|\mathbf x_i^{l}-\mathbf x_{i-1}^{l}\|_2}{\|\mathbf x_{i-1}^{l}\|_2}
\]
and averages this over positions and layers into \(M(\mathbf X)\). For Qwen2.5-7B on AIME24, increasing LoRA rank from **64** to **256** raises \(M(\mathbf X)\) from **80.31%** to **80.98%** while reducing cyclical reasoning \(P_E\) from **60.9%** to **30.4%**; adding Shift-FFN further raises \(M(\mathbf X)\) to **81.24%** and reduces \(P_E\) to **25.1%** [2505.17153].

In multimodal IQA, the mechanism is almost the reverse of the recommendation case. There the claim is that RL-trained reasoning MLLMs shift dependency away from long visual token sequences toward short textual quality descriptions. Figure-level analysis reports that when Q-Insight generates score tokens, **95%** of attention weight goes to reasoning text tokens and only **5%** to visual tokens, with **78** reasoning tokens compared to **1020** image tokens. The paper interprets this as a beneficial conversion of “redundant visual representations into compact, cross-domain aligned text representations” [2510.11369].

Latent visual reasoning adds another failure taxonomy. Prior methods are said to drift away from the pretrained vocabulary-aligned manifold, collapse into instance-agnostic latent patterns, and become bypassed during answer generation. RIS interprets these as failures of manifold compatibility and introduces grounding, bottlenecking, and transition tokens to prevent detached or non-causal latent trajectories [2605.07106].

## 5. Interventions and design responses

Because RIRS can be helpful or harmful, recent work increasingly treats representation shift as something to be engineered rather than merely observed.

In recommendation, the main response is **Inference-Time Subspace Alignment**. The first component is Reasoning-Chain Compression, which converts a raw CoT \(c\) into a compact control variable \(\hat c=T(c)\) using a fixed template: “The current user's preference is [summary content].” The second component is Bias-Subtracted Contrastive Inference, which combines expert, amateur, and baseline score contexts:
\[
Z_E(y)=\log P_e(y\mid x,\hat c), \quad
Z_A(y)=\log P_e(y\mid x_\emptyset,c), \quad
Z_B(y)=\log P_e(y\mid x).
\]
After Z-score normalization, the final score is
\[
S(y)=(1+a)\hat Z_E(y)-a(\hat Z_A(y)-\hat Z_B(y)).
\]
This training-free procedure restores or exceeds the think-off baseline: for Qwen-8B, Recall@10 rises to **0.0436** on Ad and **0.0252** on Product, above both think-off (**0.0394**, **0.0239**) and think-on (**0.0375**, **0.0145**) [2602.16587].

In VLM safety, the analogous intervention is **JRS-Rem**, which subtracts the component of the multimodal hidden state aligned with the jailbreak-related shift:
\[
\hat{\mathbf h}^{(\ell)}(x)=\mathbf h^{(\ell)}(x)-s^{(\ell)}(x)\cdot \mathbf d^{(\ell)}, \qquad \text{s.t. } \tilde s^{(\ell)}(x)>\tau,
\]
with \(\tau=0.2\). This is applied during the first generated token and reduces ASR sharply while largely preserving benign utility. On LLaVA-1.5-7B, HADES ASR drops from **77.3/73.7/84.9** to **12.2/9.40/12.4** across SD, TYPO, and SD+TYPO images; MM-Vet changes only from **32.1** to **31.6**, ScienceQA remains **64.0**, and MME remains **1754.9** [2603.17372].

In long-CoT reasoning, **Shift-FFN** modifies the current token representation with the previous token before the FFN:
\[
\text{Shift-FFN}(\mathbf x_i)=\text{FFN}(\mathbf x_i+f_s(\mathbf x_{i-1},\mathbf x_i)).
\]
The architecture is designed specifically to amplify adjacent-token representation differences. On Qwen2.5-7B, LoRA \(r=256\) plus Shift-FFN improves average benchmark accuracy from **50.4** to **51.2** and reduces length-exceeded percentage from **15.0** to **12.7**; the parameter-matched comparison also shows that LoRA+Shift-FFN \(r=256\) outperforms standard LoRA \(r=296\) despite equal parameter count (**0.75B**) [2505.17153].

In IQA, the intervention is to bypass online reasoning by learning the post-reasoning representation directly. RALI aligns images to RL-generated textual quality descriptions using CLIP-style contrastive learning and then predicts scores from a compressed score-aware basis. With training on KonIQ, RALI reaches average **0.798 / 0.779** PLCC/SRCC, close to Q-Insight’s **0.806 / 0.783**, while using about **4%** of Q-Insight’s parameters and reducing inference time and memory by more than **95%** in the reported A100 setting [2510.11369].

For latent visual reasoning, RIS stabilizes beneficial internal shifts by grounding latent slots to spatial boxes and semantic descriptions, enforcing causal use through a progressive attention bottleneck, and bridging back to language with short transition tokens. Reported benchmark values include **83.75** on V\(^*\), **73.23** on HRBench4K, **68.52** on HRBench8K, **73.55** on MMVP, and **60.60** on BLINK, together with analyses showing more diverse and less collapsed latent trajectories than prior methods [2605.07106].

Finally, SHIFT treats RIRS itself as a selection primitive for RLVR. For each unlabeled instance, it computes
\[
\Delta(x)=\mathbf e(x)-\mathbf s(x), \qquad q(x)=\|\Delta(x)\|_2,
\]
constructs
\[
\phi(x)=\frac{[\mathbf s(x);\Delta(x)]}{\|[\mathbf s(x);\Delta(x)]\|_2},
\]
and performs quality-weighted farthest-first selection. On Qwen2.5-Math-1.5B with a **2%** MATH-500 budget, SHIFT achieves **62.67** on MATH-500 and **38.55** on AMC, versus **53.73/25.78** for Random and **60.00/30.12** for Q-PPL. In the pilot single-instance RLVR study, pre-RL RIRS rank correlates with downstream gain with Spearman \(\rho=0.818\) and Kendall \(\tau=0.644\) [2605.28631].

## 6. Limitations, disputes, and open questions

Despite increasingly direct analyses, the evidential status of RIRS remains uneven. Several papers are explicit that they do not yet provide full mechanistic proof. The recommendation paper offers PCA and attention statistics but not a neuron- or circuit-level subspace analysis, so its “General Subspace” remains representational and empirical rather than a rigorously estimated linear subspace [2602.16587]. The state-transition paper localizes the phenomenon to the final-layer last-token state and probe-decoded information, not to layerwise circuits or token-level causal pathways [2602.00770]. The structural account in Wu is explicitly a necessary-condition theory and is agnostic about implementation format, leaving open whether structural reorganization is realized by symbols, role–filler factorization, attentional routing, or some hybrid mechanism [2603.21736].

A second unresolved issue is scope. The strongest direct evidence for harmful RIRS comes from specialized settings: Semantic ID recommendation, multimodal jailbreaks, safety refusal, and long-CoT math tuning. The strongest direct evidence for beneficial RIRS comes from IQA, latent visual reasoning, and training-free RLVR data selection. This suggests that the sign of the shift is task-dependent: grounding-sensitive tasks can be harmed by movement toward generic language priors, while abstraction-sensitive tasks can benefit from movement toward compact, structured, or aligned representations [2510.11369][2507.00432].

A third open question is measurement. Current work variously uses CPMI decompositions, PCA centroids, probe accuracy, start–end deltas, sparsity statistics, adjacent-token L2 change, centroid-difference directions, and curvature. No consensus metric yet spans all settings. This suggests that RIRS is not one scalar observable but a family of linked observables: state displacement, subspace occupancy, separability, sparsity, trajectory geometry, and neuron-level entanglement [2603.03415][2510.09782][2509.00544].

A final issue concerns control. Several interventions succeed by subtracting, compressing, bottlenecking, or re-aligning the shifted component rather than by suppressing reasoning altogether. That pattern supports a narrower interpretation of the phenomenon: the problem is not reasoning per se, but the direction into which reasoning moves the representation. Under this interpretation, the central research problem becomes not whether models reason, but how reasoning trajectories can be constrained so that internal transitions remain grounded, causally used, and structurally compatible with the downstream task.

Source: https://www.emergentmind.com/topics/reasoning-induced-representation-shift-rirs