Reasoning Vector Arithmetic

• Reasoning vector arithmetic is a framework that defines and manipulates directional shifts in model parameters or activations to represent specific reasoning abilities.
• It leverages fine-tuning and linear algebra to extract reasoning vectors, enabling performance improvements and interpretability across linguistic, visual, and symbolic domains.
• Empirical studies show that controlled vector interventions yield substantial gains in reasoning benchmarks and cross-modal task performance.

Reasoning vector arithmetic refers to the family of mathematical and computational techniques that discover, represent, and manipulate directions in model parameter space, embedding space, or activation subspace that correspond specifically to reasoning abilities or transformations. This paradigm underpins both the empirical transfer of reasoning skills across models and modalities, and the analytical dissection of reasoning mechanisms within neural, symbolic, or hybrid systems. Its theoretical, algorithmic, and practical foundations span massive LLMs, neural embedding models, vector-symbolic architectures, and specialized solvers, yielding a broad, cross-modal account of reasoning as an emergent property of high-dimensional linear structure.

1. Formal Definitions and Core Principles

At its core, reasoning vector arithmetic operationalizes the idea that reasoning capacity—whether linguistic, visual, logical, or symbolic—can be localized as a direction or subspace in model parameter or representation space. The canonical formulation (for LLM weights) is:

$v = \theta_{\mathrm{post}} - \theta_{\mathrm{pre}}$

where $\theta_{\mathrm{pre}} \in \mathbb{R}^D$ and $\theta_{\mathrm{post}} \in \mathbb{R}^D$ are flattened parameter vectors of a pre-trained and a reasoning fine-tuned model, respectively, and $v$ is the extracted "reasoning vector" (Oguchi et al., 4 Aug 2025, Zbeeb et al., 1 Sep 2025).

In activation space, analogous constructs are extracted via contrastive averaging of residual activations between appropriately selected prompt conditions:

$$\Delta_{i,l}^{(r)} = \bar a_{i,l}^{+} - \bar a_{i,l}^{-}$$

where $+$ and $-$ index activations under strong and weak reasoning prompts, respectively (Wang et al., 26 Apr 2026).

In embedding or latent representation spaces, arithmetic over concept or task vectors (e.g., $z_B - z_A$, analogy parallelograms) formally mediates transformations between reasoning contexts (Ethayarajh et al., 2018, Thoms et al., 2023, Feucht et al., 22 Nov 2025).

These constructions assume that the effect of reasoning-specific fine-tuning or control manifests linearly in the model’s weight space or in suitable low-rank subspaces of representational geometry.

2. Extraction and Manipulation of Reasoning Vectors

The extraction and application workflow for reasoning vectors in neural models is systematically defined by:

1. Pretraining/Fine-tuning: Begin with a pre-trained checkpoint $\theta_{\mathrm{pre}}$. Fine-tune a copy on reasoning-intensive data (e.g., supervised chain-of-thought or RL-based fine-tuning) to obtain $\theta_{\mathrm{post}}$ (Oguchi et al., 4 Aug 2025, Zbeeb et al., 1 Sep 2025).
2. Vector Delta Calculation: Compute $\theta_{\mathrm{pre}} \in \mathbb{R}^D$0 (elementwise across all parameters or within selected blocks).
3. Scaling and Injection: For a compatible target model $\theta_{\mathrm{pre}} \in \mathbb{R}^D$1, form enhanced parameters

$\theta_{\mathrm{pre}} \in \mathbb{R}^D$2

with $\theta_{\mathrm{pre}} \in \mathbb{R}^D$3 a scaling hyperparameter (Oguchi et al., 4 Aug 2025).

1. Activation Modulation: In activation-space interventions, add reasoning vectors to residual stream activations at inference for specific steering (e.g., $\theta_{\mathrm{pre}} \in \mathbb{R}^D$4 with $\theta_{\mathrm{pre}} \in \mathbb{R}^D$5 denoting reasoning type) (Wang et al., 26 Apr 2026).
2. Subspace Decomposition: Refined extraction may involve decomposing vectors using sparse autoencoders and enforcing complementary and subspace-preserving constraints, yielding vector sets for different subtypes of logical reasoning with controlled linear independence and complementarity (Wang et al., 26 Apr 2026).

This vector-arithmetic approach is also realized in embedding spaces for analogical reasoning by calculating difference vectors or "rule vectors" and then applying them to novel examples (e.g., $\theta_{\mathrm{pre}} \in \mathbb{R}^D$6) (Thoms et al., 2023).

3. Mathematical Structure and Theoretical Foundations

The success of reasoning vector arithmetic derives from the approximately linear and low-rank nature of fine-tuning-induced transformations and of relational parallels in embedding spaces. For SGNS-like word embeddings, analogical reasoning is characterized by parallelogram geometry, and operationalized as (Ethayarajh et al., 2018):

$\theta_{\mathrm{pre}} \in \mathbb{R}^D$7

This is theoretically grounded in the factorization of shifted PMI matrices and the alignment of relational ratios with vector differences:

$\theta_{\mathrm{pre}} \in \mathbb{R}^D$8

For LLM parameter space, the linear mode connectivity principle justifies that fine-tuned models originating from the same initialization inhabit connected low-loss valleys, and thus interpolating or extrapolating along the vector connecting two such models is loss-preserving and task-additive (Zbeeb et al., 1 Sep 2025).

In neural symbolic systems and vector-symbolic architectures (VSAs), arithmetic (addition, subtraction) and logical (binding, unbinding) operations on vectors implement symbolic transformations, e.g.,

$\theta_{\mathrm{pre}} \in \mathbb{R}^D$9

with $\theta_{\mathrm{post}} \in \mathbb{R}^D$0 denoting binding and $\theta_{\mathrm{post}} \in \mathbb{R}^D$1 its inverse (Sun et al., 21 Jan 2025, Hersche et al., 2024).

4. Empirical Efficacy and Metrics

Empirical studies across domains establish the effectiveness of reasoning vector arithmetic:

• LLM Weight Space Transfer: Injecting reasoning vectors (e.g., from GRPO-fine-tuned to supervised instruction-tuned models) yields consistent, often substantial, performance improvements across diverse reasoning benchmarks (GSM8K: +4.9%, HumanEval: +4.3%, SciQ: +1.7%, BigBenchHard: +12.3% for Qwen2.5-1.5B) (Zbeeb et al., 1 Sep 2025). In "Enhancing Japanese LLMs with Reasoning Vectors," $\theta_{\mathrm{post}} \in \mathbb{R}^D$2 led to 6 more correct answers (from 4 to 10 out of 30) in Japanese on AIME24, surpassing the original reasoning model (Oguchi et al., 4 Aug 2025).
• Activation Steering: Arithmetic addition of independently extracted logical reasoning vectors for deductive, inductive, and abductive subtypes in Llama-3.1-8B-it increases task accuracy: e.g., deductive unsteered 48.95% $\theta_{\mathrm{post}} \in \mathbb{R}^D$3 mono-steer 55.22% $\theta_{\mathrm{post}} \in \mathbb{R}^D$4 complementary 56.46% (Wang et al., 26 Apr 2026).
• Embedding Space Analogies: Word-level parallelogram arithmetic in concept-lensed subspaces of Llama-2-7B achieves 80% nearest-neighbor accuracy on capitals analogies (vs. 47% in raw hidden states) (Feucht et al., 22 Nov 2025).
• Visual and Multi-modal Reasoning: In visual analogical tasks, VAE-based vector routines score up to 8.8% (ConceptARC) via latent-space arithmetic (Thoms et al., 2023); in multi-modal relation reasoning, cross-modal subtraction/addition in a shared embedding space raises 2-term test accuracy from 23.3% to 35.5% (+52%) under reinforcement fine-tuning (Xu et al., 21 Apr 2026).
• Vector-Symbolic and Logical Reasoning: VSAs implement systematic arithmetic reasoning for RPM tasks, approaching perfect accuracy on complex grids and large value ranges, outperforming LLMs on length generalization (Hersche et al., 2024, Sun et al., 21 Jan 2025).
• Specialized Domains: In self-supervised speech models, phonological features such as voicing are realized as principal directions, verified by analogy arithmetic and continuous control (e.g., $\theta_{\mathrm{post}} \in \mathbb{R}^D$5, with 93% analogy success across 96 languages) (Choi et al., 21 Feb 2026).

5. Domain-Generalization, Transferability, and Compatibility

The generality of reasoning vector arithmetic is supported by diverse experimental evidences:

Aspect	Empirical Finding	Reference
Language transfer	Reasoning vectors computed from English LLMs transfer to Japanese LLMs, raising performance on Japanese reasoning tasks	(Oguchi et al., 4 Aug 2025)
Domain transfer	Reasoning vectors derived from code or math domains transfer across tasks, though in-domain vectors yield largest gains	(Zbeeb et al., 1 Sep 2025)
Architecture constraints	Successful vector addition requires identical model architecture (layer-wise match, parameter ordering, tokenizer)	(Oguchi et al., 4 Aug 2025)
Low-rank subspaces	Lenses composed of top-k attention head OV-matrices yield high analogy accuracy at low effective rank (down to 256)	(Feucht et al., 22 Nov 2025)
Parameter scaling	Empirical optimal scaling found at $\theta_{\mathrm{post}} \in \mathbb{R}^D$6; both under- and over-scaling degrade performance	(Oguchi et al., 4 Aug 2025)

These findings indicate that reasoning knowledge is realized as a transferable, approximately linear direction in either parameter or feature space, provided careful architectural alignment. Transfer between disjoint model families or across significant tokenization changes remains limited.

6. Interpretability, Mechanistic Insights, and Systematicity

Reasoning vector arithmetic opens a route to causal and mechanistic interpretability:

• Orthogonality and Complementarity: Logical reasoning subtypes (deduction, induction, abduction) correspond to (nearly) orthogonal vectors in representation space. Refinement procedures introduce controlled overlap, facilitating shared but distinct knowledge representations (Wang et al., 26 Apr 2026).
• Core Feature Analysis: Sparse autoencoder decompositions reveal that refined reasoning vectors activate meaningful and human-interpretable features, e.g., "therefore," "since," for deduction, and "more plausible" for abduction (Wang et al., 26 Apr 2026).
• Symbolic Arithmetic and Rule Induction: In VSAs, arithmetic and logical rules are abducted and executed by explicit vector-algebra, yielding systematic and interpretable reasoning traces with compositional generalization (Sun et al., 21 Jan 2025).
• Activation Patching: Mechanistic probing via head activation patching shows that steering along reasoning vectors causally upregulates specific attention heads and latent units associated with reasoning, concentrating activity and sharpening relevant subspace occupation (Wang et al., 26 Apr 2026).

This structural clarity supports both high-level modular composition—e.g., summing vectors for code and math reasoning—and layer/block-specific interventions.

7. Limitations, Open Questions, and Extensions

Despite demonstrated efficacy, reasoning vector arithmetic faces foundational and practical constraints:

• Model-family constraints: All known successful transfers require identity of architecture, parameter shapes, and tokenization (Oguchi et al., 4 Aug 2025, Zbeeb et al., 1 Sep 2025).
• Calibration and drift: Over-scaling or misalignment can induce catastrophic model drift or degrade prior capabilities; empirical tuning of scaling parameters is recommended.
• Skill vector composition: The algebraic interactions among multiple reasoning vectors (e.g., addition vs. interference) are not fully characterized.
• Out-of-distribution transfer: Generalization to unseen tasks, input distributions, or entirely new domains remains open.
• Transparency in multi-layer reasoning: In transformer architectures, mapping how vector directions propagate through successive nonlinearities for arbitrarily deep models is not analytically complete.

Future work aims to elaborate tangent-space editing, cross-family vector mapping, compositional reasoning skill vectors, and integrate learning-to-learn mechanisms with reasoning vector arithmetic paradigms.

---

In sum, reasoning vector arithmetic provides both a formalism and a practical toolkit for extracting, modulating, and interpreting reasoning capabilities in neural and hybrid systems. Its empirical generality, theoretical grounding in linear subspaces and information-theoretic geometry, and modular applicability across linguistic, visual, and symbolic domains constitute a foundational advance in modeling and transferring higher-level cognitive functions in machine intelligence (Oguchi et al., 4 Aug 2025, Zbeeb et al., 1 Sep 2025, Wang et al., 26 Apr 2026, Ethayarajh et al., 2018, Thoms et al., 2023, Hersche et al., 2024, Sun et al., 21 Jan 2025, Feucht et al., 22 Nov 2025, Choi et al., 21 Feb 2026, Xu et al., 21 Apr 2026, Kong et al., 6 Feb 2026, Rath et al., 2024, Bu et al., 13 Aug 2025).