Weight-Space Skill Injection

• Weight-space skill injection is a method that embeds new skills into neural model weights while mitigating catastrophic forgetting using techniques like EWC and task vectors.
• It leverages information-theoretic regularization and modular task vectors to balance efficient skill acquisition with retention of existing linguistic abilities.
• Empirical evaluations show that these approaches significantly enhance specialized tasks such as arithmetic and reasoning without degrading prior model performance.

Weight-space skill injection refers to a suite of methodologies for incorporating new capabilities—especially reasoning or domain-specific skills—directly into the parameter space of pretrained or fine-tuned neural LLMs without catastrophic forgetting or interference. These approaches operate at the level of network weights, extracting, transferring, or protecting skill-relevant parameter structure. Recent research formalizes and operationalizes weight-space skill injection through loss-based regularization, modular task “vectors,” alignment and symmetrization in parameter space, and explicit manipulation of surrogate instruction parameters with subsequent weight distillation. This enables efficient and modular adaptation of LLMs to emergent requirements, with rigorous trade-offs between new skill acquisition and retention of core capabilities (Sharma et al., 2022, Tang et al., 16 Jan 2026, Horoi et al., 13 Nov 2025, Costa, 29 Aug 2025).

1. Catastrophic Forgetting and the Skill Injection Challenge

LLMs such as BERT, DistilBERT, and GPT-2 exhibit strong linguistic generalization but demonstrate limited proficiency in systematic arithmetic or other non-linguistic domains without targeted adaptation (Sharma et al., 2022). Naive fine-tuning or further pretraining on skill-specific datasets (e.g., arithmetic problems) results in parameter drift that destroys large swaths of pre-existing linguistic competency, a phenomenon known as catastrophic forgetting. The key research challenge is to devise training regimes or weight-compositional methods that inject new skills—such as arithmetic, reasoning, or tool use—without sacrificing prior linguistic or agentic abilities.

2. Information-Theoretic Regularization: Fisher Analysis and Elastic Weight Consolidation

A central methodology for protecting legacy skills during skill injection is the combination of parameter sensitivity analysis and continual learning regularization. Specifically, the Fisher information is used to quantify the importance of individual parameters to the original task. Given model parameters $\theta$ and a prior data distribution, the diagonal Fisher information matrix $F_i$ estimates the expected squared sensitivity of the model's log-likelihood to each parameter.

During skill injection, an Elastic Weight Consolidation (EWC) penalty is added to the loss:

$$\mathcal{L}(\theta) = \mathcal{L}_{\mathrm{skill}}(\theta) + \sum_{i} \frac{\lambda}{2} F_i (\theta_i - \theta^{*}_{\mathrm{base},i})^2$$

where $\lambda$ governs trade-off, $\mathcal{L}_{\mathrm{skill}}$ is the skill-specific task loss (e.g., cross-entropy for arithmetic), and $F_i$ penalizes movement along skill-critical directions. This regularization constrains the most vital linguistic weights from drifting, thus retaining prior capability while learning the new skill (Sharma et al., 2022).

Empirical evidence shows that this approach achieves nearly optimal skill task performance while substantially restoring downstream linguistic metrics compared to naive fine-tuning, where skill acquisition immediately erodes pre-existing abilities. Table 1 (reproduced below) summarizes the typical results:

Model	ln RMSE (arithmetic)	CoLA	MNLI	MRPC	SST-2	STS-B
Base DistilBERT	3.54	0.4827	0.8074	0.8797	0.8967	0.8740
+ Arithmetic fine-tune	0.44	0.0000	0.3553	0.7524	0.8761	0.3998
+ EWC-regularized injection	0.44	0.4193	0.7951	0.8570	0.8962	0.8626

Loss of performance from naive fine-tuning is ameliorated by EWC, with minimal trade-off in skill acquisition (Sharma et al., 2022).

3. Modular Skill Transfer via Task Vectors and Alignment

An alternative approach to weight-space skill injection exploits the locality and modularity of parameter updates induced by different adaptation strategies. If $\Delta W_{\mathrm{SFT}}$ (supervised fine-tune update) and $\Delta W_{\mathrm{RL}}$ (reinforcement learning update) are nearly orthogonal, as observed empirically and justified theoretically, then a skill vector $s = W_{\mathrm{RL}} - W_{\mathrm{SFT}}$ can be composed additively:

$$W_{\mathrm{injected}} = W_{\mathrm{SFT}}^{(\mathrm{target})} + \alpha s$$

where $\alpha$ controls the injection strength. This "Parametric Skill Transfer" (PaST) protocol linearly grafts RL-acquired skills into a newly SFT-adapted network without negative transfer, as the orthogonality ensures that the new knowledge and skill subspaces do not destructively interfere (Tang et al., 16 Jan 2026).

Empirical results demonstrate substantial gains on SQuAD (QA), LooGLE (long-context QA), and ToolBench (zero-shot agentic tool use) benchmarks, with injection yielding up to +9.9 points over SOTA on SQuAD and robust improvements in agentic and reasoning performance.

4. Parameter-Space Alignment and Symmetry-Aware Injection

The efficacy of linear task vector injection is undermined by architectural non-identities between networks, especially when models have diverged due to independent fine-tuning or employ features such as Grouped-Query Attention (GQA) or SwiGLU MLP blocks. Leveraging fundamental permutation, rotation, and scaling symmetries within transformer blocks, parameter-space alignment becomes critical for robust skill transfer.

The alignment process consists of:

• Rotation (Orthogonal Procrustes): SVD-based rotation aligning weight blocks or activations across models.
• Permutation: Assignment solving to permute MLP hidden neurons or heads.
• Scaling: One-dimensional rescaling within attention pairs post-rotation.

After alignment, task/skill vectors are extracted and transferred in parameter space, typically via:

$$\theta_{\mathrm{target}}^{\mathrm{new}} = \theta_{\mathrm{target}}^{\mathrm{aligned}} + \alpha (\theta_{\mathrm{ref}}^{\mathrm{skill}} - \theta_{\mathrm{ref}}^{\mathrm{base}})$$

For reasoning transfer, this pipeline provides state-of-the-art improvements on mathematical benchmarks, and ablation confirms the dominant contribution from rotation, with scaling yielding additional, smaller gains (Horoi et al., 13 Nov 2025).

5. Instruction-Level Surrogates and Distillation into Weight Space

Instruction-Level Weight Shaping (ILWS) treats system instructions, user preferences, and tool signatures as explicit, version-controlled pseudo-parameters. Skill acquisition proceeds through in-context edits guided by a Reflection Engine. Once a sufficient volume of synthetic, rating-weighted data have accumulated, distillation is triggered:

• Distillation objective:

$$\theta^* = \arg\min_{\theta'} \sum_{(x,K,y) \in D_{\mathrm{syn}}} w(x,y) \mathcal{L}_{\mathrm{CE}}(f_{\theta'}(x, K), y)$$

This process converts matured, high-utility instruction-space gains into the core parameter space. As shown explicitly, small instruction edits induce bounded, low-rank weight updates comparable to LoRA/IA³. The protocol achieves 2.4–5.0× throughput increases in enterprise SRE support and ~80% hallucination reduction, validating the efficacy of policy-driven, feedback-gated instruction refinement and subsequent weight-space integration (Costa, 29 Aug 2025).

6. Empirical Evaluation, Limitations, and Generalizability

Weight-space skill injection techniques are validated on a variety of QA, reasoning, support, and agentic tool use tasks. Trade-offs between skill transfer, old task retention, and interference are quantitatively assessed, revealing:

• EWC-based methods prevent forgetting with negligible impact on new skill convergence.
• PaST and task arithmetic pipelines reliably transfer RL or specialized reasoning skills, with alignment enabling transfers even across divergent architectures or model families.
• ILWS delivers dynamic, auditable adaptation by integrating instruction- and weight-space techniques.

Documented limitations include the restriction of certain methods to arithmetic skills (e.g., EWC studies address only addition/subtraction), the approximation quality of diagonal Fisher for parameter importance, the risk of under-transfer without iterative vector extraction, and the need for robust parameter-space alignment when models incorporate advanced features such as GQA and SwiGLU.

7. Future Directions

Research trajectories in weight-space skill injection recommend:

• Extending skill transfer frameworks to broader algebraic and symbolic tasks, such as logic and domain-specialized reasoning.
• Employing richer posterior approximations (Kronecker-factored or subspace Gaussians) for tighter old skill preservation.
• Exploring continual learning priors and dynamic or per-layer adaptation coefficients for finer retention-control.
• Systematizing activation-based versus weight-based alignment, and automating prompt selection for activation alignment.

These developments reflect the maturation of weight-space skill injection as a central paradigm for efficient, safe, and modular adaptation of large-scale language and reasoning systems (Sharma et al., 2022, Tang et al., 16 Jan 2026, Horoi et al., 13 Nov 2025, Costa, 29 Aug 2025).