Task Arithmetic: Model Editing Paradigm

• Task Arithmetic is a method that represents task-specific model changes as weight differences, facilitating efficient editing without retraining.
• Arithmetic operations on task vectors enable multi-task learning, effective task forgetting, and analogy-based skill synthesis.
• TA leverages linear mode connectivity, weight disentanglement, and local orthogonality to drive innovations in continual, federated, and compressed learning.

Task Arithmetic (TA) is a training-free model editing paradigm in which task-specific capabilities are abstracted as vectors in weight space—formed by subtracting the parameters of a pre-trained model from those of fine-tuned models. Arithmetic operations (addition, negation, scalar multiplication) on these vectors enable efficient model merging for multi-task learning, task forgetting, transfer, and even analogy-based synthesis without retraining or access to original data. TA’s success is theoretically underpinned by linear mode connectivity, weight disentanglement, and local orthogonality in the tangent space, and it has fueled diverse innovations in continual learning, federated learning, compressed vector bases, fairness control, explainability transfer, and speech-language expansion.

1. Mathematical Definition and Core Principles

Task Arithmetic represents downstream tasks by “task vectors” $\tau_t$ computed as the weight difference between a fine-tuned model $\theta_t$ and a shared base $\theta_0$: $\tau_t = \theta_t - \theta_0$ A merged model supporting $T$ tasks is constructed as a linear combination

$$\theta_\mathrm{merge} = \theta_0 + \sum_{t=1}^T \alpha_t \tau_t$$

where $\{\alpha_t\}$ are scalar coefficients detected by grid search, closed-form solution, or theoretical analysis (Ilharco et al., 2022, Ortiz-Jimenez et al., 2023, Chen et al., 27 Feb 2025, Dai et al., 15 Apr 2025). The three canonical operations are:

• Addition (multi-task): superpose vectors from multiple tasks to create a model competent on all.
• Negation (forgetting): subtract a task vector to suppress a capability, e.g., debiasing or unlearning.
• Analogy: infer new skills $D$ via $\tau_D \approx \tau_C + (\tau_B - \tau_A)$ when tasks exhibit linear relationships (Ilharco et al., 2022).

This arithmetic is model-agnostic—applicable to both dense and adapter-based networks—and leverages the observed linear mode connectivity of neural loss landscapes (Ilharco et al., 2022, Ortiz-Jimenez et al., 2023).

2. Weight Disentanglement, Orthogonality, and Tangent Space Linearization

TA effectiveness is governed by weight disentanglement, the property that each task vector mainly manipulates its own local domain in the input space. For a perfectly disentangled model $F(x; \theta_0 + \sum_t \alpha_t \tau_t)$, its output on $x \in D_t$ depends only on $\tau_t$ (Ortiz-Jimenez et al., 2023). This is formalized by the Task Arithmetic Property

$$F(x; \theta_0 + \sum_t \alpha_t \tau_t) = F(x; \theta_0 + \alpha_t \tau_t) \,\,\text{if} \,\, x \in D_t$$

A Taylor expansion shows first-order merging gap

$$G_i \approx k_i \sum_{j \neq i} \alpha_j \langle \tau_i, \tau_j \rangle$$

is minimized when task vectors are pairwise orthogonal (Xiong et al., 27 Nov 2024).

Tangent-space Task Arithmetic (NTK regime): TA performance is amplified under linearization, where the neural tangent kernel (NTK) $$k_\mathrm{NTK}(x, x') = \nabla_\theta f(x; \theta_0)^\top \nabla_\theta f(x'; \theta_0)$$ localizes eigenfunctions in support of each task, further disentangling vectors (Ortiz-Jimenez et al., 2023, Jin et al., 9 Jul 2024).

Selective module-wise fine-tuning (e.g., attention-only adapters) exploits naturally linear subspaces to realize high orthogonality and efficient multi-task adaptation (Jin et al., 9 Jul 2024, Dai et al., 15 Apr 2025). Empirical studies confirm that representation backbones support disentanglement, while task heads degrade it.

3. Advanced Model Merging: Layer-Aware, Subspace, and Trust-Region Extensions

Vanilla TA merges all task vectors naively, but several refinements exist:

• Layer-Aware Task Arithmetic (LATA): Assigns per-layer weights based on the cosine similarity between each layer's task and instruction-following components, amplifying task-specific knowledge while attenuating generic instruction patterns. Variants include Linear-Drop-by-Rank, Log-Drop-by-Rank, and Drop-with-Threshold (Chen et al., 27 Feb 2025). LATA robustly improves utility metrics (perplexity, GSM8K/code scores) and enables precise task forgetting.
• Task Arithmetic in Trust Region (TATR): Mitigates knowledge conflicts by projecting unconstrained task vector sums into a trust region defined by each task’s loss gradient. The merged update $$\delta^* = \mathrm{argmin}_{\delta \in \mathbb{R}^d} \frac{1}{2}\|\delta-u\|^2$$ subject to $|g_i^T \delta| \leq \epsilon_i$ (for all $i$) enforces minimal cross-task interference (Sun et al., 25 Jan 2025).
• Submodule Linearity: Leveraging linearity at the level of model submodules (e.g., layers, attentions, MLP blocks) allows canonical closed-form estimation of optimal merging weights, yielding non-trivial performance gains compared to global TA (Dai et al., 15 Apr 2025). Statistical analysis (non-linearity score, projection distance) supports this decomposition.

4. Scalability and Compression: Task Vector Bases and Continual Learning

Maintaining full task vector collections is space and compute-intensive for large $T$. Task Vector Bases compress $T$ task vectors into $M$ learned bases $b_j$, reconstructing each $\tau_i \approx \sum_j c_{i,j} b_j$ with theoretical guarantees on generalization and unlearning (Zeng et al., 3 Feb 2025). This enables TA operations in coefficient space, supports large task libraries, and even improves addition accuracy by mitigating interference.

In continual learning, LoRA-based TA stores only low-rank adapters $\Delta W_i = B_i A_i$ per task, merges via simple vector addition, and circumvents catastrophic forgetting, recovering near-offline accuracy with up to $5 \times$ reduced FLOPs (Chitale et al., 2023).

5. Applications and Problem-Specific Adaptations

TA has been adopted and analyzed for diverse problems:

• Zero-Shot Information Retrieval: TA transfers domain knowledge (e.g., biomedical, multilingual) and re-ranks via arithmetic with domain vectors, improving NDCG@10 and P@10 by up to $18\%$ and $15\%$ over strong baselines (Braga et al., 1 May 2025). Scaling coefficients are optimally tuned per target.
• Fairness Control: Scalar or targeted injection of subgroup task vectors enables direct trade-off between predictive accuracy and group fairness metrics (Demographic Parity, Equalized Odds), with "Editor’s term" being "coefficient-based fairness control" (Naganuma et al., 30 May 2025). TA enhances fairness on hate-speech domains via simple grid search.
• Molecular Design: "Molecular Task Arithmetic" learns property directions by fine-tuning on abundant negative samples, then moves models in the opposite direction to generate positive molecules, supporting zero-shot and few-shot design tasks (Özçelik et al., 23 Jul 2025). TA consistently increases design diversity and successful clusters over standard fine-tuning.
• Visual Explainability Transfer: Task arithmetic enables transfer of “explainability vectors” (the difference between models with and without explanation supervision) to new domains, allowing efficient prediction-plus-explanation (self-explaining) models with quality rivaling expensive inference algorithms (Yoshikawa et al., 6 Jul 2025). Universality is demonstrated with a single vector learned on ImageNet+X.
• Speech Translation Language Expansion: TA merges multiple ST task adapters adding a language-control vector to avoid language confusion, or synthesizes new tasks by analogy over MT/ST pairs. Merging adapters with TA plus language control recovers BLEU improvements up to +4.66 on MuST-C (Cheng et al., 17 Sep 2024).

6. Limitations, Theory, and Robust Model Merging

TA is formally equivalent to one-shot Federated Averaging (FedAvg), and thus inherits its dependence on data and training heterogeneity. Data heterogeneity induces a persistent error term in multi-task objectives, while training heterogeneity yields objective inconsistency (Tao et al., 27 Nov 2024). Adaptations from Federated Learning—FedNova, coordinate-wise Median, gradient mask aggregation, centered clipping—significantly improve TA’s performance and robustness without retraining.

Adaptive Weight Disentanglement (AWD): Post-hoc orthogonalization of task vectors by extracting and subtracting a learned redundant component $\delta$ reduces cosine similarity and enhances fusion: gains up to +6.1 absolute accuracy over TA and +2.8 over TIES-Merging (Xiong et al., 27 Nov 2024).

Selective Task Arithmetic (STA): Masks each task vector by a loss-sensitive importance score (first-order Taylor) before fusion, eliminating noisy updates, reducing hyperparameter tuning, and enabling precise task forgetting (Bowen et al., 25 Nov 2024). STA yields fusion accuracy gains up to +7.6 points over TA and improved control accuracy in forgetting experiments.

7. Practical Guidelines and Future Directions

• Coefficient Tuning: Optimal scaling coefficients $\alpha_t$ vary by application; grid search or closed-form solutions are recommended (Zhou et al., 17 Jun 2024, Zeng et al., 3 Feb 2025, Zhou et al., 17 Jun 2024).
• Orthogonality and Selective Fusion: Explicitly orthogonalize or mask vectors to reduce interference.
• Submodule and Layer-Level Merging: Prefer submodule-specific weight estimation for high model linearity (Dai et al., 15 Apr 2025).
• Compression and Continual Learning: Use task vector bases or LoRA adapters to scale to many tasks.
• Federated Learning Insights: Minimize data and training heterogeneity in fine-tuning for optimal TA merge outcomes (Tao et al., 27 Nov 2024).
• Model Exclusivity and Privacy: MetaGPT provides data-agnostic, closed-form coefficient estimation for multi-task LLM merging (Zhou et al., 17 Jun 2024).
• Disentanglement Analysis: Use tangent-space and attention-only fine-tuning for both weight disentanglement and parameter efficiency (Jin et al., 9 Jul 2024).

Promising directions include: second-order (Hessian-aware) trust-regions, dynamic bases, continual online selective arithmetic, cross-modal transfer, and multi-objective vector scaling for fairness or robustness (Sun et al., 25 Jan 2025, Zeng et al., 3 Feb 2025, Bowen et al., 25 Nov 2024).