---
title: Orthogonal Iterative Merging (OIM)
url: https://www.emergentmind.com/topics/orthogonal-iterative-merging-oim
type: topic
---

# Orthogonal Iterative Merging (OIM)

Orthogonal Iterative Merging (OIM) is a class of training‐free, geometry-aware algorithms that sequentially integrate a collection of fine-tuned or specialized models into a single unified model. OIM enforces mutual orthogonality of task-specific updates in parameter space, thereby minimizing interference and catastrophic forgetting when merging knowledge from diverse tasks. Contemporary OIM methods are prominent in continual learning, model merging, and modular systems for deep neural networks, exhibiting empirical gains in accuracy, stability, and reversibility across a range of architectures and benchmark scenarios [2501.09522, 2507.20997, 2602.05943, 2411.16101].

## 1. Mathematical and Geometric Foundations

OIM is rooted in orthogonalization processes and projections in high-dimensional parameter spaces. For neural models with parameter vector $\theta_0 \in \mathbb{R}^d$ (the shared base), independently fine-tuned models $\theta_i$ ($i=1,\ldots,N$) define "task deltas" $\Delta_i = \theta_i - \theta_0$. Orthogonal iterative merging enforces $\langle\Delta_i^\perp,\Delta_j^\perp\rangle=0$ for $i\neq j$, so each merged task's update lies outside the subspace of previously integrated deltas.

For specialized settings where the model weights are linear operators, OIM extends to the orthogonal group $O(n)=\{O\in\mathbb{R}^{n\times n}:O^\top O=I\}$, and uses the associated Lie algebra so$(n)$ of skew-symmetric matrices. In this regime, merging operations are performed on the Riemannian manifold of the orthogonal group, preserving geometric structure of the weight tensors and enabling principled manifold-aware integration of orthogonal adaptations [2602.05943].

## 2. Core OIM Algorithms

The generic OIM workflow is as follows:

1. **Orthogonalization:** Each task delta $\Delta_i$ is projected onto the orthogonal complement of the subspace spanned by previous deltas, typically via Gram–Schmidt:
   $$
   \Delta_i^\perp = \Delta_i - \sum_{j=1}^{i-1} \frac{\langle\Delta_i, \Delta_j^\perp\rangle}{\|\Delta_j^\perp\|^2} \Delta_j^\perp.
   $$
   High-dimensional variants may employ SVD-based truncation [2501.09522] or block orthogonalization [2507.20997].

2. **Iterative merging:** The merged parameters are formed by adding all orthogonalized deltas to the base, with optional learnable combination coefficients to balance performance:
   $$
   \theta^* = \theta_0 + \sum_{i=1}^N \alpha_i \Delta_i^\perp,
   $$
   with the $\alpha_i$ typically learned by minimizing an aggregate objective across task losses with a consistency penalty [2507.20997].

3. **Gradient-based refinement:** OIM optionally incorporates a further stage of joint optimization over $\theta$ to directly minimize the sum of per-task validation losses plus a quadratic regularization towards the naive sum-of-orthogonally-merged deltas.

4. **Continual and modular update:** New tasks can be added incrementally by projecting their delta against the current orthogonal set and repeating the merge, without revisiting or re-projecting previous deltas. Removal ("unmerge") of a task is achieved by subtracting its orthogonalized delta, ensuring minimal collateral effect on other tasks [2507.20997].

For model weights representable as orthogonal matrices (e.g., after Orthogonal Fine-Tuning, OFT), merging is done on the orthogonal group via Lie algebra averaging. Each expert model's orthogonal update $O_i$ is mapped to so$(n)$ by $Q_i=\log O_i$, averaged as $\overline{Q}=\sum w_i Q_i$, optionally norm-corrected, and mapped back to $O(n)$ by $\exp(\overline{Q})$. If models are not OFT-trained, OIM applies the orthogonal Procrustes solution to extract the nearest orthogonal component and merges the residual in weight space [2602.05943].

## 3. Theoretical Properties and Analysis

A primary property of OIM is provable minimization of task interference. By construction, new task deltas are orthogonal in parameter space to those of previous tasks,
$$
\langle\Delta_i^\perp, \Delta_j^\perp\rangle = 0,\quad i\neq j,
$$
which, under quadratic or locally linearized loss, ensures that cross-terms in the merged loss are eliminated, minimizing backward transfer (BWT) degradation [2501.09522, 2507.20997].

For vector sets, OIM-type randomized pairwise orthogonalization (akin to Kaczmarz-type methods) converges almost surely to an orthonormal basis, increasing the $n$-dimensional volume $\det(A^\top A)^{1/2}\to 1$, and attaining $\varepsilon$-volume accuracy in $O(n^2\log(1/\varepsilon))$ pairwise steps [2411.16101].

In manifold settings, OIM methods operate on the canonical Riemannian metric of $O(n)$, with merging via geodesic averaging in the Lie algebra. The merged update remains on the orthogonal group, ensuring preservation of weight geometry and hyperspherical energy, and mitigating norm collapse via explicit magnitude correction [2602.05943].

Under continual integration, the norm of the aggregate merged update remains bounded by the largest task drift, and memory complexity is $O(|\theta|)$ per merge, independent of the number of tasks [2501.09522].

## 4. Computational and Practical Considerations

OIM is designed for scalability and memory efficiency. Only the current base, merged parameters, and the incoming expert model are required at each step, with memory overhead independent of the total number of tasks [2501.09522, 2507.20997]. Projection and orthogonalization cost is $O(d k)$ per step (with $d$ model dimension and $k$ number of tasks), or $O(\min(m,n)mn)$ per-layer for SVD operations. Large-scale variants exploit sparse representations or blockwise processing.

Parallelization is straightforward: pairwise or blockwise orthogonalizations can be performed on disjoint parameter subsets [2411.16101], and projection steps can be implemented efficiently for high-dimensional and sparse deltas. Gradient-based merging phase admits standard optimizers, early stopping, and validation-based model selection [2507.20997].

Elastic Weight Consolidation (EWC) and synthetic replay mechanisms can be layered atop OIM's merge objective, further stabilizing model parameters and preventing drift away from $\theta_0$ during repeated merges [2507.20997].

## 5. Empirical Performance and Comparative Findings

OIM has been extensively validated on vision (CIFAR-100, ImageNet-100), language (AG News, Yahoo, DBpedia), and multimodal (CLIP-ViT) benchmarks. Across these domains, OIM achieves:

- **Higher average accuracy**: E.g., OIM achieves 78.4% average accuracy on continual CIFAR-100 compared to 72.1% for prior TIES-Merging [2507.20997].
- **Reduced catastrophic forgetting**: Positive BWT (+0.12), indicating prior task performance is preserved after new merges [2507.20997].
- **Near-constant memory and rapid composition**: Recovery and unmerge times are consistently lower than retraining-based alternatives.
- **Interference minimization and reversibility**: Empirical results demonstrate that OIM’s orthogonal projections enable unmerging of tasks with minimal accuracy drop (1.8% vs. 8–15% for baselines) [2507.20997].

In the large language model setting, Orthogonal Model Merging (OrthoMerge, leveraging the OIM paradigm) has demonstrated superior task synergy and less forgetting compared to Euclidean model arithmetic and norm-uncorrected manifold averaging. Merging five OFT-trained Llama-3.1-8B experts, OIM achieved 46.25% average accuracy over 41.46% for task arithmetic and 44.71% for expert averaging. The advantage grows with increased task count [2602.05943].

Ablations show that omitting orthogonal projections, magnitude correction, or stability penalties results in substantial accuracy degradation.

## 6. Extensions, Applications, and Limitations

OIM's modular design enables extensions to:

- **Model compliance and reversibility**: OIM enables not only continual integration but also the ability to "unmerge" selected task adaptations efficiently, crucial for compliance with privacy and data retention requirements [2507.20997].
- **Hybrid merge strategies**: For models not natively OFT-trained, OIM with Orthogonal–Residual Decoupling solves an orthogonal Procrustes problem to extract group elements, merging on the manifold and in Euclidean space for residuals [2602.05943].
- **Streaming and blockwise orthogonalization**: Streaming data or large parameter sets can be handled by online or blockwise variants [2411.16101].

Limitations include increased per-step computation over naive interpolation methods due to orthogonalization and optimization steps, storage of orthogonal deltas for unmerge capability, and possible sensitivity to parameter sparsity in extremely high-dimensional settings.

## 7. Relationship to Classical and Contemporary Orthogonalization Methods

OIM is conceptually related to classical Gram–Schmidt and Householder QR, but offers distinct advantages in randomized, parallel, and modular settings. Pairwise randomized (Kaczmarz-inspired) OIM is simple, parallelizable, and monotone in $n$-volume, though Householder QR remains the worst-case optimal for numerical precision. OIM's innovation is in extending these principles to model parameter space for scalable, interference-minimizing continual merging, offering a principled and robust foundation for modular AI systems [2411.16101, 2501.09522, 2507.20997].

Source: https://www.emergentmind.com/topics/orthogonal-iterative-merging-oim