Core Space Merging Framework

• Core Space Merging is a method that projects model updates into a compact, aligned joint basis, ensuring lossless linear merging.
• It employs joint SVD and alignment techniques to reduce computational complexity dramatically compared to traditional averaging.
• Empirical benchmarks show significant accuracy and efficiency improvements, with up to 600× speedup in merging tasks.

A Core Space Merging Framework is an algorithmic and mathematical methodology for merging information-rich, structured representations—such as neural network adaptations, graph succinct data structures, or model weight matrices—within an aligned, reduced-dimensional joint basis (“core space”) rather than operating directly in original, potentially uncoordinated spaces. The concept addresses key limitations in traditional merging approaches, enabling both higher accuracy and dramatic efficiency gains in resource-constrained environments. This aligned core space captures task-specific adaptations or representations in a form that supports high-fidelity combination, guarantees no loss of crucial information under linear strategies, and vastly reduces computational complexity.

1. Definition and Motivation for Core Space Merging

Core Space Merging refers to projecting multiple input objects—such as low-rank weight update matrices (LoRA), full-rank model deltas, or graph representations—into a shared aligned subspace, called the core space, prior to performing any merging operation. Technically, this involves decomposing the parameter (or update) matrices of each model into the same basis via joint Singular Value Decomposition (SVD), or by constructing an alignment using SVD on stacked matrices (e.g., for LoRA, via joint SVD of the $A$ and $B$ matrices). Each model’s adaptation is then represented by a compact “core” matrix that encodes its contribution in the aligned space.

Traditional merging approaches, such as entrywise averaging or direct arithmetic in the original parameter space, are prone to “interference”—loss of fidelity due to random alignment or polysemanticity in fine-tuned weights. Core Space Merging formalizes a procedure that prevents this interference by constructing a mathematically optimal basis for combination, which can be shown to be lossless for linear merging functions.

2. Mathematical Construction

For LoRA low-rank adaptation, each task $t$ provides weight updates $B^{(t)} A^{(t)}$, where $A^{(t)} \in \mathbb{R}^{r \times n}$ and $B^{(t)} \in \mathbb{R}^{m \times r}$. The Core Space procedure is:

• Stack the $A^{(t)}$ and $B^{(t)}$ matrices (across all $T$ tasks).
• Compute joint SVDs:

$$A^{(t)} = U_A^{(t)} \Sigma_A^{(t)} V_A^{(t)\top} \qquad B^{(t)} = U_B^{(t)} \Sigma_B^{(t)} V_B^{(t)\top}$$

• Define a core matrix per task:

$$M^{(t)} = \Sigma_B^{(t)} V_B^{(t)\top} U_A^{(t)} \Sigma_A^{(t)}$$

• Realign each $M^{(t)}$ into a common reference basis via linear transformation using the reference low-rank bases (from SVD of stacked $A^{(t)}$, $B^{(t)}$). After alignment:

$\tilde{M}^{(t)} = R_B^{(t)} M^{(t)} Q_A^{(t)\top}$

All merging operations are performed among $\tilde{M}^{(t)}$ in the compact $\mathbb{R}^{Tr \times Tr}$ core space.

For full-rank delta-based merging (Chaichana et al., 29 May 2025), the process is analogous: stack the weight deltas across tasks, perform SVD to obtain a joint basis $U, \Sigma, V^\top$, and partition $V^\top$ into task-specific blocks.

3. Guarantees and Formal Properties

The framework provides a formal proof of information preservation in linear merging strategies: merging in core space ($\tilde{M}^{(t)}$) and then reconstructing via reference bases ($B_{\text{ref}}, A_{\text{ref}}$) yields exactly the same update as merging in the original space. Specifically,

$$\mathcal{M}(\{\Delta W^{(t)}\}) = B_{\text{ref}} \cdot \mathcal{M}(\{\tilde{M}^{(t)}\}) \cdot A_{\text{ref}}^{\top}$$

where $\mathcal{M}$ is any linear merge function. The framework presents bounds on reconstruction error for non-ideal alignment and empirical evidence that the error is negligible for practical settings and all tested architectures. Efficiency is also theoretically analyzed—merging complexity in core space is $O(n^2 T r)$ for LoRA-adapted layers (with $T, r \ll n$), whereas previous alignment-based methods scale as $O(n^3 T^2)$.

4. Algorithmic Procedures and Complexity

The key steps in a typical implementation are:

1. Preprocessing: For each fine-tuned task, extract the low-rank update matrices or compute weight delta. Normalize and secure numerical stability via renormalization of basis vectors (Chaichana et al., 29 May 2025).
2. Alignment: Compute global SVD (across tasks) to generate reference bases. Map all task deltas/updates into the aligned core space.
3. Merging: Apply chosen merge strategy (e.g., arithmetic averaging, non-linear strategies like TSV or Iso-C) directly within the core space.
4. Reconstruction: Map the merged core representation back to the original parameter space via the reference bases, assemble the merged model.

The framework’s scalable implementation is publicly available (Panariello et al., 22 Sep 2025), including PyTorch pseudocode for each stage. Notably, merging multiple LoRA adaptations of Llama 3 8B was executed with up to a 600$\times$ speedup over KnOTS-space methods.

5. Empirical Performance and Benchmarking

Performance metrics reported include normalized accuracy (ratio of merged to fine-tuned model accuracy) and merging/inference runtime. Experiments spanned large-scale LLMs (Llama 3 8B), vision backbones (ViT-B/32, ViT-L/14), and established LoRA benchmarks. Findings demonstrate:

• Core Space strategies consistently outperform KnOTS-space and full-space merges in normalized accuracy.
• Efficiency gains, both computational and memory, permit scaling to dozens of tasks and large models.
• For non-linear merges, running them in core space results in higher Subspace Alignment Ratio (SAR) and better generalization across tasks.

6. Implications, Limitations, and Applications

Core Space Merging advances both the theory and practice of model combination in parameter-efficient transfer learning and multitask scenarios. By guaranteeing information preservation for linear mergers and demonstrating strong empirical results for non-linear approaches, it facilitates:

• Efficient multitask model construction by merging LoRA-adapted models;
• Generalization across diverse data modalities, including vision and language tasks;
• Deployment in resource-limited environments where scaling prior merging algorithms was prohibitive.

A plausible implication is that the approach may generalize to other types of model adaptation (e.g., adapters, prompt tuning) given an appropriate basis construction.

The framework’s limitations are bound to the dimensionality of the core space ($Tr$) and the assumptions underlying linearity and alignment error. Non-linear or highly entangled adaptation functions may not fully benefit without further research.

7. Comparative Table and Implementation

Below is a comparative summary of key properties:

Method	Space Complexity	Info Preservation	Efficiency
Core Space	$O(n^2 T r)$	Proven (linear)	High (600$\times$ speedup over KnOTS)
KnOTS/Align	$O(n^3 T^2)$	Not guaranteed	Moderate
Full Space	$O(n^3)$	Not guaranteed	Low

Implementations are available (Panariello et al., 22 Sep 2025) and provide reference pipelines for efficient merging in various architectures.

Summary

Core Space Merging Frameworks represent a convergence of mathematical insight and efficient algorithmic engineering. By aligning model adaptations in a compact joint basis, they enable rapid, information-preserving merges of diverse models in multitask learning, large-scale transfer learning, and resource-constrained deployment. Theoretical proofs, complexity analysis, empirical benchmarking, and open-source codebases collectively substantiate the framework’s impact on efficient model merging practice.