Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution

Abstract: Continual Model Merging (CMM) sequentially integrates task-specific models into a unified architecture without intensive retraining. However, existing CMM methods are hindered by a fundamental saturation-redundancy dilemma: backbone-centric approaches face parameter saturation and representation interference within fixed capacities, whereas Mixture-of-Experts (MoE) variants resort to indiscriminate expansion, incurring expert redundancy and a routing bottleneck reliant on additional data-driven optimization. To resolve these challenges, we propose MADE-IT (Manifold-Aware Dynamic Expert Evolution and Implicit rouTing), an adaptive CMM method that orchestrates expert management and activation by grounding intrinsic expert representations in manifold geometry. We introduce a projection-based subspace affinity metric coupled with a distribution-aware adaptive threshold mechanism to guide autonomous expert evolution, harmonizing diversity with architectural parsimony. Furthermore, to bypass parameterized gating networks, we design a data-free and training-free implicit routing mechanism that activates experts via feature-subspace alignment. Extensive experiments demonstrate that MADE-IT consistently outperforms strong baselines in accuracy and robustness across long-horizon and shuffled task sequences, while significantly pruning redundant experts, particularly within generic modules and early layers.

• The paper introduces MADE-IT, a framework leveraging manifold geometry and truncated SVD to merge task-specific models without data retraining.
• It employs a projection-based subspace affinity metric to autonomously consolidate experts, reducing redundancy while maintaining task performance.
• Empirical results on ViT backbones show up to 95% expert reduction, improved average accuracy, and minimized catastrophic forgetting.

Adaptive Continual Model Merging via Manifold-Aware Expert Evolution

Motivation and Problem Statement

The proliferation of task-specific models derived from pre-trained backbones rapidly exacerbates storage and deployment complexity in large-scale machine learning ecosystems. Conventional model merging methods, predicated on the assumption of batch model availability, are ill-suited for sequential or streaming scenarios due to catastrophic forgetting and parameter saturation. Continual Model Merging (CMM) seeks to sequentially integrate task-specialized models into a single architecture, but current CMM formulations manifest a saturation-redundancy dilemma: backbone-centric aggregations induce severe capacity saturation and interference, while Mixture-of-Experts (MoE) schemes suffer from unchecked expert proliferation and routing bottlenecks. Critically, previous MoE merging approaches depend on explicit, parameterized gating, which necessitates data access and retraining and thus undermines the adaptability of CMM in strictly data-free, training-free environments.

MADE-IT: Manifold-Aware Dynamic Expert Evolution and Implicit Routing

The paper introduces MADE-IT, an adaptive CMM framework that addresses both redundancy and routing constraints through manifold geometry and geometric expert evolution. The approach leverages low-rank priors in fine-tuning updates and considers each modular expert as a transformation-invariant subspace on the Grassmann manifold. Key methodological innovations encompass:

• Dynamic Expert Evolution via Subspace Affinity: Each incoming task induces modular weight updates, which are decomposed via truncated SVD into principal input and output subspaces. These are represented as points on the Grassmann manifold, effectively denoising and compressing task increments.
• Projection-Based Subspace Affinity Metric: Expert redundancy/consolidation is resolved by calculating the Hilbert-Schmidt inner product between projection operators tied to each subspace, yielding a scale-invariant, transformation-agnostic affinity metric. This enables the autonomous consolidation or creation of modular experts, harmonizing diversity and parsimony.
• Adaptive Thresholding: A distribution-aware adaptive threshold, calibrated from historical affinity scores, determines whether to merge or instantiate experts, producing a differentiated and context-sensitive evolution trajectory for each module.
• Implicit Routing Mechanism: During inference, the most suitable expert is chosen per module by evaluating the projection alignment (cosine similarity) between incoming feature activations and expert input subspaces, obviating the need for explicit gating networks. A hierarchical expert dependency graph ensures global semantic path consistency across modules, resolving bifurcations through propagated task identity constraints.

  Figure 1: Overview of MADE-IT, including manifold-based expert evolution via module-wise SVD and implicit expert routing through feature-subspace projection alignment.

Empirical Results and Analysis

MADE-IT is evaluated on sequential CMM benchmarks built from ViT-B/32, ViT-B/16, and ViT-L/14 backbones, covering 8, 14, and 20-task settings. The method delivers strong numerical results, consistently outperforming all evaluated baselines in both average accuracy (ACC) and backward transfer (BWT). In the 20-task regime with ViT-B/32, MADE-IT achieves 81.4% ACC, surpassing MINGLE by 4.3% and OPCM by 15.7%. Comparable improvements are observed across deeper architectures, with MADE-IT narrowing the gap to individually fine-tuned models.

An essential characteristic of MADE-IT is its generic-to-specific expert allocation hierarchy: shallow modules consolidate universal experts with high geometric overlap, while deeper layers retain specialized experts for task fidelity. This behavior is visualized through module-wise expert evolution trajectories.

Figure 2: Module-wise visualization of MADE-IT expert evolution over 20 tasks using ViT-L/14, highlighting shared (universal) vs. specialized expert allocation.

Quantitative analysis confirms high expert reduction rates—up to 95% in generic modules—while MLP modules and deeper layers preserve greater diversity, facilitating both parameter efficiency and plasticity.

Figure 3: Quantitative expert allocation analysis; MLPs retain highest expert diversity, reduction rates peak in generic modules and shallow layers.

Critically, the projection-based subspace affinity metric provides vastly superior discriminability for expert redundancy detection compared to parameter-space cosine similarity, yielding higher multi-task accuracy and less forgetting.

Figure 4: Similarity heatmaps for mlp.fc1 and self_attn.k.proj (ViT-B/32, layer 6), showing affinity-based clustering of related tasks.

Further analysis reveals that the subspace affinity metric produces a broader, more informative distribution of pairwise expert similarity scores, while cosine similarity collapses to near-zero due to ambient parameter orthogonality.

Figure 5: Density distribution of pairwise similarity scores, illustrating superior inter-expert discriminability with projection-based affinity.

Hyperparameter Sensitivity

MADE-IT's robustness is evidenced by sensitivity studies of key hyperparameters. For rank ratio $\rho$, performance plateaus for $\rho \leq 0.6$, but degradation ensues as ranks approach unity due to loss of geometric specificity and injection of task-irrelevant noise. The margin coefficient $\beta$ supports diverse ensemble formation for $\beta \leq 1.0$, but further increases do not translate to performance gains and instead increase redundant parameter counts.

Figure 6: Sensitivity of ACC and BWT to rank ratio $\rho$ and margin coefficient $\beta$ for ViT-L/14.

Implications and Future Directions

MADE-IT advances CMM by introducing a geometric, subspace-centric framework for expert evolution and implicit activation. The manifold-based affinity metric addresses functional equivalence and redundancy detection beyond ambient parameter similarity, suggesting extensibility to other continual learning and parameter-efficient adaptation settings. The implicit routing mechanism's data-free, training-free property provides a practical solution for large-scale, privacy-sensitive deployments and paves the way for modular, scalable architectures unconstrained by explicit gating.

Theoretically, MADE-IT demonstrates how transformation-invariant subspace representations enable robust, efficient merging of functionally correlated models, which may inform future work in modular LLMs, federated adaptation, and dynamic ensembles. Practically, its strong empirical gains and architectural parsimony propose a framework for efficient sequential model integration without catastrophic forgetting, excessive parameter growth, or retraining requirements.

Conclusion

MADE-IT reframes continual model merging through manifold geometry, subspace affinity-driven expert management, and training-free implicit routing. The framework achieves state-of-the-art multi-task accuracy and robustness in long-horizon, sequential merging, while efficiently pruning redundant experts, particularly in generic modules and early layers. Its methodological contributions and empirical strengths establish a foundation for adaptive, scalable, and privacy-preserving continual model merging (2604.22464).