Compatibility-Aware Parameter Splicing (CAPS)

• CAPS is a framework that integrates parameters from multiple deep models by evaluating local uncertainties and global entropy measures.
• It employs both hard and soft splicing mechanisms to blend model parameters smoothly, preserving complementary strengths while avoiding interference.
• Validated across recommendation, NLP, and multimodal tasks, CAPS improves performance metrics by up to 20% compared to traditional fusion methods.

Compatibility-Aware Parameter Splicing (CAPS) is a principled framework for fusing parameters from multiple deep learning models, with the objective of integrating complementary strengths while minimizing information loss and model interference. CAPS operates by assessing and leveraging both local and global compatibility signals between sets of model parameters, and then synthesizing a unified set of parameters through adaptive splicing procedures. Its methodology has been validated in Recommendation, Natural Language Processing, and Multimodal LLM (MLLM) settings. CAPS enables the creation of efficient, versatile hybrid models without incurring inference overhead, addressing the challenges of model specialization and parameter incompatibility.

1. Theoretical Foundation and Motivation

Deep neural network ensembles and parameter merging techniques often face critical trade-offs: ensemble methods demand linearly increasing inference costs with the number of models, while naïve parameter averaging or pruning either oversimplifies model knowledge or discards potentially valuable features. CAPS is motivated by the observation that independently trained models—whether fine-tuned on different domains, data splits, or objectives—possess unique, often complementary, knowledge. However, these parameters may be incompatible or misaligned; directly combining them can harm performance due to destructive interference.

CAPS addresses this by explicit compatibility assessment before integration, allowing the composition of models with heterogeneous strengths and the mitigation of underutilization or detrimental interaction among parameters.

2. Parameter Compatibility Assessment

The core of CAPS is a dual-perspective compatibility assessment, combining:

• Local-level Uncertainty: Quantifies differences at each parameter location, based on the hypothesis that consistent values across models indicate mutual reliability.

For any two parameter matrices $W_A$ and $W_B$ (e.g., weights from two models) at position $[i, j]$:

$V_L^{(A)}[i,j] = f_L(|W_A[i,j] - W_B[i,j]|)$

where $f_L$ may be the identity or a non-linear mapping. For $n$ models,

$$V_L^{(k)}[i,j] = \sum_{W_l \neq W_k} f_L(|W_k[i,j] - W_l[i,j]|)$$

• Global-level Entropy: Evaluates the richness of information encoded across all parameters, using entropy as a surrogate for diversity and capacity.

The entropy for a parameter matrix $W$ is

$E(W) = -\sum_{t=1}^u p_t \log p_t$

where $p_t$ is the proportion of parameters falling in histogram bin $t$.

The global compatibility between models $A$ and $B$:

$V_G^{(A)} = f_G(|E(W_A) - E(W_B)|)$

These perspectives are fused at each parameter location:

$$V^{(A)} = V_G^{(A)} \cdot [1-\exp(-V_G^{(A)} V_L^{(A)})]$$

A softmax-like normalization ensures that compatibilities across models sum to $1$ at each parameter:

$$V^{(A)} := \frac{\exp(V^{(A)})}{\sum_k \exp(V^{(k)})}$$

This approach is generalizable to multi-model settings and to modular components such as LoRA adapters within large models.

3. Parameter Splicing Mechanisms

After compatibility maps are established, CAPS combines parameters by splicing, offering two primary approaches:

• Hard Splicing: For each parameter, select the value from the model exhibiting the highest compatibility score at that location:

$$W[i,j] = \begin{cases} W_A[i,j], & \text{if } V^{(A)}[i,j] = 1 \ W_B[i,j], & \text{if } V^{(B)}[i,j] = 1 \end{cases}$$

This approach equates to a binary mask, favoring certainty but potentially causing sharp transitions.

• Soft Splicing: Linearly blend corresponding parameters across models, weighted by compatibility:

$$W[i,j] = \sum_{k} W_k[i,j] \cdot V^{(k)}[i,j], \qquad \sum_{k} V^{(k)}[i,j] = 1$$

This enables smoother integration and finer-grained knowledge synergy.

The splicing process is performed in a layerwise or adapterwise fashion when applied to modularized models.

4. Extension to Domain-specialized and Modular Large Models

Recent work has adapted CAPS for the fusion of multimodal LLMs (MLLMs) and their modular low-rank adaptation (LoRA) layers. In this context, compatibility signals are enriched with additional mechanisms:

• Channel-wise Functional Attribution: Assesses the contribution or difference of each output channel between base and expert (domain-specialized) modules. Channel-wise absolute differences are passed through a learnable gating network:

$$d_i = \sum_j |\mathbf{W}_b[i, j] - \mathbf{W}_g[i, j]|, \qquad \mathbf{w}_{\rm local} = \sigma(\phi(\mathbf{d}))$$

• Global Entropy-based Gating: Utilizes the difference in discrete entropy between modules to inform macro-level weighting:

$$w_{\rm global} = \frac{a}{c} \arctan(c[H(\mathbf{W}_b) - H(\mathbf{W}_g)]) + \frac{1}{2}$$

• Dual-gate Fusion: Both local and global signals determine the weights for final parameter integration:

$$\mathbf{W}_{\text{fused}} = w_b \odot \mathbf{W}_b + w_g \odot \mathbf{W}_g$$

Further, an activation-based compatibility filter, leveraging mean absolute activations, sparsity, and variance, pre-selects module pairs suitable for integration, empirically correlating ($\rho=0.86$) with actual fusion benefit.

The CAPS procedure operates with minimal inference overhead, as it is applied only to compact adaptation modules and requires no additional forward passes, enabling scalable, plug-and-play compositionality in expert-driven architectures.

5. Empirical Evidence and Performance Metrics

Experimental validation spans recommendation, NLP, and MLLM domains:

• Recommendation Tasks (Amazon-Beauty, Douban-Book, Douban-Music, Movielens-1M): CAPS (within CKI) consistently improves NDCG@5 by up to 10–20% over strong baselines, including model pruning and output ensembles.
• Language Tasks (SST-2, RTE): CAPS-integrated models achieve top results (e.g., SST-2 accuracy 0.8544 vs. best ensemble 0.8532), demonstrating that the framework surpasses or matches state-of-the-art aggregation methods.
• Multimodal Benchmarks (MathVista, HumanEval, MMMU, MME): In the Graft framework, CAPS-based fusion achieves results superior or comparable to prior domain merging techniques (e.g., MathVista 52.2% vs. 49.9% for best single expert).
• Efficiency: CAPS incurs no additional inference cost compared to a single model; unlike ensembles, it does not require concurrent evaluation or increased storage at inference.
• Ablation studies: Removing local or global compatibility signals reduces performance, verifying the necessity of both perspectives.

6. Practical Applications and Implications

CAPS offers utility in various concrete scenarios:

• Recommendation systems: Allows unification of models specialized for different user cohorts, time segments, or objectives, improving coverage and robustness under non-stationary data.
• Natural Language Processing: Enables aggregation of models with divergent pretraining histories or data domains, supporting robustness to distributional shifts and facilitating continual learning.
• Multimodal LLMs: Facilitates the modular integration of domain-specific expertise (e.g., mathematics, code, medicine), providing an efficient mechanism for reusing and compositing LoRA-based adapters.

The approach directly addresses the challenge of parameter incompatibility, yielding models better suited for dynamic, real-world environments, and serving as improved starting points for further fine-tuning.

7. Comparison with Alternative Integration Techniques

Aspect	CKI / CAPS	Model Pruning	Output Ensemble	Parameter Averaging
Goal	Optimize incompatible params	Remove parameters	Combine output predictions	Direct parameter fusion
Inference Cost	Unchanged (single model)	Reduced	Scales with number ($\times n$)	Unchanged
Performance	Best or near-best	Often degrades	Second-best	Often unstable
Resource Demand	No increase	Reduced	Increased	Same as base models
Granularity	Per-parameter, dual-view	Per-parameter	Output-level	Parameter-level
Innovation	Compatibility assessment (local/global)	Removes information	Leverages complementarity	Lacks compatibility bias

A distinctive feature of CAPS is its ability to fine-grain knowledge integration by weighing local parameter alignment and global information content, without the inefficiencies or performance trade-offs characteristic of prior methods.

8. Illustrative Formulations and Methodological Summary

Key mathematical formulations implemented in CAPS include:

• Local compatibility: $V_L^{(A)}[i,j] = f_L(|W_A[i,j] - W_B[i,j]|)$
• Global compatibility: $E(W) = -\sum_t p_t \log p_t$
• Dual fusion: $V^{(A)} = V_G^{(A)}[1-\exp(-V_G^{(A)} V_L^{(A)})]$, normalized across all models
• Soft parameter splicing: $W = \sum_k W_k \odot V^{(k)},\ \sum_k V^{(k)} = 1$

A typical pipeline involves:

1. Computing local and global compatibilities for each set of models.
2. Softmax normalization of compatibility scores at each parameter.
3. Either hard or soft splicing of parameters, based on application requirements and empirical performance.
4. Optional further fine-tuning of the fused model.

---

CAPS constitutes a systematic, efficient approach for parameter integration in deep neural architectures, grounded in quantitative compatibility estimation and experimentally validated to provide robust, scalable gains in complex learning scenarios. Its adoption enables the construction of hybrid models that efficiently synthesize domain knowledge, maintain inference efficiency, and yield strong empirical performance across a spectrum of AI tasks.