Efficient Representation Drift Repair and Alignment Across Dissimilar Models

Develop efficient methods for repairing representation drift and aligning intermediate representations across dissimilar source models in representation-level model fusion.

Background

Representation-level fusion uses intermediate hidden representations to guide parameter weighting, local matching, or target-model repair. Existing methods depend on calibration data, layer correspondence, compatible modules, and reliable representation similarity. The survey identifies efficient drift repair and alignment across dissimilar models as unresolved issues, particularly when source models differ in architecture, tokenizer, or internal representation structure.

References

Open problems include efficient drift repair and alignment across dissimilar models.

— From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models  (2609.19553 - Cai et al., 17 Sep 2026) in Section 3.2, Representation-Level Fusion

AIMMerging \citep{feng2025aimmergingadaptive}, NUFILT \citep{qiu2025nullspace}, and K-Merge \citep{shenaj2025kmerge} study continual fusion for LLMs, but stable long-term fusion remains open.

— From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models  (2609.19553 - Cai et al., 17 Sep 2026) in Appendix, Section \ref{app:continual_large_scale_challenges}, “Continual Fusion Can Easily Cause Forgetting”