Ability Transfer and Recovery via Modularized Parameters Localization

Abstract: LLMs can be continually pre-trained or fine-tuned to improve performance in specific domains, languages, or skills, but this specialization often degrades other capabilities and may cause catastrophic forgetting. We investigate how abilities are distributed within LLM parameters by analyzing module activations under domain- and language-specific inputs for closely related models. Across layers and modules, we find that ability-related activations are highly concentrated in a small set of channels (typically <5\%), and these channels are largely disentangled with good sufficiency and stability. Building on these observations, we propose ACT (Activation-Guided Channel-wise Ability Transfer), which localizes ability-relevant channels via activation differences and selectively transfers only the corresponding parameters, followed by lightweight fine-tuning for compatibility. Experiments on multilingual mathematical and scientific reasoning show that ACT can recover forgotten abilities while preserving retained skills. It can also merge multiple specialized models to integrate several abilities into a single model with minimal interference. Our code and data will be publicly released.

• The paper demonstrates that ability-related activations in LLMs are sparsely concentrated in less than 5% of channels, enabling effective modular transfer.
• It introduces the ACT framework, which identifies and transfers specific ability channels using cross-model activation differences and masked task vectors.
• Empirical findings show that ACT recovers multilingual abilities with minimal parameter changes, surpassing traditional fine-tuning methods in performance.

"Ability Transfer and Recovery via Modularized Parameters Localization" Overview

The paper "Ability Transfer and Recovery via Modularized Parameters Localization" (2601.09398) provides an in-depth investigation into how different abilities are distributed within LLMs and introduces an innovative method, Activation-Guided Channel-wise Ability Transfer (ACT), to transfer these abilities efficiently without causing interference. This research is crucial for understanding the modularity in LLMs and mitigating the challenges of catastrophic forgetting during specialization.

Distribution and Localization of Abilities in LLMs

The authors rigorously explore the distribution of ability-related activations across layers and modules of LLMs. They discover that these activations are highly concentrated in less than 5% of channels, which are typically disentangled and exhibit sufficiency and stability. These channels, when identified using cross-model activation differences, indicate the possibility of recovering forgotten abilities while maintaining existing skills.

Introducing ACT: A Novel Framework

ACT is proposed to address the challenge of efficient ability transfer. This method localizes ability-relevant channels using activation differences between models and restricts parameter transfer to these identified channels. The framework follows a three-stage process: identifying ability-specific channels, transferring parameters using masked task vectors, and applying lightweight fine-tuning for model stabilization and compatibility.

Key Experimental Findings: The empirical results demonstrate that ACT effectively recovers forgotten multilingual abilities, showing superior performance compared to traditional fine-tuning and model merging techniques. The framework achieves significant performance with minimal parameter modification across various language-domain combinations.

Figure 1: Global CCDF showcasing the concentration of large activation differences in a small fraction of channels across different language-domain combinations.

Empirical Validation of Cross-Model Activation Analysis

Through robust experimentation, the authors validate the sparsity and heavy-tailed nature of activation differences within various models. The analyses reveal that although the activation differences are sparse, they are distributed throughout the network layers, maintaining consistency across different abilities and model scales.

Figure 2: Cross-pair consistency of ability-specific channels, demonstrating substantial overlap and stability across different LLM variations.

Implications and Future Directions

This research not only advances the understanding of how abilities are encapsulated within LLMs but also offers a practical solution for ability transfer without catastrophic interference. The implications of this work extend to improving specialization in LLMs while preserving versatility, crucial for developing models that can adapt and learn continually without forgetting.

Speculative Future Developments: Potential future directions include extending this ability transfer framework to models with varying architectures or pre-training corpora, thereby expanding its applicability across diverse AI systems. Additionally, exploring the activation-based structures in other ability categories beyond reasoning domains could further enhance the robustness and application scope of LLMs.

Conclusion

The paper presents a comprehensive study on the localization and efficient transfer of abilities within LLMs using the ACT framework. By leveraging modularized parameters and activation-guided identification, the proposed method sets a new benchmark in mitigating catastrophic forgetting while enhancing model specialization and integration. This work lays the groundwork for future innovations in dynamic and adaptive AI systems.