Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities (2408.07666v4)

Abstract: Model merging is an efficient empowerment technique in the machine learning community that does not require the collection of raw training data and does not require expensive computation. As model merging becomes increasingly prevalent across various fields, it is crucial to understand the available model merging techniques comprehensively. However, there is a significant gap in the literature regarding a systematic and thorough review of these techniques. This survey provides a comprehensive overview of model merging methods and theories, their applications in various domains and settings, and future research directions. Specifically, we first propose a new taxonomic approach that exhaustively discusses existing model merging methods. Secondly, we discuss the application of model merging techniques in LLMs, multimodal LLMs, and 10+ machine learning subfields, including continual learning, multi-task learning, few-shot learning, etc. Finally, we highlight the remaining challenges of model merging and discuss future research directions. A comprehensive list of papers about model merging is available at \url{https://github.com/EnnengYang/Awesome-Model-Merging-Methods-Theories-Applications}.

• The paper presents a detailed taxonomy of model merging techniques, categorizing them into pre-merging and during-merging phases to reduce task interference.
• It examines theoretical frameworks like linear mode connectivity and flat minima generalization to explain the success of merging approaches.
• The study highlights diverse applications across LLMs, MLLMs, and generative models, demonstrating enhanced performance in multitask and domain-specific learning.

Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities

The paper "Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities" offers a comprehensive survey of model merging techniques, exploring their theoretical underpinnings, methodological advancements, and diverse applications. As model merging continually gains traction across the machine learning community, this paper fills a critical gap in the existing literature by systematically categorizing and analyzing the current state of the art in model merging.

Methods and Theories of Model Merging

Classification of Model Merging Methods

The paper classifies existing model merging techniques into two main phases: pre-merging and during-merging.

Pre-Merging Methods are preparatory steps designed to enhance the efficiency and effectiveness of the merging operation:

1. Linearization Fine-tuning: Fine-tuning model parameters along the tangent space to achieve weight space disentanglement, thus reducing interference during merging.
2. Architecture Transformation: Transforming heterogeneous models into homogeneous architectures to allow for parameter-level merging.
3. Weight Alignment: Permuting model weights to align them in the same basin, leveraging the concept of linear mode connectivity (LMC) to ensure successful integration.

During-Merging Methods focus on the act of merging the model parameters to achieve a cohesive and high-performing resultant model:

1. Basic Merging Methods: Direct weighted averaging or task arithmetic-based merging, though these often result in suboptimal performance.
2. Weighted-based Merging Methods: Utilizing advanced strategies to compute optimal merging coefficients, such as evolutionary algorithms, Bayesian optimization, and gradient descent, to improve the merging accuracy.
3. Subspace-based Merging Methods: Projecting models into sparse subspaces before merging to mitigate task interference.
4. Routing-based Merging Methods: Dynamically merging models based on input during inference, thus achieving adaptive performance.
5. Post-calibration based Methods: Aligning representations of merged models with those of individual models to correct potential biases and performance degradation.

Theoretical Insights

The paper explores the theoretical aspects of model merging, primarily focusing on:

1. Linear Mode Connectivity (LMC): Analysis of why models trained from the same initialization, albeit with different hyperparameters, can be successfully merged.
2. Flat Minima Generalization: Discussion on how merging weights can lead to flatter loss landscapes, thereby achieving better generalization.
3. Weight Disentanglement: Proposing weight disentanglement as a precondition for effective model merging, with theoretical support from neural tangent kernel (NTK) analysis.

Applications of Model Merging

Applications in Foundation Models

The paper outlines several applications of model merging across foundation models, including LLMs, multimodal LLMs (MLLMs), and image generative models.

• LLMs:
  • Human Preference Alignment: Combining models to align with diverse human preferences using techniques like task vectors and reinforcement learning from human feedback.
  • Detoxification: Reducing the toxicity in LLM outputs through model merging methods that align or negate specific task vectors.
  • Knowledge Unlearning: Removing specific knowledge (e.g., copyrighted material) from LLMs without retraining from scratch.
  • Faster Training: Accelerating LLM training by merging intermediate checkpoints or combining existing models.
  • Combining Expert LLMs: Merging domain-specific expert models to enhance general or task-specific capabilities.
• Multimodal LLMs (MLLMs):
  • Multimodal Fusion: Creating unified models capable of handling multi-modal data by merging specialized models.
  • Cross-modal Knowledge Transfer: Leveraging knowledge from high-resource modalities to improve performance on low-resource modalities.
• Image Generative Models:
  • Style Mixing: Merging models pre-trained on different styles to generate images with mixed styles.
  • Training Cost Reduction: Leveraging merging techniques to reduce the cost of training generative models.
  • Enhancing Faithfulness: Improving the accuracy of generated images with respect to their textual descriptions.

Applications in Other Machine Learning Subfields

Model merging extends beyond foundation models to various machine learning subfields:

• Continual Learning: Addressing catastrophic forgetting by merging models trained on new tasks with previously learned models.
• Multi-Task / Multi-Domain / Multi-Objective Learning: Enabling a single model to perform multiple tasks, handle multiple data domains, or optimize multiple objectives through strategic model merging.
• Out-of-Distribution / Domain Generalization: Enhancing generalization to unseen domains or distributions by leveraging merged models trained under diverse conditions.
• Federated Learning: Aggregating local models from different clients in a decentralized manner to build a robust global model.
• Zero-shot / Few-shot Learning: Integrating related models to boost the model’s generalization ability on new tasks with limited data.
• Adversarial Learning: Employing model merging as both an attack vector and a defensive mechanism, as well as for intellectual property protection.

Future Directions

The paper identifies several challenges and future directions for model merging research:

1. Performance Gap: Bridging the performance gap between merged models and independently trained models.
2. Theoretical Frameworks: Developing more comprehensive theoretical analyses to support model merging techniques.
3. Trustworthiness: Ensuring reliable and secure model merging, addressing issues such as intellectual property protection and backdoor defenses.
4. Efficiency and Scalability: Enhancing the efficiency and scalability of model merging methods to accommodate larger and more complex models.
5. Heterogeneous Models: Extending model merging techniques to effectively combine heterogeneous model architectures.
6. Cross-disciplinary Applications: Exploring interdisciplinary applications of model merging to unlock new potentials and address diverse challenges across fields.

In summary, model merging presents a versatile and efficient approach for enhancing model capabilities, offering substantial benefits across a broad spectrum of AI applications. As the research in this area continues to evolve, overcoming the identified challenges will be pivotal in realizing the full potential of model merging technologies.