Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging (2503.20641v2)

Abstract: The transition from System 1 to System 2 reasoning in LLMs has marked significant advancements in handling complex tasks through deliberate, iterative thinking. However, this progress often comes at the cost of efficiency, as models tend to overthink, generating redundant reasoning steps without proportional improvements in output quality. Long-to-Short (L2S) reasoning has emerged as a promising solution to this challenge, aiming to balance reasoning depth with practical efficiency. While existing approaches, such as supervised fine-tuning (SFT), reinforcement learning (RL), and prompt engineering, have shown potential, they are either computationally expensive or unstable. Model merging, on the other hand, offers a cost-effective and robust alternative by integrating the quick-thinking capabilities of System 1 models with the methodical reasoning of System 2 models. In this work, we present a comprehensive empirical study on model merging for L2S reasoning, exploring diverse methodologies, including task-vector-based, SVD-based, and activation-informed merging. Our experiments reveal that model merging can reduce average response length by up to 55% while preserving or even improving baseline performance. We also identify a strong correlation between model scale and merging efficacy with extensive evaluations on 1.5B/7B/14B/32B models. Furthermore, we investigate the merged model's ability to self-critique and self-correct, as well as its adaptive response length based on task complexity. Our findings highlight model merging as a highly efficient and effective paradigm for L2S reasoning, offering a practical solution to the overthinking problem while maintaining the robustness of System 2 reasoning. This work can be found on Github https://github.com/hahahawu/Long-to-Short-via-Model-Merging.

Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging

The paper introduces a comprehensive paper on model merging and its applicability in optimizing Long-to-Short (L2S) reasoning in LLMs. The transition from System 1 to System 2 reasoning models has greatly enhanced LLMs' capabilities in handling complex tasks. However, this shift comes with efficiency concerns due to overthinking tendencies, which often result in verbose and redundant reasoning outputs. The paper advocates for model merging as an efficient method to integrate the agile, intuitive thinking characteristics of System 1 models with the deliberate, systematic reasoning features of System 2 models, effectively tackling this challenge.

Key Findings

The paper systematically investigates various model merging methodologies applied to different LLM scales (e.g., 1.5B, 7B, 14B, and 32B) and reveals several critical insights:

1. Efficiency in Task-Vector Based Merging: Among the explored methodologies, task-vector-based approaches such as Task Arithmetic (TA) and Ties-Merging show substantial efficacy in achieving L2S reasoning, facilitating up to a 50% reduction in reasoning length while preserving or slightly enhancing accuracy.
2. Limitations of SVD-Based Techniques: SVD-based model merging methods deliver moderate results. Their performance is contingent upon the inherent spectral characteristics of task vectors, highlighting their restricted applicability when task vectors deviate from ideal low-rank distributions.
3. Promising Future for Activation-Based Merging: Activation-based merging methods exhibit impressive potential by notably improving reasoning accuracy and processing efficiency, emphasizing their promise as future leading strategies in model merging.
4. Scale-Dependent Merging Effectiveness: The analysis underscores the challenge smaller models face in acquiring comprehensive reasoning abilities through merging, whereas larger models pose difficulties in effectively balancing reasoning performance and output length compression.

Practical and Theoretical Implications

The findings from this paper propose model merging as a practical approach to mitigate reasoning inefficiencies without compromising the robustness of System 2 reasoning. By reducing unnecessary reasoning elements, L2S optimization through model merging offers computational efficiency and performance stability, addressing concerns regarding redundant deliberations in current LLM configurations. The promising results from task-vector and activation-based methodologies signify a move towards strategic model optimization, where models can intelligently adapt output length based on task complexity.

Future Directions

The paper motivates further exploration into hyperparameter tuning for merging methods to enhance performance consistency and robustness. Additionally, it suggests the need for developing hyperparameter-insensitive approaches or frameworks capable of automatic parameter optimization during merging processes. The paper also identifies the necessity to address large performance disparities between candidate models for effective merging and to expand the exploration of calibration data selection in activation-based merging methods.

Overall, the paper establishes model merging as an impactful method in the AI landscape, offering an efficient resolution to the overthinking problem prevalent in LLMs, while presenting opportunities for ongoing research and advancement in model optimization strategies.