Improving Instruct Models for Free: A Study on Partial Adaptation (2504.11626v1)

Abstract: Instruct models, obtained from various instruction tuning or post-training steps, are commonly deemed superior and more usable than their base counterpart. While the model gains instruction following ability, instruction tuning may lead to forgetting the knowledge from pre-training or it may encourage the model being overly conversational or verbose. This, in turn, can lead to degradation of in-context few-shot learning performance. In this work, we study the performance trajectory between base and instruct models by scaling down the strength of instruction-tuning via the partial adaption method. We show that, across several model families and model sizes, reducing the strength of instruction-tuning results in material improvement on a few-shot in-context learning benchmark covering a variety of classic natural language tasks. This comes at the cost of losing some degree of instruction following ability as measured by AlpacaEval. Our study shines light on the potential trade-off between in-context learning and instruction following abilities that is worth considering in practice.

An Analysis of Performance Trade-offs in Instruct Model Adaptation

The paper "Improving Instruct Models for Free: A Study on Partial Adaptation" explores the intricate balance between in-context learning (ICL) capabilities and instruction-following abilities in LLMs. This investigation is conducted through the lens of "partial adaptation" (PAd), a method that merges base and instruct models without additional training costs. The focus of this research is to understand how scaling the strength of instruction tuning affects the performance of LLMs, particularly regarding their few-shot learning capabilities.

The authors approach this problem by employing a suite of 18 open-weight LLMs, analyzing their performance across a benchmark set of 21 classic natural language tasks. These tasks are designed to evaluate models in scenarios like sentiment analysis, entity recognition, reading comprehension, and commonsense reasoning, among others. The methodology involves varying the extent to which base model weights are adapted towards instruct models, essentially creating intermediary models ($M_\lambda$) characterized by a scaling factor ($\lambda$) applied to the weights adjustment. This partial adaptation is observed to improve the models' performance on few-shot ICL tasks compared to either pure base or instruct models.

Key Findings and Numerical Results

The paper finds a consistent trend across all evaluated models: scaling down the instruction tuning strength enhances ICL performance. For all 18 models, the optimal few-shot ICL performance was reached at $0 < \lambda < 1$. The improvement over purely instruct-tuned models often exceeded 0.5 percentage points and reached up to 2.5 points on models like Llama-3 8B. The research highlights the largest improvements for $\lambda$ values typically between 0.5 to 0.6.

Despite these gains, the paper acknowledges a trade-off. Enhanced ICL capabilities come with a reduction in the model's instruction-following performance, as measured by AlpacaEval 2.0 benchmarks. For instance, the best ICL model, not necessarily the instruct model, was found by optimizing $\lambda$ for ICL evaluations. However, this came at an observed decrease in instruct capabilities, typically measured as a reduced win-rate in AlpacaEval evaluations.

Implications and Future Directions

The findings of this paper have practical implications. They suggest a nuanced approach when deploying LLMs for specific tasks; models can be tailored to either enhance ICL performance or instruction adherence based on task requirements. The partial adaptation method proposed offers a training-free mechanism to refine model performance and could be particularly beneficial for tasks requiring conciseness and precision.

Theoretically, this work opens up avenues for further exploration of the dynamics between pre-training knowledge retention and post-training fine-tuning effects. Future research might explore more granular aspects of the SFT and RLHF stages to dissect their contributions to the observed trade-offs. Moreover, extending this paper to multilingual models could yield insights into the universality of these observations across linguistic domains.

Conclusively, the paper offers a detailed empirical evaluation of partial adaptation in LLMs, reinforcing the notion that model adaptation needs to be strategically managed to balance competing capabilities effectively. As these models continue to evolve, methodologies like partial adaptation will prove crucial in optimizing their utility in diverse real-world applications.