An Emulator for Fine-Tuning Large Language Models using Small Language Models (2310.12962v1)

Abstract: Widely used LLMs (LMs) are typically built by scaling up a two-stage training pipeline: a pre-training stage that uses a very large, diverse dataset of text and a fine-tuning (sometimes, 'alignment') stage that uses targeted examples or other specifications of desired behaviors. While it has been hypothesized that knowledge and skills come from pre-training, and fine-tuning mostly filters this knowledge and skillset, this intuition has not been extensively tested. To aid in doing so, we introduce a novel technique for decoupling the knowledge and skills gained in these two stages, enabling a direct answer to the question, "What would happen if we combined the knowledge learned by a large model during pre-training with the knowledge learned by a small model during fine-tuning (or vice versa)?" Using an RL-based framework derived from recent developments in learning from human preferences, we introduce emulated fine-tuning (EFT), a principled and practical method for sampling from a distribution that approximates (or 'emulates') the result of pre-training and fine-tuning at different scales. Our experiments with EFT show that scaling up fine-tuning tends to improve helpfulness, while scaling up pre-training tends to improve factuality. Beyond decoupling scale, we show that EFT enables test-time adjustment of competing behavioral traits like helpfulness and harmlessness without additional training. Finally, a special case of emulated fine-tuning, which we call LM up-scaling, avoids resource-intensive fine-tuning of large pre-trained models by ensembling them with small fine-tuned models, essentially emulating the result of fine-tuning the large pre-trained model. Up-scaling consistently improves helpfulness and factuality of instruction-following models in the Llama, Llama-2, and Falcon families, without additional hyperparameters or training.

• The paper decouples pre-training and fine-tuning to isolate the distinct knowledge acquired during each stage.
• It introduces Emulated Fine-Tuning (EFT) that leverages reinforcement learning to combine scaling strategies for enhanced factuality and helpfulness.
• The approach enables dynamic test-time adjustments, providing an efficient alternative to retraining for improving large model performance.

Overview of the Paper: An Emulator for Fine-Tuning LLMs Using Small LLMs

The paper, authored by Eric Mitchell and colleagues from Stanford University, introduces a novel method called Emulated Fine-Tuning (EFT), which serves as an emulator to effectively fine-tune LLMs utilizing Small LLMs. The paper explores the bifurcated structure typically seen in the LLM training pipeline, distinguishing between pre-training and fine-tuning stages, and examines the distributed knowledge and skills cultivated in each distinct phase.

Key Contributions and Claims

1. Decoupling of Pre-Training and Fine-Tuning Knowledge: The research provides a methodology to separate and paper the knowledge gained during the pre-training stage from that acquired during fine-tuning. This separation is critical for understanding which capabilities are developed in each training stage and at what scale.
2. Emulated Fine-Tuning (EFT): EFT is introduced as a principled and practical framework that leverages reinforcement learning concepts to assimilate knowledge from different model scales. The process involves combining the foundational knowledge from a large pre-trained model with the refined behavioral adjustments learned by a smaller fine-tuned model.
3. Empirical Evidence and Experimental Justification: The authors present extensive experiments showing that up-scaling (using a large base model and a small fine-tuned model) improves factuality, whereas down-scaling enhances helpfulness. This finding is consistently demonstrated across multiple LLM families, including Llama and Falcon models, and in diverse datasets such as Anthropic Helpful-Harmless (HH) and ELI5.
4. Dynamic Test-Time Adjustments: EFT enables on-the-fly adjustments to model behavior at test time without retraining. The paper specifically explores the potential of modulating between helpfulness and harmlessness objectives dynamically, providing insights into maintaining task adherence while considering various ethical dimensions.
5. Efficiency and Practicality of Up-Scaling: Up-scaling is highlighted as a resource-efficient approach for improving model performance. It maintains the advantageous characteristics of large-scale fine-tuning while avoiding the significant computational costs associated with directly fine-tuning large models.

Implications and Future Directions

The paper's findings have pivotal implications for how future LLM training might be approached, particularly in the context of increasing computational limitations and requirements for ethical AI alignment. By effectively decoupling the scales of pre-training and fine-tuning, researchers can independently modulate these stages, optimizing both factual and task-related capabilities without corresponding computational expenses.

The introduction of EFT as a viable strategy bypasses the need for resource-intensive training cycles by enabling the emulation of large-scale behavioral changes using smaller models. This methodology is likely to influence future techniques for training and deploying LLMs, particularly in resource-constrained environments.

Furthermore, the ability to dynamically adjust model outputs offers a promising frontier for adaptable AI systems, where requirements and ethical considerations change with context and user intent. This adaptability is increasingly necessary as AI systems are deployed in more complex and sensitive real-world scenarios.

Speculations on Future Developments

As LLM architectures continue to evolve, we speculate that techniques like EFT will become integral to optimizing the performance of generative models. Future enhancements may include improved mechanisms for automating the test-time adjustment process and refining reward interpolation techniques, leading to better integration of fine-tuning objectives. Additionally, applying EFT to other domains of AI beyond language processing might uncover broader applicability, further enhancing the versatility and efficiency of AI training paradigms.