--- title: 'FIRE: Fine-Tuning with Integrated Replay & Efficiency' url: https://www.emergentmind.com/topics/fine-tuning-with-integrated-replay-and-efficiency-fire type: topic --- # FIRE: Fine-Tuning with Integrated Replay & Efficiency Fine-Tuning with Integrated Replay and Efficiency (FIRE) is a data-centric framework for efficient adaptation and continual improvement of machine learning models, primarily targeting large language models (LLMs), code generation systems, and machine-learned interatomic potentials (MLIPs). FIRE encompasses a suite of synergistic methodologies that combine replay, targeted data selection, regularization, and algorithmic innovations, enabling sample-efficient, stable, and scalable fine-tuning for both supervised and reinforcement learning scenarios. ## 1. Core Principles and Conceptual Framework FIRE addresses two leading challenges in modern fine-tuning: excessive data and compute requirements, and catastrophic forgetting of previously acquired capabilities during adaptation. Its defining principle is the systematic integration of two ingredients: (i) replay—reusing informative samples from past or auxiliary distributions, and (ii) algorithmic enhancements for selecting, prioritizing, or stabilizing new training examples. This dual approach is instantiated through various concrete techniques across different domains: - Adaptive difficulty selection and rollout replay in LLM RLFT [2506.05316] - Prioritized experience replay with targeted beam/program selection for code LLMs [2410.12236] - Reset-based replay with hybrid SFT/preference objectives for preference optimization [2508.06412] - Replay-augmented fine-tuning with continual sampling and buffer mechanisms for MLIPs [2601.17847, 2506.15223] - Regularized approximate replay in LoRA-based instruction tuning [2512.22337] - Integration with scalable continual learning via LoRA, consolidation, and merging in multi-task settings [2505.12512] These approaches employ replay both as a means to improve data utilization efficiency and as a tool to enforce a plasticity-stability trade-off essential for practical continual learning. ## 2. Replay Mechanisms and Data Selection FIRE incorporates multiple replay mechanisms tailored to domain requirements: - **Rollout Replay in LLM RLFT**: In on-policy policy-gradient training (GRPO), recent rollouts are buffered and reused in subsequent updates, reducing per-step sample complexity by 11–13%. Off-policy corrections are made via importance weighting, maintaining theoretical consistency (akin to off-policy PPO) [2506.05316]. - **Possibility and Pass-rate Prioritized Experience Replay (PPER)**: In code generation, beam search is used to collect multiple candidate programs per prompt, each scored by generation likelihood and empirical pass rate. The P2Value metric, a convex combination of model and empirical signals, governs replay prioritization, which is implemented through rank-based sampling [2410.12236]. - **Replay Buffers in MLIP Fine-tuning**: Small buffers comprising representative pretraining samples are interleaved with target-specific data during stochastic optimization. Partial or continual fine-tuning updates are performed using a composite loss that balances task adaptation against general knowledge retention [2601.17847]. - **Regularized Approximate Replay**: In supervised LLM fine-tuning, batches of samples drawn from a pretraining-like corpus (e.g., openwebtext) are introduced during every update, combined with a KL-regularization penalty to the base model outputs, preventing catastrophic forgetting [2512.22337]. Data selection often targets examples expected to maximize learning signal (e.g., moderate-difficulty problems in RLFT) or sample efficiency (e.g., clustering-based selection in MLIPs or pass-rate prioritization in code) [2506.05316, 2601.17847, 2410.12236]. ## 3. Algorithmic Structure and Loss Formulations The FIRE framework generalizes to a range of optimization and learning setups by specifying structured procedures for integrating replay: - **Reinforcement Learning Fine-tuning (LLMs)**: - Compute adaptive difficulty for each prompt as the mean failure rate under the current policy. - Use attention-based methods to cheaply predict adaptive difficulty for all data points. - Sample new rollouts near the target difficulty (typically 0.5), replay a fraction of buffered rollouts, and assemble mixed training batches. - Update the model using importance-weighted likelihood ratios and a clipped surrogate RL loss with a KL penalty [2506.05316]. - **Supervised and Preference-based Optimization**: - Hybridize supervised losses (SFT) with preference/contrastive objectives (e.g., DPO), maintaining replay of both new and original preference data [2508.06412]. - Periodically reset model weights toward the initial state to restore plasticity ("shrink-and-perturb"), combating primacy bias from overexploited replay [2508.06412]. - Losses in MLIPs combine squared energy/force error on target samples and replayed pretraining samples, with tunable weights for loss components [2601.17847]. - **Continual Learning and LoRA-based Methods**: - Use low-rank adaptation (LoRA) for per-task fine-tuning, with replayed data introduced in each batch to mitigate forgetting. - After each task, enter consolidation phases where replayed samples dominate, or employ sequential merging to stabilize accumulated knowledge [2505.12512]. - For LoRA-based LLM replay, the fine-tuning loss is: $$ L(\phi) = \mathbb{E}_{(x, y) \sim D_{\text{new}}}[ - \log \pi_{\theta+\phi}(y|x) ] + \beta\, \mathbb{E}_{x \sim D_{\text{new}}}[ D_{KL}( \pi_{\theta+\phi}(\cdot|x) \| \pi_{\theta_0}(\cdot|x) ) ] + \rho\, \mathbb{E}_{x \sim D_{\text{replay}}}[ -\log \pi_{\theta+\phi}(x_t | x_{