Defeating the Training-Inference Mismatch via FP16 (2510.26788v1)

Abstract: Reinforcement learning (RL) fine-tuning of LLMs often suffers from instability due to the numerical mismatch between the training and inference policies. While prior work has attempted to mitigate this issue through algorithmic corrections or engineering alignments, we show that its root cause lies in the floating point precision itself. The widely adopted BF16, despite its large dynamic range, introduces large rounding errors that breaks the consistency between training and inference. In this work, we demonstrate that simply reverting to \textbf{FP16} effectively eliminates this mismatch. The change is simple, fully supported by modern frameworks with only a few lines of code change, and requires no modification to the model architecture or learning algorithm. Our results suggest that using FP16 uniformly yields more stable optimization, faster convergence, and stronger performance across diverse tasks, algorithms and frameworks. We hope these findings motivate a broader reconsideration of precision trade-offs in RL fine-tuning.

• The paper demonstrates that switching from BF16 to FP16 nearly eliminates the training-inference mismatch by reducing rounding errors and bias in gradient updates.
• It presents empirical evidence that FP16 improves training stability, accelerates convergence, and achieves up to 24× reduction in mismatch compared to BF16.
• The approach simplifies RL optimization in LLMs by removing the need for complex algorithmic corrections, enabling the use of standard, unbiased policy gradient estimators.

Defeating the Training-Inference Mismatch via FP16: A Technical Analysis

Introduction

The paper "Defeating the Training-Inference Mismatch via FP16" (2510.26788) addresses a persistent source of instability in RL-based fine-tuning of LLMs: the numerical mismatch between training and inference policies. The authors identify the root cause as the floating-point precision standard—specifically, the widespread use of BF16—and demonstrate that switching to FP16 precision fundamentally resolves the mismatch. This essay provides a technical summary of the paper's findings, experimental evidence, and implications for RL optimization in LLMs.

Training-Inference Mismatch: Problem Formulation

Modern RL frameworks for LLM fine-tuning typically employ distinct computational engines for training (gradient computation) and inference (rollout), leading to subtle but impactful numerical discrepancies. These discrepancies manifest as a mismatch between the training policy $\trainerpi$ and the inference policy $\rolloutpi$, resulting in biased gradients and a deployment gap. The bias arises because samples are drawn from $\rolloutpi$ but gradients are computed with respect to $\trainerpi$, violating the assumptions of standard policy gradient estimators. The deployment gap further degrades performance, as the model parameters optimized for $\trainerpi$ are not optimal for $\rolloutpi$.

Algorithmic corrections, such as importance sampling (IS) at the token or sequence level, have been proposed to mitigate these issues. However, these methods introduce additional computational overhead (e.g., extra forward passes) and suffer from high variance, especially in sequence-level IS, which impedes convergence and stability.

Precision as the Root Cause

The authors systematically analyze the floating-point formats used in LLM training. BF16, the current standard, offers a wide dynamic range (8 exponent bits) but limited precision (7 mantissa bits), making it robust for pre-training but susceptible to rounding errors in fine-tuning. FP16, by contrast, provides higher precision (10 mantissa bits) at the cost of a reduced dynamic range (5 exponent bits). The paper demonstrates that the lower precision of BF16 is the primary source of the training-inference mismatch, as minor implementation differences between engines are amplified by BF16's rounding behavior.

Switching to FP16, with its higher precision, ensures that the outputs of training and inference engines are numerically consistent, effectively eliminating the mismatch. The adoption of FP16 is straightforward, requiring only minor configuration changes in mainstream frameworks, and is supported by mature loss scaling techniques to address its limited dynamic range.

Empirical Evidence

The paper presents extensive experimental results across multiple frameworks (VeRL, Oat), model families (Qwen, OctoThinker), and training regimes (dense, MoE, LoRA). The key findings are:

• FP16 consistently yields more stable RL training, faster convergence, and higher final rewards compared to BF16.
• Algorithmic corrections in BF16 (token-level TIS, sequence-level MIS) prolong training but ultimately fail to match FP16's stability and performance.
• The training-inference mismatch, measured by token-level and sequence-level probability ratios, is reduced by approximately 24$\times$ in FP16 relative to BF16.
• FP16 enables the use of classic, unbiased policy gradient estimators without the need for complex corrections.

  Figure 1: Training reward comparison between BF16 and FP16 across diverse settings, demonstrating superior stability and convergence for FP16.

  

  Figure 2: FP16 significantly reduces the training-inference mismatch at both token and sequence levels, as shown by concentrated probability distributions.

  

  Figure 3: Switching from BF16 to FP16 stabilizes and prolongs RL training, with FP16 outperforming all BF16 baselines.

  

  Figure 4: Performance comparison of various RL algorithms under FP16, showing convergence and minimal differences due to reduced mismatch.

  

  Figure 5: Ablation paper on precision combinations, highlighting the superior stability and efficiency of FP16 for both training and inference.

  

  Figure 6: Evaluation comparisons between BF16 and FP16 across frameworks, algorithms, datasets, and training regimes, confirming FP16's generalization benefits.

Generalization and Scaling

The superiority of FP16 is validated across a range of scenarios:

• MoE RL: FP16 mitigates instability arising from precision-sensitive operations (e.g., top-$k$ expert selection) and parallelization strategies in MoE models.
• LoRA RL: FP16 maintains stable training in LoRA-based RL, whereas BF16 collapses early.
• Large Dense Models: FP16 accelerates reward growth and achieves higher validation accuracy in large-scale models (e.g., Qwen3-14B-Base).
• Alternative Model Families: FP16 stabilizes training in models with different initialization and numerical characteristics (e.g., OctoThinker-3B).

Theoretical and Practical Implications

The findings prompt a reconsideration of precision trade-offs in RL fine-tuning. While BF16 remains optimal for pre-training due to its dynamic range, FP16's higher precision is critical for stable and efficient RL optimization. The results also clarify the bias-variance trade-off in RL algorithms: under BF16, low-variance but biased methods collapse, while unbiased methods suffer from high variance and slow convergence. FP16 resolves this tension by reducing both bias and variance, enabling robust optimization with standard policy gradient estimators.

From a practical perspective, the adoption of FP16 is trivial and does not require architectural or algorithmic changes. The efficiency gains are substantial, as FP16 avoids the computational overhead of algorithmic corrections and supports fast inference. The approach generalizes across frameworks, model architectures, and training regimes, making it a universally applicable solution for RL-based LLM alignment.

Future Directions

The paper suggests that further reductions in precision (e.g., FP8) may be feasible, provided engineering challenges related to dynamic range are addressed. The community is encouraged to explore precision-aware optimization strategies and to revisit foundational choices in the LLM training stack, especially for post-training and alignment phases.

Conclusion

The training-inference mismatch in RL fine-tuning of LLMs is fundamentally a problem of numerical precision. The paper demonstrates that switching from BF16 to FP16 virtually eliminates this mismatch, leading to more stable training, faster convergence, and superior performance. FP16 should be reconsidered as a foundational option for robust RL fine-tuning, and future research should further investigate precision-aware optimization strategies for scalable and reliable LLM alignment.