Online Dynamic Batching with Formal Guarantees for LLM Training

Abstract: Modern LLM training breaks a core assumption behind offline batch samplers: the true training cost of a sample is only observable after preprocessing, augmentation, templating, tokenization, and multimodal visual-token expansion. Unless one pays for a preprocessing- and augmentation-dependent length cache, batch construction is therefore blind to the quantity that determines padding, memory use, and GPU saturation. We introduce Online Dynamic Batching (ODB), a DataLoader-side drop-in system that moves batch formation to this point of accurate observability while preserving DDP step alignment. We formalize this synchronization requirement as the Distributed Group Alignment Problem and prove deadlock-free bounded termination with default join-mode identity coverage and opt-in non-join sample-quota closure. ODB requires no model, optimizer, or attention-kernel changes and is released as online-dynamic-batching with lightweight trainer adapters. Across public 2B/8B Qwen3-VL runs on UltraChat/LLaVA/ShareGPT4o, ODB improves literal emitted-sample throughput vs. fixed-batch Standard by 1.58-2.51x on single-node Full FT/LoRA and 1.71-3.78x on two-node Full FT, with Standard-comparable quality; production MM-Mix reaches 4.43x. Against GMT/BMT offline token-budget oracles, ODB is within 15% on UltraChat/LLaVA and faster on high-CV ShareGPT4o: 2.24-2.39x single-node Full FT/LoRA and 3.06-3.69x two-node Full FT. Together, ODB occupies the online/drop-in regime for high-heterogeneity LLM fine-tuning: large throughput gains at Standard-comparable quality, formal DGAP guarantees, and no length-cache precompute or kernel rewrites.

• The paper presents an online dynamic batching method (ODB) that forms variable-size batches at runtime with formal guarantees for DDP alignment.
• ODB boosts throughput by 1.58× to 4.43× in both full fine-tuning and multimodal scenarios while preserving standard validation metrics.
• The protocol ensures deadlock-free bounded termination and unbiased per-token loss scaling without modifying the model, optimizer, or kernels.

Online Dynamic Batching with Formal Guarantees for LLM Training

Problem Formulation and Motivation

Standard LLM and multimodal fine-tuning workflows exhibit a fundamental observability gap during batch formation: sample cost (e.g., sequence length after augmentation, tokenization, multimodal expansion) is only realized post-pipeline, making offline batch samplers dependent on costly preprocessing or static length caches with limited adaptivity. This limitation results in inefficient fixed-batch paradigms—either excessive padding or underutilized compute—and precludes optimal batching on highly heterogeneous (high-CV) data. The contract required by Distributed Data Parallel (DDP)—step count alignment across ranks—becomes nontrivial with variable batch sizes. This paper introduces Online Dynamic Batching (ODB), a DataLoader-level approach that forms variable-size batches at the point of accurate length observability, avoiding model, optimizer, or kernel modifications, and provides formal guarantees for DDP-aligned operation via the Distributed Group Alignment Problem (DGAP).

ODB System Design and Protocol

ODB wraps the PyTorch DataLoader boundary, extracting post-pipeline sample lengths for online grouping and batch construction. Its collate process greedily forms length-grouped batches, targeting a user-specified per-step token budget ($L_{\max}$) via $B(l) = \max(\lfloor L_{\max}/l \rfloor, 1)$. A Max-Based Bidirectional Group Alignment protocol synchronizes group counts across ranks, solving DGAP using a Gloo-based metadata exchange, ensuring bounded termination, strict identity coverage (join mode), or sample-quota closure (non-join mode), and deadlock-freedom under DDP constraints. Token-level loss scaling is applied to normalize gradients, mitigating bias from variable batch shapes; exact loss scaling ensures per-token loss equivalence across ranks.

No changes are required to model, optimizer, attention kernels, or datasets. The collate subprocess is isolated from NCCL, incurs negligible communication overhead, and enables overlapping data preparation with GPU computation. ODB is released as an open-source Python package with minimal trainer adapters.

Empirical Evaluation and Results

ODB delivers substantial throughput gains under both Full FT and LoRA regimes and in production-class multimodal deployment scenarios:

• Public 2B/8B Qwen3-VL Fine-Tuning: ODB improves emitted-sample throughput by 1.58–2.51× on single-node Full FT/LoRA and 1.71–3.78× on two-node Full FT, holding validation and benchmark metrics within the Standard-comparable band (see Table 1 for downstream tasks).
• MM-Mix Production Case Study: ODB achieves 4.43× throughput in a multimodal deployment scenario (CV ≈ 0.8, $f_s$ ≈ 0.37), aggregating short-form OCR/VQA samples for compute-dense updates, far exceeding CV-only predictions.
• Comparison with Oracle Baselines: Against offline token-budget oracles (GMT/BMT/HFG), ODB is within 15% of throughput on UltraChat/LLaVA, and is faster (2.24–2.39× single-node, 3.06–3.69× two-node Full FT) on datasets with higher length heterogeneity.
• Padding and Compute Saturation: ODB minimizes padding (<2.1% on high-CV datasets) and achieves high compute saturation by maximizing useful tokens per step. The mechanism is distinguished from fixed-batch approaches, which cannot simultaneously optimize spatial efficiency, compute saturation, and input overlap on variable-length data.

Quality is consistently maintained. Downstream validation loss and benchmark scores (MMLU/MMMU-MC/IFEval) remain within the Standard-comparable regime, demonstrating that throughput gains are not achieved at the expense of model performance.

Numerical Highlights

• ShareGPT4o: Fixed batch size bs=1 is throughput limited (Standard: 2.37 sam/s), ODB reaches 5.83 sam/s (2.46×).
• UltraChat: Standard at 5.77 sam/s (8B), ODB at 10.23 sam/s (1.77×).
• LLaVA: Standard at 14.38 sam/s (8B), ODB at 24.87 (1.73×).

Formal Guarantees and Algorithmic Contributions

The paper provides constructive solutions to the DGAP, proven by per-rank state machines and Lyapunov potential analysis:

• Deadlock-Free Bounded Termination: Every logical iteration terminates in $O(N/W) + O(D)$ rounds (Theorem 3), with no rank blocking on any allgather, regardless of heterogeneity or buffer depth.
• Strict Identity Coverage (Join Mode): Default join-mode guarantees every sampler view is emitted with zero discard, ensuring full dataset coverage, relevant for curriculum or RLHF workflows.
• Sample-Quota Closure (Non-Join Mode): Opt-in non-join termination ensures cumulative counts reach the full quota without sample leak, empirical checks confirm $\eta_{quota}=0$ across main runs.

Loss scaling achieves unbiased per-token loss estimation, maintaining performance consistency in the presence of variable batch sizes.

Practical and Theoretical Implications

ODB occupies the runtime, drop-in batching regime for heterogeneous LLM fine-tuning, where offline static grouping, sequence packing, and length caches are impractical due to frequent augmentation-policy churn or multimodal expansion. The method enables efficient utilization of GPUs without requiring kernel rewrites or cache reconstruction. Practical deployment guidance: for high-CV or short-heavy datasets, ODB achieves maximal ROI with minimal integration overhead. The empirical boundaries of ODB’s gains are strictly data-dependent, with less room for acceleration as length homogeneity increases.

ODB’s guarantees and empirical results substantially close the gap between fixed-batch training and offline or model-side batching methods. The methodology is extensible to larger world sizes, multi-node settings, and may compose with gradient accumulation, sequence/context parallelism, and varlen-attention frameworks.

Limitations and Future Directions

Gains are contingent upon data heterogeneity; uniform workloads (low CV) yield less benefit. The activation-memory proxy ($L_{\max}$) must be swept per hardware/config, and further validation is required on larger world sizes or more complex distributed setups. Sequence packing and offline global batching retain advantage in text-only, varlen-attention-enabled stacks. Batching within ODB remains greedy local, not global-optimal. Future developments include extension to ZeRO-3, FSDP, and hybrid integration with sequence packing or context parallelism. The formal protocol may support adaptive curricula, bandit-style data selection, and stateful RLHF pipelines.

Conclusion

ODB delivers a robust, formally grounded solution for online variable-length batching in LLM training, with empirical throughput improvements of up to 4.43× over fixed-batch baselines, maintaining standard-comparable training quality. The Max-Based Bidirectional Group Alignment ensures DDP step alignment, strict identity coverage, and deadlock-free bounded termination. ODB narrows the gap between fixed-batch and offline/model-side batching while eliminating dependency on scalar caches and kernel rewrites, making it a practical tool for high-heterogeneity and multimodal LLM fine-tuning deployments (2606.19989).