DualPipe Microbatch Overlap in DNN Training

• DualPipe Microbatch Overlap is a technique that duplicates the computational graph into two interleaved pipelines to mitigate network stalls during deep neural network training.
• The method is generalized by kFkB scheduling, which groups k forward and backward passes to offer tunable memory-compute tradeoffs and finer-grained overlap.
• Experimental outcomes demonstrate that adaptive scheduling using DualPipe or kFkB can achieve 10–25% throughput gains under variable network conditions.

DualPipe Microbatch Overlap refers to a class of pipeline parallel training strategies for deep neural networks in which multiple streams of computation and communication are interleaved to mitigate the deleterious impact of network stalls, particularly in environments subject to resource preemption. The DualPipe family duplicates the computational graph into two pipelines and alternates execution, enabling one pipeline to proceed with computation while the other is blocked on communication. More generally, recent work—including Ada-Grouper's adaptive kFkB scheduling—frames DualPipe as a two-stream (k=2) case in a broader category of group-wise microbatch overlapping scheduling, where $k$ forwards and $k$ backwards per stage are grouped and overlapped to maximize throughput and hide communication latency. This approach achieves explicit and tunable memory-compute tradeoffs and allows dynamic adaptation to fluctuating resource constraints (Wang et al., 2023).

1. Motivation and Problem Setting

In large-scale training of deep learning models, pipeline parallelism divides the model into $S$ stages distributed across devices. Traditional scheduling such as 1F1B (one-forward, one-backward) interleaves the forward and backward computations of micro-batches, but with each micro-batch processed strictly sequentially. When inter-stage communications are delayed, as occurs in cloud and shared resource environments, pipeline stages exhibit increased idle time (“bubbles”), reducing utilization and overall throughput. DualPipe-style overlap arose to address these network-induced bottlenecks by allowing computation in one pipeline to proceed while the other pipeline is stalled, thereby improving resource utilization under adverse network conditions (Wang et al., 2023).

2. DualPipe and Generalized kFkB Scheduling

The archetypal DualPipe approach duplicates the entire computational graph, yielding two “sub-pipelines” that are interleaved temporally. When one sub-pipeline is blocked by data transfer, the other can proceed with computation. However, this results in a static two-stream overlap and incurs a 2× memory overhead due to duplicated state for activations and gradients.

Ada-Grouper introduces the kFkB (“k-forwards, k-backwards”; Editor’s term) scheduling scheme as a generalization. Here, a global batch is divided into $N$ micro-batches, grouped into size-$k$ units. Within each group, $k$ forward passes for micro-batches $0\ldots k-1$ are executed in sequence across the pipeline, followed by $k$ backward passes in reverse order. This enables each pipeline stage to have $k$ micro-batches in flight, allowing communication and computation to overlap more flexibly than DualPipe’s two-stream rigid splitting. The memory overhead is increased by a factor $k$, but can be tuned to balance hardware constraints and communication hiding, resulting in controllable overlap granularity and improved utilization in the presence of unpredictable network preemption (Wang et al., 2023).

3. Mathematical Modeling of Group Overlap

Key performance parameters in kFkB scheduling are:

• $k$0, $k$1: time for forward and backward computation at each pipeline stage
• $k$2: time to communicate activations or gradients between adjacent stages for a single micro-batch
• $k$3: number of pipeline stages
• $k$4: group size for each kFkB unit

For each schedule group, total computation and communication times are given by:

$k$5

$k$6

Memory pressure is

$k$7

where $k$8 is micro-batch size and $k$9 is the activation footprint at stage $S$0. As $S$1 increases, memory required for simultaneously stored activations grows linearly, so $S$2 is constrained by the device limit $S$3. The optimal $S$4 and $S$5 are searched over the Pareto-frontier of memory-performance combinations, with an online adaptation component selecting $S$6 at runtime based on measured costs and network state (Wang et al., 2023).

4. Dynamic Adaptation and Scheduling Workflow

Ada-Grouper implements an adaptive scheduler that periodically profiles communication times ($S$7) and recomputes cost estimates for each candidate $S$8 within the allowed memory budget. The objective is to minimize the estimated pipeline length $S$9, incorporating pipeline fill, steady-state compute, and pipeline drain, using a cost model derived from profiled compute and communication times under the kFkB execution order. The scheduler scans for $N$0, applying plan switches without requiring an ILP solver or reconstruction of the pipeline graph.

Metric collection and adaptation operate continuously in a background thread, ensuring responsiveness to network preemptions. When communication slows due to preemption events, the estimated cost of smaller $N$1 increases, favoring larger group sizes that inject more computation to hide stalls (Wang et al., 2023). As a result, Ada-Grouper can maintain performance benefits of 10–25% (observed on GPT-medium with 8 GPUs, global batch 192), up to 30% in heavily preempted regimes.

5. Comparative Analysis: DualPipe vs. kFkB and Related Approaches

Distinctive contrasts between DualPipe and generalized kFkB microbatch overlap are summarized below:

Method	Memory Overhead	Overlap Granularity	Adaptivity to Network
DualPipe	2×	2 streams	Rigid (requires graph rebuild)
kFkB (k>1)	k×	k streams	Adaptive (online tuning possible)

In addition, kFkB requires no computational graph duplication and achieves more fine-grained overlap. Under highly preempted network conditions, k up to 6 can be used, allowing up to six overlapping streams, providing greater slack for hiding communication latency than DualPipe’s static two-stream design. A plausible implication is that kFkB is suitable for a broader range of dynamic and heterogeneous computational environments than classical DualPipe.

DynaPipe adopts a dynamic micro-batching and pipeline scheduling strategy for multi-task model training, optimizing makespan for variable-length micro-batches and incorporating deadlock-free, memory-aware scheduling. While DynaPipe does not duplicate the pipeline, it demonstrates that optimal overlapping and scheduling—achieving up to 4.39× and 3.25× throughput gains over uniform-packing 1F1B baselines—can be realized through combinatorial optimization and adaptive cycle-based scheduling, confirming the broader utility of group-wise microbatch overlapping principles in both model-parallel and multi-task training regimes (Jiang et al., 2023).

6. Experimental Outcomes and Practical Impact

Ada-Grouper’s adaptive kFkB scheduler demonstrates pronounced advantages under network variance:

• On GPT-medium (8 GPUs), k=2…6 realized 10–25% higher throughput than 1F1B, with kFkB plans remaining within 97–105% of best throughput during network-busy periods.
• For weak scaling (U-Net Base, 8 GPUs) with highly preempted links, k=2…4 led to 2–14% throughput gain over 1F1B.
• For strong scaling on GPT-medium (64 global batch, 2–8 GPUs), kFkB delivered 15–20% better performance than 1F1B and outperformed data-parallel SPMD by 1.5–2× FLOPs/s.
• Adaptive scheduling under fluctuating network preemption consistently selected optimal k values (ranging from 4 to 6), achieving throughput >20% above 1F1B throughout four one-hour intervals of shifting network conditions.

These results establish group-based microbatch overlap as a key tool for achieving stable, high-throughput DNN training in dynamic, resource-constrained environments (Wang et al., 2023).

7. Significance and Future Directions

Group-wise microbatch overlap, encompassing DualPipe and adaptive kFkB methods, generalizes pipeline parallel training to accommodate non-deterministic communication latency and constrained memory. The explicit, tunable tradeoff between memory and compute enables practitioners to deploy large-scale training tasks robustly on commodity and cloud ecosystems lacking dedicated network resources. By avoiding static duplication and offering dynamic adaptation, these methods extend the frontier of practical and efficient distributed deep learning. A plausible implication is future research will integrate these group-overlap frameworks into diverse model architectures and increasingly heterogeneous, multi-tenant hardware—potentially with tighter integration of micro-batch optimization and cycle-based adaptive runtime scheduling observed in systems such as DynaPipe and Ada-Grouper (Wang et al., 2023, Jiang et al., 2023).