DawnPiper: Scalable Pipeline Training

• DawnPiper is a memory-scalable pipeline parallel training framework that leverages node-level partitioning and theoretical guidelines to balance compute and memory across GPUs.
• It employs deep learning compilation-based profiling to optimize fine-grained operations, enabling efficient micro-batch scheduling and reducing memory imbalances.
• Cost-model-driven memory optimization in DawnPiper enables up to 11× larger batch sizes and significant throughput improvements compared to prior pipeline systems.

DawnPiper is a memory-scalable pipeline parallel training framework designed to address the limitations of existing deep learning pipeline systems, particularly the GPU memory imbalance that restricts model size and overall training efficiency. By combining fine-grained model partitioning enabled by deep learning compilation-based profiling, a rigorously derived pipeline partitioning theorem, and a cost-model-driven approach for memory optimization, DawnPiper significantly increases maximum trainable batch size and improves training throughput relative to established baselines such as vPipe and PipeDream (Peng et al., 9 May 2025).

1. Motivation and Core Problem

Pipeline parallelism, in the tradition of frameworks such as GPipe and PipeDream, partitions a neural network into $\ell$ stages through which micro-batches are streamed, with each stage assigned to a distinct GPU. The central objective is to minimize the maximal per-stage computation time,

$$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$

while overlapping computation with inter-stage communication. However, layers within deep models often exhibit heterogeneous ratios of compute-time to activation and parameter memory. Pipeline schemes, especially asynchronous variants like PipeDream’s 1F1B, impose further memory overheads via uneven parameter replication—stage $x$ must store $(\ell-x)$ extra copies. Consequently, partitions balanced by compute can cause extreme memory imbalances, with over 40% of GPU memory left idle on certain stages, severely capping trainable batch sizes and diminishing scalability (Peng et al., 9 May 2025).

2. Deep Learning Compilation-Based Profiling

DawnPiper imports PyTorch models using torch.fx to generate a detailed tensor-level directed acyclic graph (DAG). Each node corresponds to a fine-grained operation, unleashing a substantially expanded space for partition and memory optimization at the node, rather than layer, granularity, and facilitates automatic code generation per partition.

The system profiles each node $n$ for:

• $t^\text{f}_n$, $t^\text{b}_n$: forward and backward execution times,
• $m^\text{a}_n$, $m^\text{p}_n$: activation and parameter memory requirements,
• Saved-tensor timing: Determined via torch.autograd.graph.saved_tensors_hooks to precisely track activation lifespan.

This granular profiling informs both partition search and stage-wise memory optimization, enabling the placement of partition points at a much finer scale than previous frameworks.

3. Performance-Optimal Partitioning Theorem

DawnPiper formalizes the pipeline partitioning task as

$$\min \max_{1 \leq x \leq \ell} T_x \quad \text{subject to} \quad \text{peak memory per stage} \leq M_\text{GPU}.$$

A full search over partitionings is exponential in number of nodes. DawnPiper leverages empirical observation that over 90% of nodes consume $$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$0 150 MB; modest adjustment of partition points entails limited overhead.

Theorem 1 (Partition-Range Theorem):

Given two-stage splitting, let $$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$1 denote the compute-balanced cut (equalizing total $$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$2) and $$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$3 the memory-balanced cut (equalizing stage peak memory under the execution schedule). If

1. Cumulative compute and memory grow monotonically with node order,
2. Inter-stage communication time is negligible,
3. Memory optimization opportunities are roughly uniform, then the globally optimal cut must fall within $$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$4. Shifting the cut right of $$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$5 worsens both compute and memory balance, while moving it left of $$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$6 creates a new bottleneck stage.

4. Binary Pipeline Partitioning Algorithm

DawnPiper extends Theorem 1 recursively to $$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$7 using a binary (divide-and-conquer) search over permissible cut ranges at each hierarchy level. The process is implemented as follows:

$t^\text{f}_n$6

Here, CompMemBalancedCut rapidly computes $$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$8 and $$T_x = \sum_{n \in \mathcal{N}_x}(t^\text{f}_n + t^\text{b}_n),$$9 using prefix sums. The search interval $x$0 reduces search complexity from exponential to approximately $x$1, with empirical $x$2 for models such as BERT (which has over 1,000 nodes).

5. Cost-Model-Based Memory Optimization

When a stage’s memory exceeds GPU limits after partitioning, DawnPiper integrates a tensor-level cost model based on Capuchin [Peng et al., ASPLOS '20] to select between swapping and recomputation for each tensor:

• For tensor $x$3:
  • $x$4: freed memory,
  • $x$5,
  • $x$6: incremental recomputation time,
  • $x$7: overlappable swap duration,
  • $x$8,
  • $x$9,
  • $(\ell-x)$0 for both swap and recompute.

The greedy optimization algorithm sorts by minimal $(\ell-x)$1 and applies the cheaper action for top candidates until the stage’s peak memory fits within $(\ell-x)$2. This process yields near-optimal memory plans minimizing total execution time.

6. Empirical Evaluation and Benchmark Analysis

Experiments were conducted on servers with 8× NVIDIA A100 40 GB GPUs and PCIe 4.0 ×16 interconnects, across models including BERT (340M), GPT-2 (770M), T5 (780M), and AmoebaNet (28M), with $(\ell-x)$3 and $(\ell-x)$4 pipeline stages. Baselines were ZeRO-2/3, torchgpipe (GPipe), PipeDream, and vPipe.

Maximum Trainable Batch Size

• Synchronous regime ($(\ell-x)$5): DawnPiper-S enables up to $(\ell-x)$6 larger batch size compared to vPipe-S, and up to $(\ell-x)$7 larger than PipeDream (asynchronous).
• Asynchronous regime (1F1B): DawnPiper-AS supports $(\ell-x)$8 (ℓ=4) and $(\ell-x)$9 (ℓ=8) larger batch than vPipe-AS, and consistently $n$0–$n$1 over PipeDream.

Training Throughput

• Under identical batch sizes, DawnPiper delivers up to $n$2 speedup over vPipe synchronously, and $n$3 asynchronously as memory optimization becomes active.
• On 4 GPUs: mean speedup over vPipe-AS—BERT: $n$4, GPT-2: $n$5, T5: $n$6, AmoebaNet: $n$7.
• On 8 GPUs (ℓ=8): asynchronous DawnPiper is $n$8 faster on GPT-2, $n$9 on T5.

Per-Stage Balance Illustration

For T5 with $t^\text{f}_n$0:

• vPipe: max–min memory gap $t^\text{f}_n$1; DawnPiper: $t^\text{f}_n$2,
• GPU memory utilization: $t^\text{f}_n$3 higher,
• Compute-time skew (longest vs. shortest stage): $t^\text{f}_n$4 (vPipe) reduced to $t^\text{f}_n$5 (DawnPiper).

A plausible implication is that such fine-grained partitioning and optimization may enable training of models disproportionately large for a fixed hardware budget.

7. Relationship to Prior Work and Significance

The DawnPiper framework extends beyond previous partitioning approaches such as GPipe [Huang et al., NeurIPS ’19], PipeDream [Narayanan et al., SOSP ’19], and vPipe [Zhao et al., TPDS ’22], which generally partition at the coarse, layer-wise level and do not jointly optimize for memory heterogeneity and peak GPU occupancy. Tensor-level memory cost modeling is adapted from Capuchin [Peng et al., ASPLOS ’20], but integrated here directly into the partitioning process.

DawnPiper’s advances lie in its combination of DL-compiled per-node profiling, theorem-constrained partitioning, and aggressive yet cost-aware tensor memory management, collectively yielding up to 4–11× larger supported batch sizes and significant speedup. This positions DawnPiper as a scalable, theoretically grounded alternative in large-model pipeline training (Peng et al., 9 May 2025).

Framework	Partitioning Granularity	Memory Balancing	Max Batch Size (relative to PipeDream)
GPipe	Layer	No	1×
PipeDream	Layer	No	1×
vPipe	Layer	Partial	1.1–2×
DawnPiper	Node (fine-grained)	Yes	4×–11×

These results highlight the substantial impact of node-level partitioning and cost-based memory optimization for pipeline-parallel scaling of modern deep models.