AMDP: Async Multi-Directional Pipeline Parallelism

• AMDP is an asynchronous training method that reduces gradient mismatch to at most one update by enforcing a two-minibatch limit at stage 0.
• It leverages multiple concurrent pipelines and gradient accumulation to maximize hardware utilization while mitigating pipeline bubbles.
• Empirical results demonstrate up to 17% throughput gains on GPT and BERT models with convergence rates close to synchronous baselines.

Asynchronous Multi-Directional Pipeline Parallelism (AMDP) is an asynchronous training scheme for large-scale Transformer models that combines bounded-staleness pipeline execution, multiple concurrent pipelines with alternating directions, and gradient accumulation to preserve high hardware utilization without inheriting the depth-dependent parameter mismatch of earlier asynchronous pipeline methods. Its defining structural rule is that stage $0$ of each pipeline reads at most two minibatches before the first backward pass arrives, which bounds the forward–backward mismatch at every stage by at most one update, independent of pipeline depth or placement. To recover utilization lost by such a short warmup, AMDP launches multiple concurrent pipelines and adapts their number to depth; it further accumulates gradients and applies them in a single update so that only a bounded number of minibatches experience mismatch, limited to within one optimization step (Chen et al., 28 May 2026).

1. Definition and motivating problem

Pipeline parallelism partitions a model into $d$ sequential stages and streams microbatches through them. In synchronous schemes such as GPipe, DAPPLE, Inter‑1F1B, Chimera, and ZB‑V, gradients are accumulated and weights are updated once per iteration, so forward and backward for each microbatch see the same parameters, but warmup and drain phases introduce pipeline bubbles. In asynchronous schemes such as PipeDream, PipeDream‑2BW, XPipe, and vNAG, stages update without global barriers, which largely removes bubbles but introduces parameter mismatch between forward and backward passes (Chen et al., 28 May 2026).

AMDP targets the structural source of that mismatch. If stage $0$ reads $n$ minibatches before receiving any backward signal, then the mismatch at stage $i$ is

$\operatorname{mismatch}(i) = \min(n, d-i)-1.$

Earlier asynchronous designs effectively take $n=d$, yielding

$\operatorname{mismatch}(i)=d-i-1,$

so the earliest stages accumulate the largest staleness as depth grows. AMDP fixes $n=2$, which gives

$\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$

The mismatch bound is therefore constant rather than depth-dependent (Chen et al., 28 May 2026).

The term “multi-directional” refers to the use of multiple concurrent pipelines with different logical directions over the same physical GPUs. AMDP inherits the bidirectional intuition of Chimera but departs from it in two ways: it is fully asynchronous, and it uses $d$0 concurrent pipelines together with a one-step mismatch bound rather than synchronous iteration boundaries (Chen et al., 28 May 2026).

2. Formal model and bounded-delay theory

AMDP is analyzed as delayed stochastic optimization. The paper formulates training as

$d$1

with gradients computed at past parameters,

$d$2

where $d$3 under AMDP’s structural constraint. Under the assumptions that $d$4 is $d$5-smooth and bounded below, gradients are unbiased with bounded variance, and the stepsize satisfies $d$6, the iterates obey

$d$7

The additional error from staleness is therefore second order in $d$8, and the method is characterized as “near-synchronous” (Chen et al., 28 May 2026).

A second formal property is topology independence. Because AMDP enforces FIFO on each GPU and does not reorder the causal order of forward and backward events, the mismatch bound depends only on the injection limit $d$9, the logical stage index $0$0, and the depth $0$1, but is independent of the mapping between stages and physical GPUs, multi-node placement, or multi-directional permutations (Chen et al., 28 May 2026).

This bounded-delay view aligns AMDP with broader asynchronous pipeline analyses that model each stage or node by a delayed gradient evaluated on $0$2 and applied to $0$3. A plausible implication is that AMDP can be interpreted as a special bounded-delay instantiation in which the delay is structurally capped at one update rather than merely controlled statistically (Ajanthan et al., 2 May 2025).

3. Multi-directional scheduling and bubble reduction

AMDP keeps the per-pipeline schedule short and then composes many such pipelines. Within each pipeline, it uses 1F1B execution and enforces that stage $0$4 injects only two minibatches before the first backward pass arrives. A single such pipeline has active ratio approximately

$0$5

To fill the device, AMDP therefore runs about $0$6 pipelines concurrently. The paper states this explicitly: since stage $0$7 reads two minibatches, setting the number of pipelines to $0$8 fully eliminates bubbles (Chen et al., 28 May 2026).

The mapping from logical stage $0$9 in pipeline $n$0 to physical GPU is alternating. For even $n$1,

$n$2

and for odd $n$3,

$n$4

For $n$5, one example is: $n$6 for pipeline $n$7, and

$n$8

for pipeline $n$9 (Chen et al., 28 May 2026).

When multiple pipelines contend for the same GPU, AMDP resolves conflicts by FIFO: operations are queued, the earliest executes first, and later ones wait. The schedule is organized in segments of $i$0 minibatches and distinguishes leading, trailing, and middle bubbles, the last arising from forward/backward cost asymmetry. To reduce leading and trailing bubbles, AMDP uses controlled minibatch preloading at segment boundaries; the number of preloaded minibatches is

$i$1

This preloading is coupled to gradient accumulation so that mismatch remains bounded in optimization time (Chen et al., 28 May 2026).

A useful contrast is with synergistic tensor–pipeline schedules that are multi-directional only within an iteration. In braided TP+PP schedules, a GPU may simultaneously host forward compute, backward compute, TP AllReduce, and PP communication, but the overall method remains globally synchronous. AMDP instead relaxes the global iteration ordering itself while retaining a one-step mismatch bound (Qi et al., 31 Oct 2025).

4. Gradient accumulation, update semantics, and ZeRO integration

Naively combining multiple asynchronous pipelines with immediate synchronization after every backward pass would increase both communication and mismatch. AMDP instead accumulates gradients over an update window and applies a single optimizer step when an accumulation threshold $i$2 is reached. If

$i$3

then the individual minibatch gradients that contribute to $i$4 are computed at parameters $i$5 with $i$6. If the accumulation threshold equals pipeline depth $i$7, each minibatch in the window experiences a one-step mismatch; if $i$8, only the first $i$9 minibatches in the window are mismatched and the remaining $\operatorname{mismatch}(i) = \min(n, d-i)-1.$0 are effectively synchronous (Chen et al., 28 May 2026).

The update rule is presented in a general adaptive form,

$\operatorname{mismatch}(i) = \min(n, d-i)-1.$1

where $\operatorname{mismatch}(i) = \min(n, d-i)-1.$2 is a momentum-smoothed gradient and $\operatorname{mismatch}(i) = \min(n, d-i)-1.$3 is a preconditioner, so the theory covers SGD- and Adam-like optimizers (Chen et al., 28 May 2026).

AMDP couples this with ZeRO-style ownership to avoid replicating optimizer state across all concurrent pipelines. For stage $\operatorname{mismatch}(i) = \min(n, d-i)-1.$4, GPU $\operatorname{mismatch}(i) = \min(n, d-i)-1.$5 is the owner of the parameters and optimizer state for that stage; other GPUs hosting replicas send gradients to the owner, the owner performs reduction and the optimizer step, and updated parameters are broadcast back. The paper states that this replaces all-reduce with reduce plus broadcast of the same complexity, reduces optimizer-state memory per GPU to approximately $\operatorname{mismatch}(i) = \min(n, d-i)-1.$6 of the naive multi-pipeline requirement, and improves throughput by about $\operatorname{mismatch}(i) = \min(n, d-i)-1.$7 when ZeRO is enabled (Chen et al., 28 May 2026).

5. Empirical performance and convergence behavior

The experiments use GPT-style and BERT-style Transformers of about $\operatorname{mismatch}(i) = \min(n, d-i)-1.$8B and $\operatorname{mismatch}(i) = \min(n, d-i)-1.$9B parameters respectively, on OpenWebText and Wikipedia, with 8× NVIDIA A800 80GB GPUs per node and 200Gb/s InfiniBand for 16-GPU runs. All methods use AdamW, mixed precision, microbatch size $n=d$0, and consistent hyperparameters (Chen et al., 28 May 2026).

On 8 GPUs and 16 GPUs, AMDP is reported as the fastest method across all tested configurations $n=d$1 for both GPT and BERT. Against the strongest asynchronous baseline, PipeDream‑2BW, the throughput gains include:

• GPT, $n=d$2: $n=d$3 vs $n=d$4 ktokens/s, a $n=d$5 gain.
• BERT, $n=d$6: $n=d$7 vs $n=d$8, a $n=d$9 gain.
• 16 GPUs, $\operatorname{mismatch}(i)=d-i-1,$0: $\operatorname{mismatch}(i)=d-i-1,$1 vs $\operatorname{mismatch}(i)=d-i-1,$2.
• 16 GPUs, $\operatorname{mismatch}(i)=d-i-1,$3: $\operatorname{mismatch}(i)=d-i-1,$4 vs $\operatorname{mismatch}(i)=d-i-1,$5 (Chen et al., 28 May 2026).

Convergence is close to synchronous baselines. At $\operatorname{mismatch}(i)=d-i-1,$6, $\operatorname{mismatch}(i)=d-i-1,$7, learning rate $\operatorname{mismatch}(i)=d-i-1,$8, and $\operatorname{mismatch}(i)=d-i-1,$9k iterations:

• GPT final train loss is $n=2$0 for AMDP versus $n=2$1 for synchronous baselines.
• BERT final train loss is $n=2$2, matching DAPPLE.
• For GPT validation perplexity, DAPPLE reports $n=2$3 and AMDP $n=2$4 (Chen et al., 28 May 2026).

Measured against wall-clock time rather than iterations, AMDP reaches target losses $n=2$5–$n=2$6 faster than the best synchronous baseline (Chen et al., 28 May 2026).

The paper also reports a direct comparison to vNAG on two GPT models, $n=2$7M and about $n=2$8B parameters, both with $n=2$9 and $\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$0k iterations. Throughput is $\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$1 vs $\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$2 on the smaller model and $\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$3 vs $\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$4 on the larger one, corresponding to $\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$5 and $\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$6 gains, while the final loss remains close to synchronous and vNAG is significantly worse (Chen et al., 28 May 2026).

Ablations show that moderate gradient accumulation is preferred. For thresholds $\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$7, corresponding to $\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$8 on 8 GPUs, throughput peaks at threshold $\operatorname{mismatch}(i)=\min(2,d-i)-1 \le 1.$9: GPT reaches $d$00 and BERT $d$01, with slight regression at threshold $d$02. Disabling ZeRO increases average GPU memory by $d$03–$d$04 and reduces throughput by about $d$05 (Chen et al., 28 May 2026).

6. Relation to adjacent asynchronous and synchronous pipeline research

AMDP’s novelty is structural. Earlier asynchronous methods such as PipeDream, PipeDream‑2BW, XPipe, SpecTrain, PipeMare, and vNAG either rely on weight stashing or on weight prediction, but do not impose a depth-independent one-step mismatch bound by construction. AMDP instead derives

$d$06

and then fixes $d$07, so $d$08 for all stages, independent of depth (Chen et al., 28 May 2026).

This places AMDP close to the constant one-step-delay regime studied in later asynchronous pipeline work. In particular, empirical analysis of PipeDream‑2BW argues that one-step gradient delay is not intrinsically unstable; degradation depends strongly on optimizer choice, and with Muon plus an Error-Feedback-inspired correction the gap to synchronous training can be closed even at $d$09B scale (Zmushko et al., 29 Jun 2026). That result complements AMDP’s structural bound: one line of work limits the delay itself, and the other studies how to optimize effectively once delay is limited.

A separate bounded-delay perspective appears in asynchronous Nesterov analysis, where delayed gradients are evaluated at a historical look-ahead point and discounted by $d$10, yielding an $d$11 convergence rate under fixed delay and strong empirical results on decoder-only models up to $d$12B parameters (Ajanthan et al., 2 May 2025). This suggests an optimizer-side path for AMDP beyond AdamW, although AMDP’s own contribution is schedule- rather than optimizer-centric.

Another orthogonal line attacks delay through geometry rather than scheduling. Basis rotation methods argue that delayed gradients become particularly harmful when the Hessian eigenbasis is misaligned with the optimizer’s coordinate basis, and report that training a $d$13B-parameter LLM with basis rotation reaches the same training loss in $d$14 fewer iterations than the best asynchronous pipeline baseline (Jung et al., 3 Feb 2026). For AMDP, that identifies a complementary optimization-side mechanism for deeper or more heterogeneous asynchronous graphs.

On the systems side, AsyncMesh extends full asynchrony to both pipeline and data parallel dimensions, using AsyncPP for the pipeline axis and asynchronous sparse averaging with EMA-based staleness correction for the data axis; on LLMs up to $d$15B parameters it matches a fully synchronous baseline while reducing communication overhead (Ajanthan et al., 30 Jan 2026). JaxPP, by contrast, provides a task-graph-based MPMD runtime for asynchronous execution of SPMD tasks and user-defined pipeline schedules, improving hardware utilization by up to $d$16 relative to the best SPMD configuration (Xhebraj et al., 2024). These systems indicate two plausible extensions of AMDP: additional asynchronous parallel axes and more general schedule runtimes.

AMDP must also be evaluated against strong synchronous baselines. Zero-bubble synchronous pipeline scheduling splits backward into activation-gradient and weight-gradient components and can outperform 1F1B by up to $d$17 under similar memory limits and up to $d$18 when memory is relaxed (Qi et al., 2023). Controllable-memory pipeline schedules further show that almost zero bubbles can be achieved while maintaining the same activation memory as 1F1B, or that memory can be reduced to one half or one third of 1F1B with competitive throughput (Qi et al., 2024). These methods do not remove the need for a staleness analysis, but they narrow the purely systems-based case for asynchrony.

7. Limitations and open directions

AMDP’s convergence analysis assumes a single nonconvex objective with one-step bounded delay, $d$19-smoothness, unbiased gradients with bounded variance, and a fixed optimizer-step structure. The paper does not analyze time-varying delay beyond the one-step structural guarantee, parameter sharing across branches with inconsistent model views, or mixed topologies combining data, tensor, and graph-structured model parallelism in one theory (Chen et al., 28 May 2026).

The schedule still leaves a “middle bubble” caused by forward/backward cost asymmetry; unlike zero-bubble synchronous methods, AMDP does not attempt to remove that asymmetry by splitting backward into activation-gradient and weight-gradient tasks (Chen et al., 28 May 2026). This suggests a possible synthesis with zero-bubble scheduling ideas (Qi et al., 2023).

A second open direction is optimizer design under bounded asynchrony. One-step-delay studies indicate that optimizer choice rather than delay alone can dominate stability, and basis-rotation work indicates that curvature alignment materially changes delayed-gradient behavior (Zmushko et al., 29 Jun 2026, Jung et al., 3 Feb 2026). This suggests that structurally bounded-delay schedulers such as AMDP could benefit from optimizer choices explicitly designed for stale updates.

A third direction is extending AMDP beyond pure pipeline parallelism. Asynchronous 2D mesh training already combines asynchronous pipeline and asynchronous data parallelism with formal consensus guarantees (Ajanthan et al., 30 Jan 2026), and MPMD runtimes already support arbitrary user-defined pipeline schedules (Xhebraj et al., 2024). A plausible implication is that AMDP can be regarded as one point in a larger design space of bounded-staleness, multi-axis asynchronous training systems, where the central question is no longer whether asynchrony is permissible, but how staleness is bounded, corrected, and composed across directions.