Inter-Operator Parallelism

Updated 5 December 2025

Inter-operator parallelism is a technique that partitions computation graphs into sequential stages across devices, enabling scalable deep learning and stream processing.
It employs pipeline scheduling methods like 1F1B and interleaved execution to overlap computation and communication, thereby maximizing hardware utilization.
The approach integrates with intra-operator parallelism and dynamic scheduling algorithms to balance memory and compute loads, achieving significant throughput improvements.

Inter-operator parallelism, also referred to as pipeline parallelism, is a model and implementation strategy for distributing computations by partitioning a computation graph or dataflow—commonly for deep neural networks or stream processing—into sequential or partially overlapping stages, each assigned to different hardware units (devices, cores, or nodes). At each stage, entire operators or layers are executed, and intermediate data are routed point-to-point between stages. This approach enables scalable execution of models and dataflows that cannot fit in the memory of a single device or require low-latency, high-throughput operation, especially for large models and distributed infrastructures.

1. Formal Definition and Distinction from Intra-Operator Parallelism

Inter-operator parallelism partitions a directed acyclic computation graph (DAG) of operators into $S$ sequential “stages,” each mapped to a distinct device or group of devices. Each stage holds contiguous or structured subgraphs such that, during execution, upstream stages compute outputs (activations), which are sent directly to downstream stages for further processing.

Mathematically, given an operator graph $\mathcal O = (V, E)$ , a partition is defined by a surjective mapping $\varphi: V \to \{d_1, ..., d_S\}$ . Each device executes its assigned subgraph $G_i=(V_i, E_i)$ where $V_i=\{v \mid \varphi(v)=d_i\}$ and $E_i=\{(u\to v) \in E \mid \varphi(u)=\varphi(v)\}$ . Whenever $e = (u \to v)$ crosses stage boundaries, tensors are communicated between the source and destination devices (Brakel et al., 6 Mar 2024, Zhuang et al., 2022, Zheng et al., 2022).

This paradigm contrasts with intra-operator (or tensor) parallelism, where large operators are partitioned internally—e.g., sharding a matrix multiplication across devices with collective communication primitives (all-reduce, all-gather, or all-to-all) at operator boundaries (Brakel et al., 6 Mar 2024, Zhuang et al., 2022).

2. Execution Models, Scheduling, and Algorithms

Pipeline Schedules

Inter-operator parallelism relies on pipelining multiple micro-batches through consecutive stages to maximize hardware utilization and minimize idle time—prototypically in the form of 1F1B schedules (one forward, one backward) or breadth-first interleaved schedules:

Vanilla 1F1B: Stage $i$ computes the forward pass on micro-batch $b$ , immediately sends activations to stage $i+1$ , which processes them in turn. After all $B$ micro-batches are streamed forward, the backward pass sweeps back through the pipeline, transmitting gradients in reverse (Zhuang et al., 2022, Brakel et al., 6 Mar 2024).
Eager-1F1B/Blockwise Interleaved: Forward computations of later micro-batches are launched ahead of their communication, overlapping cross-stage comms with computation, increasing pipeline “windows” where communication does not stall compute. This reduces communication-limited “bubbles” and approaches near-linear throughput scaling for sufficiently large micro-batch or unit sizes (Zhuang et al., 2022, Tang et al., 6 Feb 2024).

The pipeline schedule can be generalized as:

$T = \sum_{i=1}^S t_i + (B-1)\max_j t_j$

where $t_i$ is the per-stage latency and $B$ is the number of micro-batches (Zheng et al., 2022).

Partitioning and Assignment

Dynamic Programming: Stage boundaries (i.e., which operator belongs to which stage) are chosen to balance computational and memory loads via DP recurrences or cost models (Brakel et al., 6 Mar 2024, Ding et al., 2020, Zheng et al., 2022).
Mixed-Integer Programming: Joint optimization of inter-layer (pipeline) and intra-layer (tensor) parallelism may use MIQP, modeling layer-to-stage placement, micro-batch degree, memory constraints, and inter-/intra-stage communication (Lin et al., 2023).
Graph Partitioning: For streaming systems, parallelism tuning also considers operator bottlenecks, historical job structure (via Graph Edit Distance clustering), and online GNN-based bottleneck predictors (Han et al., 16 Apr 2025).

Integration with System Frameworks

Implementations leverage CUDA Streams, CUDA Graphs, MPMD/SPMD executables, or custom runtime orchestrators, enabling both concurrent and merged operator execution within or across stages (Chen et al., 2023, Ding et al., 2020, Zheng et al., 2022).

3. Communication Patterns and Cross-Mesh Resharding

When combining inter- and intra-operator parallelism, cross-mesh resharding arises—data must be transferred between disjoint device meshes with potentially different sharding layouts (Zhuang et al., 2022, Zheng et al., 2022).

Formalization: For a tensor $D$ , with sharding layouts $\mathcal X_s$ on source mesh $M_s$ and $\mathcal X_d$ on destination mesh $M_d$ , the resharding operation routes each source “data slice” $DS^s_i$ to its required target devices $M_i$ via a multicast pattern.
Broadcast-Based Scheduling: Optimal resharding uses a broadcast ring where data slices are pipelined to all destination devices, reducing the amortized latency to a single message cost for large partition numbers (Zhuang et al., 2022).
Eager Overlap: Embedding multicasts within an overlapping-friendly pipeline hides cross-mesh communication behind useful compute, eliminating nearly all resharding latency in end-to-end throughput (Zhuang et al., 2022).

Empirical evidence demonstrates up to $10\times$ speedup versus all-gather-based resharding in challenging layouts, with $10$– $50\%$ overall throughput improvement on large GPT-3/U-Transformer models (Zhuang et al., 2022).

4. Applications Across Models and Domains

Inter-operator parallelism is applied broadly across deep learning, streaming architectures, and distributed data analytics:

LLMs: Megatron-LM, Gopher, and PaLM exploit deep pipeline parallelism, often combined with tensor-parallelism within nodes and data-parallelism externally (Brakel et al., 6 Mar 2024, Zheng et al., 2022). Up to $64$-way or greater pipelining is reported in production settings.
DNN Inference: Frameworks such as IOS and Opara apply inter-operator scheduling and stream-wise kernel fusion to raise GPU utilization and throughput by factors of $1.1$– $1.68\times$ across CNN and Transformer inference tasks (Ding et al., 2020, Chen et al., 2023).
Stream Processing: Adaptive operator-level parallelism tuning in DAG-structured jobs (e.g., in Flink, Timely Dataflow) leverages per-operator parallelism degrees set by bottleneck predictors and global profiling (Han et al., 16 Apr 2025, Prasaad et al., 2018).
Pipeline-Only Systems: ZeroPP demonstrates that, for certain hardware regimes and model architectures, pipeline parallelism paired with fully sharded data parallelism matches or surpasses tensor-parallel strategies, enabling up to $33\%$ performance gain while reducing memory pressure (Tang et al., 6 Feb 2024).

5. Performance Modeling, Trade-offs, and Bottlenecks

Throughput and Utilization

The pipeline throughput in a balanced setting is limited by the slowest stage and the extent to which idle “bubbles” (due to micro-batch phase mismatches) can be reduced by increasing the number of micro-batches. Utilization approaches unity as $U \geq 2P-1$ (optimal scheduling unit sizes for $P$ pipeline stages), with utilization given by $U_{\text{pp}} \approx B/(B+P-1)$ (Tang et al., 6 Feb 2024).

Communication

Inter-Stage: Point-to-point communication of activations and gradients is lower bandwidth and more easily overlapped with compute than the all-reduce/gather collectives required by intra-operator techniques.
Cross-Mesh: Heterogeneous mesh boundaries require mapping between different sharding layouts; optimal pipelines must minimize redundant data movement during resharding.

Memory

Each stage must buffer activations for multiple micro-batches, leading to memory–throughput trade-offs; blockwise grouping (as in ZeroPP) constrains activation memory per device to $(L M_w)/P + \min(B,U)(L M_a)/P$ (Tang et al., 6 Feb 2024).

Limitations

Pipeline-only execution is susceptible to issues with unbalanced per-stage computation, excessive activation memory storage, and is sensitive to communication bandwidth when cross-node interconnects are weak (Brakel et al., 6 Mar 2024, Zheng et al., 2022, Lin et al., 2023).

6. Scheduling, Optimization, and Automation

Advanced systems jointly optimize inter- and intra-operator parallelism via hierarchical or unified optimization spaces:

Hierarchical Optimization: Alpa builds a two-level plan space: outer dynamic programming over stage assignment and mesh partitioning, inner ILP or heuristic planning for optimal intra-operator sharding plans (Zheng et al., 2022).
Unified MIQP: UniAP models both inter-layer and intra-layer strategies as binary placement and selection variables, solving a mixed-integer quadratic program with constraints for memory, contiguity, and stage balancing. This achieves up to $3.80\times$ throughput improvement and $107\times$ search speedup over prior strategies (Lin et al., 2023).
Learning-Based Search: Reinforcement learning methods (as in Learn to Shard) explore both coarse (pipeline, tensor, expert) and fine per-operator sharding dimensions, yielding $1.06\times$ over Megatron heuristics and up to $3.5\times$ over metaheuristics for LLM inference (Yin et al., 29 Aug 2025).
Streaming Tuning: Operator-level parallelism is adaptively tuned using historical DAG structure and per-node GNN encoders, enforcing monotonic bottleneck predictors for resource-efficient configuration (Han et al., 16 Apr 2025).

7. Best Practices and Implementation Guidelines

Combine intra-operator (tensor) parallelism within high-bandwidth domains (intra-node) and inter-operator (pipeline) across nodes to match hardware topology and minimize heavy collectives (Zheng et al., 2022, Brakel et al., 6 Mar 2024).
Choose micro-batch counts or scheduling window sizes that maintain pipeline utilization ( $U \geq 2P-1$ ) and fit within memory budgets (Tang et al., 6 Feb 2024).
Employ overlapping-friendly schedules that overlap cross-stage communication with compute, particularly under hybrid intra-/inter-operator models (Zhuang et al., 2022, Tang et al., 6 Feb 2024).
Exploit operator merging and concurrent execution cases for inference, maximizing the active warps and improving GPU utilization (Ding et al., 2020, Chen et al., 2023).
Leverage graph-structured historical profiling and scalable bottleneck-prediction models to tune per-operator parallelism in distributed dataflow environments (Han et al., 16 Apr 2025).
For highly irregular operator DAGs, solve partitioning with DP, ILP, or metaheuristic search, balancing per-stage compute, communication, and memory to minimize end-to-end latency (Brakel et al., 6 Mar 2024, Lin et al., 2023).
When scaling to very large models, integrate cross-mesh broadcast/multicast and sender-load balancing for efficient resharding between heterogeneous pipeline stages (Zhuang et al., 2022).

In summary, inter-operator parallelism is a foundational technique for decomposing and scaling large computation graphs over distributed hardware, particularly in the context of neural network training and inference at billion-to-trillion parameter scale, as well as high-throughput stream processing systems. It is distinguished by macro-level partitioning of the operator graph, pipelined execution with careful scheduling, and a set of optimization, communication, and scheduling considerations that define its performance and applicability. Seamless integration with intra-operator parallelism, adaptive scheduling based on profiling or reinforcement learning, and optimal communication primitives are essential for approaching ideal hardware utilization and reducing resource cost in heterogeneous large-scale deployments (Brakel et al., 6 Mar 2024, Zhuang et al., 2022, Zheng et al., 2022, Lin et al., 2023, Tang et al., 6 Feb 2024, Ding et al., 2020, Chen et al., 2023, Yin et al., 29 Aug 2025, Han et al., 16 Apr 2025, Prasaad et al., 2018).