PPipe: Inference Serving for Heterogeneous GPU Clusters

• PPipe is an inference serving framework for video analytics that partitions CNN models to match layer latencies with diverse GPU capabilities.
• It employs an MILP-based control plane to optimize model partitioning, GPU allocation, and batch sizes, ensuring balanced throughput under strict tail-latency SLOs.
• Resource reservation–based adaptive batching in PPipe leads to 32.2%-75.1% higher throughput and significantly improved low-class GPU utilization compared to baselines.

PPipe is an inference serving system for video analytics on heterogeneous GPU clusters that applies pipeline parallelism to latency-bound model inference rather than to throughput-oriented training. It is designed around the observation that per-layer inference latency ratios between low-class and high-class GPUs vary widely across a CNN, so some model stages can execute on low-class GPUs with latency comparable to that of high-class devices. PPipe exploits this layerwise and hardware heterogeneity through pool-based pipeline parallelism, an MILP-based control plane, and a resource reservation-based adaptive batching data plane. In reported evaluations on diverse workloads comprising 18 CNN models, it achieves 41.1% - 65.5% higher utilization of low-class GPUs while maintaining high utilization of high-class GPUs, leading to 32.2% - 75.1% higher serving throughput compared to various baselines (Kong et al., 24 Jul 2025).

1. Problem setting and motivation

PPipe targets modern video-analytics pipelines such as object detection or segmentation on camera streams, where systems must satisfy strict tail-latency SLOs while sustaining high throughput under bursty arrivals. The deployment setting is a heterogeneous GPU fleet, with mixes of “high-class” and “low-class” cards in public-cloud and on-premise clusters. The central systems problem is that conventional inference-serving strategies underutilize cheaper GPUs when latency targets are tight (Kong et al., 24 Jul 2025).

Two baseline strategies are identified as inadequate in this regime. Data-parallel replication of the entire model on each GPU makes low-class cards much slower on a per-request basis, so SLO compliance forces tiny batches or no useful batching at all, yielding near-zero utilization of low-class GPUs. Classic load balancing across heterogeneous servers is also limited, because assigning whole-model inference to low-class GPUs remains infeasible for many models once measured end-to-end latency is compared against the SLO. In both cases, the system fails to exploit the fact that only some portions of a model are severely disadvantaged on slower hardware (Kong et al., 24 Jul 2025).

The key empirical observation underlying PPipe is that different layers or blocks of layers within a CNN exhibit different compute patterns and tensor sizes, and therefore different low-class/high-class latency ratios. Early layers may show latency ratios close to $1\text{–}2\times$, whereas later layers with large channel counts may be $5\text{–}10\times$ slower on a P4 than on an L4. This suggests that whole-model placement is too coarse a serving abstraction for heterogeneous clusters, and that stage-level placement can recover otherwise unusable capacity (Kong et al., 24 Jul 2025).

2. Pool-based pipeline parallelism

PPipe’s core abstraction is pool-based pipeline parallelism. The model is partitioned into $D$ stages, each consisting of blocks of consecutive layers, and each stage is assigned to a “pool” of one or more identical GPUs of the same class. Inference requests are then pipelined stage by stage: any GPU in the pool for one stage can process that stage for any request before forwarding the intermediate feature map to any GPU in the next stage’s pool (Kong et al., 24 Jul 2025).

This differs from a single fixed inference chain. Pool sizes, batch sizes, and stage latencies can differ across the pipeline, and PPipe enforces only that all pools achieve the same steady-state throughput so that no stage becomes a chronic bottleneck. The design is therefore many-to-many rather than chain-like. A plausible implication is that PPipe uses pipeline parallelism as a resource-matching mechanism rather than merely as a schedule template (Kong et al., 24 Jul 2025).

The approach is particularly well suited to heterogeneous clusters because stage placement can be matched to hardware characteristics. Stages with relatively modest low-class/high-class slowdown can be assigned to low-class GPU pools, while computationally heavier later stages can remain on high-class pools. The paper characterizes this as exploiting the synergy between diversity in model layers and diversity in GPU architectures, with comparable inference latency for many layers on low-class and high-class GPUs (Kong et al., 24 Jul 2025).

A common misconception is to equate PPipe with conventional pipeline parallelism from training. In PPipe, the objective is serving latency-bound inference under tail-SLO constraints, not throughput-oriented parameter updates. Another misconception is to treat it as ordinary model partitioning alone. The defining feature is the combination of partitioning, pool allocation, unified per-pipeline batch sizing, and end-to-end reservation-aware dispatch (Kong et al., 24 Jul 2025).

3. MILP-based control plane

PPipe’s control plane is formulated as an offline Mixed-Integer Linear Program that jointly decides four quantities: how to cut the model into $D$ partitions, which GPU class to assign to each partition, how many virtual GPUs $g_{ld}$ to allocate, and the unified batch size $b_l$ for all partitions in each pipeline $l$ (Kong et al., 24 Jul 2025).

For a single DNN, the formulation uses the notation $M$ for the total number of layers or pre-grouped blocks, $D_l$ for the number of partitions in pipeline $l$, $5\text{–}10\times$0 for the number of GPU classes, $5\text{–}10\times$1 for candidate batch sizes, and $5\text{–}10\times$2 for virtual-GPU fractions such as $5\text{–}10\times$3. Profiling supplies $5\text{–}10\times$4, the latency of block $5\text{–}10\times$5 under batch size $5\text{–}10\times$6 on a $5\text{–}10\times$7-fraction of a class-$5\text{–}10\times$8 GPU; $5\text{–}10\times$9, the feature-map size at block-$D$0 boundary; and $D$1, the transfer latency of that feature map at batch $D$2 (Kong et al., 24 Jul 2025).

The decision variables include binary partition selectors $D$3, integer GPU-allocation variables $D$4, per-partition latency variables $D$5, per-partition throughput variables $D$6, inter-partition transfer latency variables $D$7, and per-pipeline throughput variables $D$8. The objective is

$D$9

The constraint system includes valid partition selection, contiguity of consecutive partitions, per-class GPU resource limits, pipeline latency SLO constraints, compute and throughput definitions, unified batch size within a pipeline, and bottleneck throughput defined as

$D$0

The “min” is converted into linear constraints, so the optimization remains MILP-solvable by Gurobi (Kong et al., 24 Jul 2025).

The compute and throughput definitions are expressed using

$D$1

where

$D$2

and

$D$3

The resulting planner chooses which partitions to use, how many GPUs of each class to allocate, and which unified batch size to apply. The output pipelines are those with $D$4 (Kong et al., 24 Jul 2025).

To reduce solver size, models are first pre-partitioned into $D$5 uniform-latency blocks. The control plane runs periodically, for example every 1–2 h or upon a major load shift, and migrates to a new plan by asynchronously loading new model weights, flushing pipelines on the order of 100 ms, and swapping in the new plan. This periodic replanning distinguishes PPipe from systems that rely only on online load balancing (Kong et al., 24 Jul 2025).

4. Resource reservation–based adaptive batching

The MILP plan assumes synchronized arrivals and negligible queueing jitter, but actual inference requests arrive asynchronously and may follow Poisson or self-similar burst patterns. PPipe’s data plane therefore uses a single adaptive batching scheduler based on resource reservation. The scheduler maintains a global reservation table recording when each GPU and each NIC link is booked in the future (Kong et al., 24 Jul 2025).

At dispatch time, the scheduler probes each pipeline at its target batch size $D$6 and chooses the pipeline with the smallest predicted waiting time. Once a candidate pipeline $D$7 is selected, the scheduler backs off batch size from $D$8 down to 1 until the predicted end-to-end completion time is no greater than the earliest deadline among queued requests. If even batch size 1 cannot satisfy the deadline, the oldest request is dropped. If the queue has not yet filled to the chosen batch size, the scheduler waits up to the last safe moment before dispatching (Kong et al., 24 Jul 2025).

The supporting probe procedure greedily tracks a time cursor, evaluates candidate GPUs in each stage pool, finds the earliest common free slots on the relevant NIC endpoints, reserves compute intervals on the chosen GPU, and returns a path, reservation list, and predicted completion time. Reserved resources include intervals on each GPU’s compute engine and each NIC link. Actual execution returns an ACK with start and end times so that the scheduler can correct drift through feedback (Kong et al., 24 Jul 2025).

Three properties are explicitly claimed for this scheme. First, no hidden queueing occurs behind fully reserved resources, so inter-partition delays are predictable. Second, NIC contention is avoided by atomic NIC reservations on both ends. Third, batch size dynamically accommodates arrival variability, so $D$9 tail SLO attainment is achieved (Kong et al., 24 Jul 2025).

The reservation model also provides PPipe’s backpressure behavior. Because each batch’s resources are reserved end to end before dispatch, no stage saturates unexpectedly, and GPUs do not sit idle waiting for upstream data. This suggests that PPipe replaces reactive queue-based coordination with proactive reservation-based coordination across compute and network resources (Kong et al., 24 Jul 2025).

5. End-to-end execution model

PPipe’s execution flow has three stages: offline profiling and partitioning, control-plane optimization, and runtime data-plane scheduling. In the profiling stage, each pre-partitioned block is profiled on every GPU class, virtual-GPU fraction, and candidate batch size to produce $g_{ld}$0. This profiling information, along with feature-map sizes and transfer latencies, feeds the MILP planner (Kong et al., 24 Jul 2025).

The control plane emits a set of $g_{ld}$1 pool-based pipelines. For each pipeline it specifies the block ranges for each partition, the GPU class for each stage, the number of virtual GPUs $g_{ld}$2, and the unified batch size $g_{ld}$3. At runtime, a single global scheduler receives incoming requests, runs the adaptive batching procedure on arrivals or service completions, packetizes the chosen batch, and sends it to the Stage 0 pool at the reserved slot (Kong et al., 24 Jul 2025).

After the first stage executes its partition, it transfers a quantized fp16 feature map to the Stage 1 GPU using NCCL or TCP, again according to the reservation. The same pattern continues through the pipeline until final-stage completion, at which point results such as object detections are returned to the client. The end-to-end execution model is therefore explicitly distributed across both compute and network resources rather than only GPU kernels (Kong et al., 24 Jul 2025).

PPipe also incorporates virtual GPUs through MPS fractions. In the control-plane contribution summary, GPU allocation is described as including MPS virtual GPUs, which allows the optimizer to assign fractions of a physical GPU to different partitions or workloads where beneficial. This is part of the joint optimization of partition points, GPU allocation, and unified batch sizes for maximum throughput under SLO constraints (Kong et al., 24 Jul 2025).

6. Evaluation, operating regimes, and limitations

The reported evaluation spans four heterogeneous 100-GPU simulations with $g_{ld}$4 high-class and $g_{ld}$5 low-class GPUs, along with 16-GPU GCP testbeds. Workloads comprise 18 diverse CNNs, including EfficientNet, ResNet variants, and detection and segmentation networks. The SLO is set to $g_{ld}$6 single-batch L4 latency, and requests are drawn from Microsoft Azure Function traces with both 2019 Poisson and 2021 bursty patterns (Kong et al., 24 Jul 2025).

The main quantitative results are summarized below.

Metric	Reported result	Comparison
Low-class GPU utilization	41.1% - 65.5% higher	Versus baselines
Serving throughput	32.2% - 75.1% higher	Versus baselines
Max load factor, Poisson	34%–65% higher	Versus No-Partition
Max load factor, Bursty	34%–162% improvement	Versus No-Partition
Throughput, Poisson	17%–47% higher	Versus DART-r
Throughput, Bursty	18%–50% higher	Versus DART-r

Against the “No-Partition” baseline using the same adaptive scheduler, PPipe achieves 34%–65% higher max load factor under the Poisson trace, corresponding to up to $g_{ld}$7 throughput, and 34%–162% improvement under the bursty trace, corresponding to up to $g_{ld}$8 throughput. Against the single-chain pipeline baseline DART-r, the reported gains are 17%–47% under Poisson and 18%–50% under bursty arrivals (Kong et al., 24 Jul 2025).

GPU-utilization results are central to the system’s claim. High-class GPUs are saturated by all methods, but low-class GPUs remain at $g_{ld}$9 utilization for No-Partition, approximately $b_l$0 for DART-r, and approximately $b_l$1 for PPipe. In SLO terms, PPipe sustains 99% tail attainment at near its MILP-predicted load factor, approximately $b_l$2, while No-Partition and DART-r drop sharply once offered load exceeds approximately $b_l$3 (Kong et al., 24 Jul 2025).

The system overheads are also quantified. With pre-partitioning to 10 blocks, the control-plane MILP solve time is approximately 3.5 s. The solver scales to 100k GPUs unchanged, or to 4 GPU classes in approximately 77 s, and is triggered hourly. Data-plane scheduling cost is $b_l$4 per dispatch, with approximately 3.6 probes on average and approximately $b_l$5 overhead per batch (Kong et al., 24 Jul 2025).

The paper also delineates operating regimes where PPipe’s advantage narrows or disappears. More aggressive SLO discount margins increase safe batch sizes but reduce ideal throughput; robust gains greater than 25% are reported across 20%–60% margins, with a peak at 40%. When SLO scales tighten to $b_l$6, low-class GPUs become unusable and PPipe falls back to No-Partition. Conversely, at a very loose $b_l$7 SLO, No-Partition already uses low-class cards, so PPipe’s relative gain shrinks (Kong et al., 24 Jul 2025).

The stated limitations are that PPipe currently handles static CNN-like models partitioned at coarse granularity. Future extensions proposed in the paper include transformer-based layers and recurrent networks, online learning of latency profiles for faster adaptation, joint optimization with model quantization such as INT8, and extending the MILP to co-serve many DNNs in a multi-tenant environment with fairness or SLAs per application. These limitations clarify that PPipe is not presented as a universal serving substrate for all model families, but as a targeted design for heterogeneous-cluster inference where stagewise latency heterogeneity is substantial (Kong et al., 24 Jul 2025).