PCIe-Aware Placement in GPU Systems

• PCIe-aware placement is a strategy that explicitly incorporates PCIe topology into scheduling to optimize data movement in GPU, MoE, and RPC systems.
• It leverages tenant migration, expert assignment, and memory placement techniques to reduce tail latency, SLO miss-rate, and inter-device traffic.
• Analytical models and empirical results confirm that topology-sensitive placement improves performance under PCIe contention in heterogeneous environments.

Searching arXiv for the cited papers and closely related terms to ground the article. PCIe-aware placement is the class of scheduling, mapping, and data-placement techniques that explicitly treat the PCIe fabric as a shared, performance-critical resource rather than a transparent transport. In the cited work, the term covers at least three distinct but related settings: moving a tenant’s process to a different MIG instance or GPU slot on a less contended PCIe path in shared GPU inference; placing Mixture-of-Experts components so that co-activated experts minimize expensive inter-GPU or inter-node transfers; and placing RPC data structures on the host or a PCIe-attached accelerator to reduce transaction-rate pressure, dereference latency, and superfluous traversals (Darzi et al., 27 Aug 2025, Shi et al., 6 Jul 2026, Zhang et al., 2024). Across these settings, the common objective is to reduce tail latency, SLO miss-rate, or communication overhead while preserving throughput or accuracy under the constraints imposed by PCIe roots, switches, NUMA locality, and device-level data movement.

1. Core concept and scope

In shared GPU serving, “PCIe-aware placement” denotes moving a tenant’s process to a different MIG instance or GPU slot that sits on a less contended PCIe path. The placement logic is explicitly aware of PCIe root complexes, downstream switches, and host-side NUMA or I/O activity, and it avoids co-locating latency-sensitive tenants with bandwidth-heavy neighbors that share the same PCIe root or a hot NUMA domain. This point is conceptually important because MIG provides hard isolation for SM and HBM, but MIG instances still share the PCIe fabric; the placement mechanism therefore targets a bottleneck that compute partitioning alone does not remove (Darzi et al., 27 Aug 2025).

In expert-parallel MoE inference, PCIe-aware placement has a different granularity. The relevant decision variable is the expert-to-GPU assignment $\pi : E \to G$, and the key observation is that communication volume is governed by “devices per token” rather than “experts per token.” Co-locating frequently co-activated experts reduces device fan-out and thus reduces inter-GPU and inter-node traffic. The corresponding topology model can distinguish same-GPU, NVLink, same-switch PCIe P2P, PCIe host-mediated, and inter-node paths through a weight $w_{g,g'}$, making the method sensitive to whether a machine is PCIe-bound, NVLink-rich, or bandwidth-constrained across nodes (Shi et al., 6 Jul 2026).

In PCIe-attached RPC acceleration, placement is not primarily a process-migration problem but a host-versus-accelerator memory-placement problem. RPCAcc uses a schema table to determine, per field, whether the target location is host CPU memory or accelerator HBM. Fields needed by offloaded kernels are placed in accelerator memory; host-only fields are accumulated for one-shot DMA to host memory. In this context, PCIe-aware placement means placing data where it will be consumed, coalescing host-bound transfers, and avoiding pointer-chasing across the PCIe boundary whenever possible (Zhang et al., 2024).

A plausible implication is that PCIe-aware placement is best understood as a cross-layer systems principle: it can operate at the level of tenant placement, expert assignment, or field placement, but in each case it treats topology and live transport costs as first-class scheduling inputs.

2. Interference mechanisms and analytical models

The immediate motivation for PCIe-aware placement in shared LLM serving is noisy-neighbor interference. Host-to-device and device-to-host DMA can saturate shared PCIe links, especially when one tenant performs ETL-style streaming from NVMe into host memory, transfers to GPU for transformation, and writes back, or when a compute-bound training tenant still generates periodic DMA through dataloading or checkpointing. In the reported LLM setup, vLLM with OLMo 2 7B Instruct uses a paged KV cache and default configuration with batch size $=1$; PCIe traffic arises at TTFT through prompt upload and runtime state movement, and episodically during generation through KV paging and microbatch scheduling. Under such conditions, bursty DMA increases queueing at the PCIe stage and inflates p99 and p999 latency (Darzi et al., 27 Aug 2025).

The paper models PCIe as a processor-sharing server of capacity $B$, with instantaneous bandwidth for tenant $i$ given by

$$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$

End-to-end latency is then decomposed as

$L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$

Queueing intuition is expressed via Kingman’s approximation,

$$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$

with $\rho \to 1$ inflating tails. SLO miss-rate is formalized as

$$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$

and throughput is

$w_{g,g'}$0

subject to the constraint $w_{g,g'}$1 (Darzi et al., 27 Aug 2025).

In MoE inference, the central model is not queueing on a shared PCIe path for multiple tenants, but communication cost under topology-aware expert placement. CAP defines

$w_{g,g'}$2

where $w_{g,g'}$3 is expected co-activation intensity, $w_{g,g'}$4 is per-co-activation bytes transferred, and $w_{g,g'}$5 encodes bandwidth, latency, and path class. A simpler tiered model uses $w_{g,g'}$6 if same GPU, $w_{g,g'}$7 if intra-node, and $w_{g,g'}$8 if inter-node; a more refined PCIe/NVLink-aware version distinguishes $w_{g,g'}$9, $=1$0, $=1$1, and $=1$2. CAP then trades communication against imbalance by approximately optimizing

$=1$3

with

$=1$4

Its runtime proxy further decomposes communication as

$=1$5

This formalism makes explicit why PCIe-only machines without P2P push $=1$6, whereas NVLink-rich nodes can tolerate more communication to improve balance (Shi et al., 6 Jul 2026).

RPCAcc introduces a third analytical perspective: PCIe can be transaction-rate-bound even before bandwidth saturates. Naïve field-by-field deserialization performs poorly because many small cross-PCIe writes incur per-transaction overhead and latency. If $=1$7 host-bound fields of total size $=1$8 are written separately, then

$=1$9

whereas one-shot DMA yields

$B$0

With a fraction $B$1 of bytes placed directly into accelerator memory, host-bound traffic becomes

$B$2

and the cost is correspondingly reduced. The same paper also formulates a break-even condition showing when extra host memory copies are beneficial because they replace deep PCIe dereferences with CPU-local contiguous layout materialization (Zhang et al., 2024).

3. Host-level PCIe-aware placement for shared GPU inference

The host-level controller in “Predictable LLM Serving on GPU Clusters” is described as host-local, fabric-agnostic, and VM-deployable. It integrates three levers: dynamic MIG reconfiguration, PCIe-aware placement, and lightweight guardrails. Its implementation uses NVML/DCGM, lspci, cgroups, and nvidia-smi, all usable inside a renter’s VM. Sampling occurs every $B$3–$B$4 s, signals are smoothed with exponential moving average and hysteresis, and action gating uses persistence, dwell, and cool-down to avoid thrash (Darzi et al., 27 Aug 2025).

Topology discovery combines NVML/DCGM with host tools. NVML/DCGM provides GPU inventory, SM and memory utilization, device UUIDs, and basic topology hints. lspci exposes the PCIe tree, including root complexes, switches, and device BDFs. NUMA maps from /sys and numactl provide CPU and memory locality, while NIC and IRQ statistics may be sampled optionally. The controller represents at least four topology attributes: PCIe root-complex or switch adjacency, lane or generation hints implicitly captured via utilization counters, NUMA locality, and optional NIC adjacency or IRQ noise. NVLink presence is not required and is not directly used for inference placement in this design (Darzi et al., 27 Aug 2025).

For each candidate MIG instance, the controller computes a scalar score

$B$5

where lower is better. The exact values of $B$6 are not published. The decision logic is conservative: if p99 exceeds $B$7 for $B$8 consecutive windows and PCIe, link, or I/O signals indicate contention, the system prefers PCIe-aware placement over immediate MIG enlargement. It first attempts intra-GPU MIG remapping to a less-contended instance; if no safe slot exists, it considers an inter-GPU move or escalates isolation with a larger MIG profile. If the minimum candidate score falls below a conservative bound, the tenant is migrated; otherwise the controller falls back to guardrails such as MPS quotas or cgroup I/O throttling, or to MIG enlargement. A short post-change window validates the decision, and regression triggers rollback to the last-known-good state (Darzi et al., 27 Aug 2025).

The action sequence is also constrained temporally. The trigger condition is p99 $B$9 for $i$0 consecutive windows, with $i$1 by default, and no active cool-down. Dwell time is 256 observations between actions, and cool-down is 128 observations after returning to performance mode; with $i$2–$i$3 s, this yields minute-scale gating. The escalation order is Guardrails $i$4 PCIe-aware placement $i$5 MIG enlargement, while relaxation requires sustained stability and a throughput check (Darzi et al., 27 Aug 2025).

This design directly addresses a common misconception: enlarging a MIG profile does not eliminate PCIe contention. The reported design takeaway is explicit that MIG isolates compute and HBM, not PCIe, and that placement must be PCIe-aware to tame tails in multi-tenant inference (Darzi et al., 27 Aug 2025).

4. Communication-aware placement for MoE inference

In CAP, PCIe-aware placement is embedded in a broader communication-aware framework for MoE inference. The first component, co-activation driven expert placement, builds per-layer co-activation graphs from routing traces. For a layer with expert set $i$6, two experts $i$7 and $i$8 are co-activated when both lie in the token’s top-$i$9 routing set, and CAP computes

$$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$0

A weighted graph with edge weights $$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$1 is then partitioned using a greedy graph-growing heuristic. Weighted degree $$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$2 identifies high-connectivity seeds; the top-$$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$3 seeds initialize GPU groups, and groups grow round-robin by repeatedly adding the unassigned expert $$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$4 maximizing $$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$5. A second hierarchical stage aggregates GPU groups onto nodes using group-level co-activation, thereby minimizing inter-node traffic as well (Shi et al., 6 Jul 2026).

This construction is capacity-constrained. Each GPU must host the same number of experts in order to satisfy memory and KV-cache capacity and to avoid per-GPU memory bottlenecks. The resulting problem is therefore a capacity-constrained weighted $$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$6-way partition rather than an unconstrained communication minimization. The reported complexity is $$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$7 to build the co-activation matrix, $$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$8 to grow $$b_i(t) = \min\left\{ \frac{B w_i}{\sum_{j \in \mathcal{A}(t)} w_j},\ g_i \right\}.$$9 groups, and $L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$0 to group GPUs to nodes, which the paper states scales well to layers with 64–128 experts (Shi et al., 6 Jul 2026).

CAP’s second component, communication–computation trade-off adjustment, starts from the communication-minimal placement and iteratively swaps experts between high-load and low-load GPUs while minimally increasing communication cost. This produces a spectrum of placements parameterized by $L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$1, which the framework profiles to pick the best operating point for the hardware. On PCIe-only nodes without P2P, the guidance is to set $L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$2 close to $L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$3, aggressively co-locate high co-activation edges, avoid scattering co-activated experts across GPUs, and prune off-GPU destinations first. On PCIe P2P machines, a mid-spectrum placement may be preferable. On NVLink machines, the guidance shifts toward more balanced placements because communication is less dominant (Shi et al., 6 Jul 2026).

The third component, communication-aware expert pruning, operates at the device level rather than the expert level. For each token and destination device $L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$4, the method defines an accuracy contribution proxy

$L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$5

and a link cost $L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$6 that can encode same-device, intra-node, inter-node, or refined PCIe/NVLink topology costs. Pruning solves

$L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$7

implemented greedily by retaining devices in descending $L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$8 order until the retained accuracy mass reaches $L_i(t) = c_i + \frac{s_i}{b_i(t)} + \epsilon(t).$9. Because it removes expensive destinations first, this procedure directly reduces devices-per-token and thus PCIe or inter-node traffic (Shi et al., 6 Jul 2026).

The emphasis on “devices per token” rather than only on expert count corrects another common misunderstanding. A plausible implication is that on PCIe-bound MoE deployments, expert placement must be formulated as topology-aware communication minimization under capacity constraints, not merely as per-GPU load equalization.

5. Host–accelerator placement in PCIe-attached RPC systems

RPCAcc extends PCIe-aware placement to the microservice and RPC domain. The system targets commodity servers in which both the NIC and the FPGA accelerator are PCIe-attached devices. The reported platform uses two-socket servers with two 16-core Xeon Silver 4514Y at 2.0 GHz, 512 GiB DDR5-4800, a dual-port ConnectX-5 100 Gb NIC, and a Xilinx U280 accelerator. PCIe links operate at x16; the paper reports PCIe latency of approximately $$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$0 ns and effective throughput of approximately $$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$1 GB/s for the accelerator path. The paper explicitly states that transaction-rate limits dominate when writes are many and small (Zhang et al., 2024).

The target-aware deserializer uses two hardware data structures for placement-aware write coalescing: a Schema Table in SRAM and a 4 KB per-deserializer Temp Buffer in SRAM. The Schema Table stores, per message class, one placement bit per field indicating host versus accelerator target memory. The Temp Buffer accumulates host-bound fields for the current RPC and flushes them in a one-shot DMA write either at end-of-message or when full. Accelerator-bound fields are written directly to HBM, and parent pointers are updated to accelerator addresses when dereference fields are placed there. Nested sub-messages are handled through an SRAM-based stack for recursive descent. Empirically, one-shot DMA improves deserialization throughput by $$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$2 overall and by $$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$3 for messages with small fields of at most $$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$4 KB (Zhang et al., 2024).

The memory-affinity CPU–accelerator collaborative serializer introduces a different form of PCIe-aware placement. “Memory-affinity” here means NUMA-aware buffer placement, CPU core pinning to the NUMA domain attached to the relevant PCIe root complex, use of pinned DMA-safe pages for the staging buffer, and explicit skipping of accelerator-resident fields during host pre-serialization. The CPU pre-serialization stage iterates over host fields, copies non-contiguous sub-objects into a contiguous DMA-safe buffer, offloads large copies to Intel DSA memcpy engines, and sends descriptors to the accelerator via MMIO. The accelerator then builds the RPC header in an SRAM TX Arena, DMA-reads the contiguous buffer, performs hardware varint encoding at 512 bits per cycle, fetches accelerator fields from HBM, and merges the result for transmit. The paper reports that this serializer is $$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$5 faster than accelerator-only ProtoACC-PCIe and $$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$6 faster than CPU-only serialization on HyperProtoBench; memcpy offload reduces CPU cycles by $$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$7 on HyperProtoBench and $$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$8 on DeathStarBench, while memcpy plus encoding offload reduces CPU cycles by $$E[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)E[S],$$9 in both suites (Zhang et al., 2024).

The automatic field update mechanism handles dynamic placement changes. When users call moveToAcc or moveToCPU on dereference fields such as strings, bytes, repeated fields, or sub-messages, RPCAcc updates the Schema Table so that future RPCs place those fields directly in the correct memory. The paper states that when a compute unit becomes unavailable or later becomes available, automatic field updating turns a one-off placement miss into the correct steady state in subsequent requests, cutting several microseconds per RPC compared with manual placement (Zhang et al., 2024).

The general placement principle distilled from this work is direct: place data where it will be consumed, batch small writes within a single RPC, avoid PCIe pointer-chasing by materializing contiguous host layouts, and use NUMA pinning so that CPU staging and PCIe devices share the relevant root-complex locality (Zhang et al., 2024).

6. Empirical effects, operational constraints, and limitations

The shared-GPU serving results quantify the effect of PCIe-aware placement within a larger control loop. The evaluation uses a single host and a 2-node cluster consisting of two AWS p4d.24xlarge instances with $\rho \to 1$0 A100-80GB per node, EFA/IB 200 Gbps, Slurm 23.02, MOFED 5.9, and DCGM 3.2. Dockerized tenants are used, CPU pinning is enabled for the sensitive tenant, and dataloaders are isolated to a separate NUMA domain when possible. Against static MIG with naive placement, the placement-only ablation reduces SLO miss-rate from approximately $\rho \to 1$1 to approximately $\rho \to 1$2 and p99 latency from approximately $\rho \to 1$3 ms to approximately $\rho \to 1$4 ms, with throughput approximately $\rho \to 1$5. The full system reaches approximately $\rho \to 1$6 miss-rate, approximately $\rho \to 1$7 ms p99 latency, and approximately $\rho \to 1$8 throughput, corresponding to an approximately $\rho \to 1$9 reduction in miss-rate and approximately $$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$0 p99 improvement at no more than $$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$1 throughput cost. In the LLM-serving evaluation with vLLM on OLMo 2 7B Instruct, TTFT p99 improves from $$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$2 ms to $$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$3 ms with no more than $$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$4–$$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$5 throughput cost. The paper also reports that MIG reconfiguration and PCIe-aware placement contribute comparably and that their benefits are additive (Darzi et al., 27 Aug 2025).

CAP reports end-to-end throughput improvements of $$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$6–$$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$7 over DeepSeek EPLB and vLLM sequential placement across single-node and multi-node settings. Its largest gains appear on Node A, which consists of $$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$8 RTX 3090 without NVLink and without GPU-Direct P2P, where all GPU-to-GPU communication goes through CPU memory and the node is communication-constrained. On Node B, with $$M_i = \Pr(L_i > \tau) \approx \frac{\text{violations}}{\text{requests}},$$9 A100 and GPU-Direct P2P over PCIe but no NVLink, mid-spectrum Placement 3 is reported as optimal, improving latency by $w_{g,g'}$00 over the unadjusted co-activation placement, by $w_{g,g'}$01 over EPLB, and by $w_{g,g'}$02 over default sequential placement. In multi-node experiments, increasing inter-node pruning cost $w_{g,g'}$03 from $w_{g,g'}$04 to $w_{g,g'}$05 reduces nodes-per-token by $w_{g,g'}$06; the latency benefit is negligible on 8$w_{g,g'}$07400 Gbps InfiniBand but reaches $w_{g,g'}$08 on 200 Gbps, indicating that topology-aware pruning matters most when bandwidth is tight (Shi et al., 6 Jul 2026).

RPCAcc reports $w_{g,g'}$09 lower serialization time than a comparable RPC accelerator baseline on HyperProtoBench in the abstract, and up to $w_{g,g'}$10 throughput improvement in the end-to-end cloud workload. The detailed results state that one-shot DMA yields $w_{g,g'}$11 higher deserialization throughput overall, the memory-affinity serializer is $w_{g,g'}$12 faster than ProtoACC-PCIe and $w_{g,g'}$13 faster than CPU-only, and the image-compression workload achieves $w_{g,g'}$14 higher throughput than ProtoACC-PCIe and $w_{g,g'}$15 higher throughput than the CPU-only baseline, together with $w_{g,g'}$16 lower average latency and $w_{g,g'}$17 lower 99th-percentile latency relative to a comparable RPC accelerator baseline (Zhang et al., 2024).

Across the cited work, several operational limitations recur. In the shared-GPU controller, topology discovery can be partially opaque in some clouds, so the policy may rely more on counters than on a full PCIe map; MIG changes may pause tenants and reload model state, and larger-scale behavior or multi-tenant fairness across nodes remains future work. CAP assumes that co-activation statistics are sufficiently stationary over the serving horizon and that gating behavior does not drift rapidly; heavy distribution shifts may therefore require periodic re-profiling. RPCAcc notes that benefits decline for very large, flat messages dominated by data transfer rather than latency, that PCIe bandwidth saturation remains a bottleneck even when transaction count is reduced, and that non-coherent shared fields cannot avoid cross-PCIe fetching (Darzi et al., 27 Aug 2025, Shi et al., 6 Jul 2026, Zhang et al., 2024).

Taken together, these results establish a consistent systems conclusion. PCIe-aware placement is not a single algorithm but a topology-sensitive control strategy: combine live counters with discovered hierarchy, prefer locality that avoids hot roots or expensive paths, coalesce transfers when movement is unavoidable, and use conservative gating or profiling to prevent oscillation. This suggests that PCIe-aware placement is most valuable precisely where compute isolation or load balancing alone is insufficient: multi-tenant LLM serving with shared DMA paths, expert-parallel MoE inference on PCIe-bound nodes, and RPC acceleration on commodity PCIe-attached devices.