Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets

Abstract: Modern Intel AI PCs ship capable integrated GPUs and NPUs with 16+ GB of unified memory, and they spend considerable time idle. That is not enough memory to fit a large model such as a 70B-parameter LLM. We show that a handful of AIPCs, working together over an ordinary network, can serve models beyond the capability of any single one. We use pipeline parallelism: a model is split by layer into per-stage shards, each pre-compiled into an OpenVINO graph, so that every machine runs one shard and passes activations to the next. Three techniques make this fast enough to be useful. First, we recover the speed of the unsplit model: a naive per-stage export runs well below monolithic inference because it misses an OpenVINO GPU optimization, and injecting a beam_idx Gather into each shard triggers that optimization (the IndirectKVCache fusion) and brings the shards to parity. Second, we leverage speculative decoding on stateful OpenVINO models. Third, the pipeline serves several users at once by interleaving their requests across the stages, each request carrying its own cache (micro-batching). Together, a two-node Llama 3.1 8B INT4 pipeline serves two concurrent users at 1.79x the single-user throughput of the unsplit model on the same hardware, and the gap widens under simulated wide-area latency. The same design scales to a 70B model that no single fleet member can hold: a four-node deployment of Lunar Lake AI PCs on Intel Tiber Cloud serves a single user at interactive speed, with output token-for-token identical to the same four-node pipeline decoding without speculation. Code, raw benchmark logs, and reproduction scripts ship as a self-contained package at https://github.com/labscommunity/pipeline-sharded-inference-paper (in the top-level reproduction/ directory).

• The paper introduces pre-compiled INT4 OpenVINO pipeline shards with stateful KV caches, achieving 43.97 tok/s for two concurrent Llama 3.1 8B users—1.79× the monolithic baseline—and 6.43 tok/s for four-node Llama 3.1 70B inference.
• The paper restores OpenVINO GPU fusion by injecting a compile-time beam_idx Gather and enables bit-exact mask-based KV rewind, producing up to 1.50× speculative-decoding gains with under 1% CPU overhead.
• The paper shows that top-1 logits compression reduces relay-mediated WAN traffic by 8.17×, while micro-batching and adaptive draft lengths preserve interactive performance when network latency increases.

Overview and thesis

This paper demonstrates that a small fleet of consumer Intel AI PCs—each equipped with an integrated GPU, 16–32 GB of unified memory, and otherwise idle—can serve LLM inference at interactive speeds, including models that no single fleet member can hold. The system partitions a model by layer into per-stage shards, each pre-compiled to an INT4 OpenVINO IR graph with stateful KV cache, connected by a minimal TCP activation relay. Three techniques compose into the headline result: a two-node Llama 3.1 8B INT4 pipeline serving two concurrent users at 43.97 tok/s aggregate, 1.79× the monolithic single-user baseline on the same hardware. The design extends to a four-node Llama 3.1 70B INT4 deployment over a relay-mediated WAN at up to 6.43 tok/s aggregate—bit-exact against the same topology's target-only path.

The paper's central claim is that the barrier to fleet inference is software, not hardware: standard export tooling fails on modern attention, monolithic compiled graphs cannot be split without surgery, and speculative decoding implementations assume paged-attention APIs absent from stateful OpenVINO models. Each gap is closed with a specific, measured technique.

Reaching monolithic parity via the beam_idx Gather injection

A naive per-stage export of Llama 3.1 8B INT4 runs 13–23% slower than openvino_genai.LLMPipeline, and splitting into more stages widens the gap. The cause is that OpenVINO's GPU plugin gates its IndirectKVCache transformation—which fuses ReadValue→Concat→Assign patterns into an IndirectSDPA op—on each KV ReadValue flowing through a Gather indexed by a beam_idx Parameter. Exports lacking this pattern silently fall back to a slower generic fusion path.

The fix is post-export graph surgery: insert a constant-indexed Gather(ReadValue, beam_idx, axis=0) after stateful transformation, mirroring what optimum-intel does internally for monolithic exports. This recovers 15% of throughput (21.26 → 24.45 tok/s), bringing the exported shard within 0.4% of the C++ monolithic loop in paired sessions. Notably, no beam search is ever run; the injection is purely a compile-time pattern that triggers the fusion, with beam_idx=[0] making the Gather an identity at runtime. In-process multi-stage splits still cost ~11–15% relative to monolithic (VTune shows XVE occupancy dropping from 60.1% to 43.6% on a 3-stage split), a structural consequence of smaller per-stage GEMMs—the paper recovers this cost through micro-batching rather than eliminating it.

Mask-based KV rewind for speculative decoding

Speculative decoding on a stateful OpenVINO target requires removing rejected draft tokens' K/V entries from the cache. The natural API—query_state()/set_state() physical trim—costs ~48 ms per call on Arc B390 with a 72-token cache, enough to push the technique below break-even. The paper's alternative leaves rejected entries physically in place and masks their positions out via the existing attention_mask input, tracking logical sequence length through caller-supplied position ids so RoPE remains coherent.

This is verified bit-exact: post-correction logits show maximum absolute difference 0.0000 against physical trim, at under 1% CPU overhead per step. Single-node results across eight prompt types yield a mean 1.33× speedup (range 1.11× on creative writing at 49.7% acceptance to 1.50× on code completion at 93.1%), rising to ~1.58× at 2048-token generations as acceptance climbs to 97.5%. Cache bloat from retained rejected drafts converges to 1.02× logical length, so no in-generation compaction is needed for realistic single-turn lengths—though the authors concede multi-turn conversations approaching the context window would require compaction they have not implemented or measured. All results are greedy-decode only; extension to sampling would need the rejection-sampling correction of Leviathan et al., which composes in principle but is unvalidated empirically.

Micro-batching and distributed composition

Each stream receives its own compile_model() call and independent InferRequest KV state, letting the coordinator interleave decode steps across stages. On the v5_beam shards this yields 1.80× two-stream scaling (16.33 → 29.34 tok/s), below the 2× balanced-stage ceiling because the faster fused baseline leaves less idle time to fill. Stacked with mask-based speculative decoding (a further ~1.50× multiplier), the composition reaches 43.97 tok/s—measured, not projected, matching the product of individually measured multipliers within 0.05 tok/s. A three-stage testbed scales to 64.67 tok/s at three streams (2.64× mono), hitting the min(users, stages) pipeline-fill bound exactly, at ~6 GB iGPU memory per stream on Lunar Lake's 16 GB budget.

Per-token breakdown attributes only ~10% of the 61 ms token to TCP round-trip (~6 ms); the rest is stage compute plus Python/OpenVINO dispatch overhead (user Python itself is 0.6% of wall time). An important methodological correction is reported: initial instrumentation claiming "70% network time" conflated TCP latency with remote compute wait.

WAN behavior and top-1 logits compression

Under simulated hop latency, the full stack degrades gracefully while naive pipeline decode falls below the 5 tok/s interactive floor past ~25 ms/hop; the full-stack-over-naive multiplier grows from 2.83× at LAN to 4.04× at 100 ms/hop. The paper reports three WAN methods and argues they should not converge: sleep-injection isolates latency, a release-time-queue TCP proxy captures cwnd dynamics, and a real Tiber Cloud deployment over Tailscale's DERP relay adds per-segment relay queueing. The decisive finding is that real-WAN cost scales per segment, not per RTT: the 501 KB FP32 logits return fragments across segments, so sending only argmax + max probability (~8 bytes) preserves greedy correctness bit-exactly and yields an 8.17× speedup on the DERP-relayed path (2.80 → 22.88 tok/s aggregate) versus 1.12× on LAN. Without compression, both streams sit at 1.40 tok/s—unusable; with it, interactive throughput is recovered. Activation compression, by contrast, fails: INT8 hidden-state quantization corrupts generation outright, and FP16 gains nothing because LAN bandwidth is not the bottleneck.

Optimal draft length $K$ shifts with regime: $K{=}7$ wins on LAN where compute dominates, $K{=}10$ at ≥50 ms/hop where packing tokens per round trip dominates. The same direction holds on the 70B deployment.

Scaling to 70B

Llama 3.1 70B INT4 (~36 GB across four ~9 GB shards) runs on four Tiber Cloud instances over DERP-relayed WAN. Target-only decode is 1.74 tok/s; speculative decoding at $K{=}10$ reaches 5.42 tok/s single-stream (3.1×, bit-exact), and 6.43 tok/s aggregate with two streams on a Panther Lake coordinator. Long-context behavior is counterintuitive: throughput rises to 5.72 tok/s at 1024 tokens (acceptance 72.2%) before falling at 4096 under KV pressure. Two caveats are stated plainly: an Arrow Lake-S instance's iGPU died mid-inference even with finer 12-layer splits, so Lunar Lake-class iGPUs are required end-to-end; and no external monolithic 70B reference exists because FP16 calibration weights (~141 GB) OOM-killed the export machine—a chunked NNCF flow or ≥256 GB RAM node is left as future work.

Additional subsystems

The paper covers several supporting mechanisms. A capture-and-resume prefix cache serializes per-stage KV state at generation end (compacting masked positions), enabling warm resume of multi-turn conversations; the multi-stage protocol lacks byte-exact certification, and TTFT savings are unmeasured. NPU stages require a separate static-shape, stateless export path (the NPU compiler rejects dynamic shapes and stateful ops); validated for correctness but never selected by the placement solver since single-stream NPU decode costs ~4× the iGPU per token. Packed multi-slot serving recovers NPU concurrency by repurposing the sequence axis with a host-written [1,1,S,T] block-diagonal mask—achieving 6.01× graph-level slot scaling (16 slots at 1.76 ms/token vs 10.57 ms single-slot), exact per-slot isolation ($\max|\Delta|=0$), and Sarathi-Serve-style split-fuse scheduling from mask layout alone—with limits: fixed uniform window partition, slot count baked into the IR, greedy-only decoding, and quality verification reaching 14B-class only single-stage. Gemma 4 E2B exercises cross-layer KV sharing across stage boundaries and required a rotary-embedding rewrite for OpenVINO 2026.1 strict type validation; its Per-Layer Embeddings resist all sub-FP32 quantization.

Negative results are documented explicitly: early exit at layer 15 fails (lm_head calibrated for layer-31 representations), asymmetric shard planning yields noise-level gains, INT8 KV cache hints run 4% slower on Arc, async draft/target overlap wastes work except in the ~0.2%-probability all-accepted case, and a C++ port of the spec loop would move nothing given Python's 0.6% wall-time share.

Limitations and open questions

The system assumes a trusted, reliable network with no fault tolerance, authentication, or encryption; all nodes must run identical software stacks. No power or thermal measurements are reported despite acknowledged throttling risk on laptop-class hardware during sustained inference. WAN sweeps use injected latency lacking real jitter and slow-start effects, though the queue-proxy and Tiber measurements partially compensate. The 70B correctness argument rests on self-consistency (bit-exactness against the same topology's target-only path) rather than an external reference. In-generation KV compaction, temperature-sampling speculation, QUIC transport benchmarking, packed-mode scaling beyond 1.5B-class graph-level measurement, and TTFT characterization for the 70B deployment all remain open.

Conclusion

The paper establishes that pre-compiled, fusion-unlocked INT4 shards, mask-based speculative rewind, and independent-state micro-batching compose multiplicatively into a fleet inference stack that exceeds single-user monolithic throughput on identical hardware, survives relay-mediated WAN conditions, and extends to 70B-class models no fleet member can host alone—all with bit-exact output guarantees against their respective baselines. Its most transferable findings are the beam_idx pattern gate on OpenVINO's IndirectKVCache fusion, the equivalence of attention-mask rewinding to physical cache trimming, and the per-segment (not per-RTT) cost model for real-WAN logits transport.