QUIC Relay Benchmarking for Distributed Inference

Benchmark a QUIC-based relay against the persistent TCP activation relay for distributed OpenVINO pipeline inference, measuring its effects on LAN latency, multi-segment logits transfers, WAN congestion behavior, and fallback requirements when UDP is unavailable.

Background

The distributed system currently uses persistent TCP connections, and the measured per-hop cost is dominated primarily by wireless scheduling and operating-system socket wakeup rather than by payload bandwidth for ordinary activation messages.

The paper identifies QUIC or another UDP-based transport as a potentially useful alternative, particularly for large multi-segment logits payloads whose TCP congestion-window behavior is costly. However, the expected gains are uncertain because some of the LAN latency floor would remain, and the deployment must retain a TCP fallback when UDP is blocked.

References

We have not yet benchmarked a QUIC relay; it is the most promising piece of future work in this layer.

— Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets  (2608.19147 - Berenbaum et al., 19 Aug 2026) in Section 5.3, “Activation Compression,” subsection “Reducing the per-hop cost”