- The paper demonstrates that real-time hardware telemetry integration improves routing decisions in geographically distributed GPU clusters.
- Experimental results show up to 1.75× higher throughput and significant tail latency reduction, validating Solyx’s dynamic control plane.
- Robust fallbacks and predictive metrics enable resilient failover and enhanced performance under diverse, heterogeneous workload conditions.
Solyx AI Grid: Hardware-Telemetry-Aware Routing for Geo-Distributed GPU Clusters
Motivation and Problem Statement
The fragmentation of GPU infrastructure across geographically distributed sites is increasingly prevalent due to hardware procurement constraints, power limitations, and diverse latency requirements. This architectural shift in LLM inference deployments introduces routing challenges unaddressed by traditional intra-cluster (e.g., NVIDIA Dynamo, vLLM Router) and cross-region load balancers (e.g., SkyWalker (Xia et al., 30 May 2025)). These approaches either assume homogeneous datacenter conditions, ignore hardware and network telemetry, or rely on static placement optimizers unsuited to dynamic per-request routing. As a result, multi-site deployments face characteristic failure modes: capability mismatch leakage, tail latency amplification from WAN jitter, and slow failover recovery.
System Architecture
Solyx AI Grid implements a hardware-telemetry-aware cross-site routing control plane, positioned above inference engines and compatible with unmodified Envoy data planes. The Solyx scoring architecture continuously aggregates ten signals per replica: four vLLM application metrics (queue depth, P95 TTFT, error rate, KV cache fill), four DCGM hardware telemetry signals (GPU utilization, VRAM utilization, pipeline occupancy, memory bandwidth), and two network signals (RTT, jitter). These are normalized and combined into a composite pressure score that guides integer weight projections for Envoy's routing configuration, triggering updates at sub-second cadence.
The routing control loop is driven by a per-replica lifecycle state machine (Loading, Ready, Draining, Failed, Terminated) informed by heartbeat TTL expiry from cellagents, enabling transactional failover within a single 500 ms interval. Telemetry source degradation triggers graceful fallback: when DCGM exporter fails, routing continues with vLLM metrics alone, ensuring robustness and zero traffic loss.
Experimental Methodology
Two campaigns were executed on provisioned US-based datacenters:
- Campaign 1 (April 2026) evaluated six NVIDIA H100/H200 SXM GPUs across three sites, varying fleet configuration (homogeneous, hardware-heterogeneous, capability-mismatch).
- Campaign 2 (May 2026) deployed nine NVIDIA RTX PRO 6000 Blackwell SE GPUs across three sites, serving Llama 3.1 70B AWQ INT4 under eight production-representative workload classes and a 216-cell SLO matrix. Routing modes compared included Round-Robin (RR), Least-Request (LR), and Solyx AI Grid (v2/v3 scorer).
Workload characterization incorporated burst/adversarial traffic, long-context and code-heavy prompts, and edge cases such as capability mismatch and telemetry failure. Routing efficacy was measured using sustained RPS at SLO, latency percentiles, leak rate under capability mismatch, failover recovery time, and reaction latency during bursts.
Empirical Findings
Throughput and Latency
Across all workload classes at tier-2 SLO, Solyx AI Grid achieved 1.56–1.75× higher sustainable throughput compared to Round-Robin, outperforming LR consistently. Latency gains were especially pronounced in P99 tail; homogeneous H100 clusters observed a 27.9% P99 tail-latency reduction.
Routing Adaptation
Solyx concentrated traffic 134:1 onto healthy, unconstrained endpoints in heterogeneous fleets exclusively via real-time telemetry. Under capability mismatch (one constrained cell among eight), Solyx reduced leak rate to 0.43% (vs. 32.11% RR, 28.71% LR), yielding 99.57% long-prompt success rate (vs. 67.89% RR, 71.29% LR). This demonstrates that load-aware policies fail to detect and avoid endpoints with systematic rejection unless an explicit error-rate signal is consumed.
Hardware Telemetry Lead Time
A key insight is that DCGM hardware telemetry predicts application-layer degradation with a median lead time of 11.2 s (e.g., under thermal or PCIe contention stress), enabling preemptive traffic drain before TTFT breaches SLO thresholds. Application-only monitoring stacks, by comparison, are inevitably reactive.
Failover and Robustness
Destructive failover testing (SIGKILL events) revealed Solyx reroutes in-flight traffic at p99 latency of 1,247 ms (vs. 4,226 ms RR, 2,104 ms LR) via heartbeat TTL expiration. Post-failover success rates were 99.76% (Solyx), 98.81% (LR), and 94.12% (RR).
During telemetry source failure (DCGM exporter killed), Solyx's fallback engaged in <1 s, maintaining a 99.90% success rate.
Network Awareness
In induced WAN jitter scenarios, network-aware routing with RTT and jitter signals yielded a 27.8% additional P95 TTFT reduction over hardware-only signal strategies, with Solyx-10-signal P95 latencies averaging 42% lower than RR in worst-case network conditions.
Discussion and Implications
The results establish that hardware telemetry and network signal integration are both necessary and sufficient for robust, adaptive routing across geo-distributed, heterogeneous GPU clusters. Conventional policies (Least-Request, Round-Robin) consistently fail under capability mismatch and degrade under network volatility. Solyx's architecture not only delivers empirical performance gains but also anticipates failures, enabling a proactive operational stance.
Boundary Condition: The only scenario where Solyx converges with LR is homogeneous, saturated, failure-free, and network-stable fleets—a fleeting, idealized configuration rarely realized in production.
Stack Integration: Solyx AI Grid is architecturally complementary to existing intra-cluster routers. The reference control-plane design (NVIDIA AI Grid [aigrid]) is empirically validated for live distributed infrastructure in this work.
Predictive Operation: The DCGM hardware telemetry lead time finding has broader operational implications; integrating low-level hardware signals enables predictive maintenance and preemptive routing action.
Generalizability: While the campaigns used limited replica counts, the failure modes and performance differentials are structural and expected to generalize to larger, more heterogeneous fleets. However, scaling control-plane aggregation and tuning hybrid objectives for cache locality remain open research directions.
Limitations: Synthetic workloads, single-model serving, and limited cost/energy evaluation delimit the scope. Control-plane scalability and correlated regional failure behaviors were not modeled.
Future Directions
- Hierarchical Control Plane Scaling: Deploying per-region sub-controllers and aggregators for fleet sizes in hundreds or thousands.
- Hybrid Routing Objectives: Combining pressure minimization with KV-cache locality for enhanced efficiency at large scale.
- Multi-Model Routing & Cost Optimization: Extending routing logic to heterogeneous model replicas, evaluating energy and cost metrics under real user traffic.
Conclusion
This empirical study demonstrates that integrating GPU hardware telemetry, application metrics, and network signals into dynamic routing decisions materially enhances throughput, reduces latency, and prevents systematic failures inherent to multi-site LLM inference deployments. Hardware-telemetry-aware routing provides predictive capability for failure mitigation and adapts efficiently to the operational realities of geographically distributed GPU infrastructure. The experimental results define a quantitative benchmark for future routing architectures and validate hardware telemetry as a fruitful signal source for control-plane design in distributed AI systems.
(2606.15050)