SwarmX: Agentic Scheduling for Low-Latency Agentic Systems

Abstract: Agentic AI applications compose multiple model calls and tool executions, creating new scheduling challenges for GPU-CPU clusters. Their inference time and model-call structure often depend on prompt semantics, making conventional scheduling approaches ineffective for low-latency serving. This paper presents SwarmX, a system that implements agentic scheduling for low-latency agentic applications. SwarmX uses scheduling-specific neural predictors to capture prompt, device, runtime, and target-model features; exposes distributional predictions to routers and scalers for tail-aware decisions; and provides mechanisms for predictor training and online adaptation. These predictors and mechanisms are integrated into a scheduler-agent framework that provides a common substrate for integration with existing scheduling and model-serving infrastructure. We evaluate SwarmX using production deployment (nearly one thousand GPUs and one million CPU cores) and controlled experiments on a 128-GPU testbed. Across multi-agent code generation, deep research, and multimodal agentic workflows, SwarmX reduces tail latency by up to 61.5% compared to state-of-the-art schedulers and sustains up to 2x the throughput of production schedulers under the same SLO.

• The paper introduces SwarmX, which employs neural predictors and distribution-aware scheduling to robustly reduce tail latency in agentic AI clusters.
• The methodology combines prompt semantic extraction with an MLP that produces quantile sketches, enabling low-overhead and tail-risk aware scheduling decisions.
• Empirical results demonstrate significant reductions in latency and improvements in throughput across both testbed and production environments compared to traditional systems.

SwarmX: Agentic Scheduling for Low-Latency Agentic Systems

Motivation and Background

Agentic AI applications, characterized by prompt-driven dynamism—variable inference times and unpredictable model-call structures—pose pronounced challenges for cluster scheduling. Traditional AI infrastructure, such as Ray and Kubernetes, rely on stateless heuristics (e.g., round-robin, random, power-of-two choices) that fail to adapt to the variability introduced by complex agentic workflows. Existing predictor-based systems (PSC, Cilantro) leverage simplistic regression models, incapable of capturing prompt-induced distributional latency shifts. Recent agentic schedulers, such as ORION, Parrot, Pie, and Murakkab, either focus on single-agent or ignore semantic prompt effects, precluding robust tail latency guarantees.

Figure 1: Overview of a typical agentic application cluster illustrating how multi-agent workflows induce complex, dynamic interactions across model services.

The primary challenge arises from:

• Prompt-dependent variable inference time: Latency varies with prompt semantics, often spanning orders of magnitude, producing heavy abnormal tails.
• Prompt-dependent model-call structure: Workflows form dynamic DAGs, where downstream dependencies are resolved at runtime.

Figure 2: Inference time distributions across models and workloads, exposing strong semantic prompt dependency and cross-model variation.

Figure 3: The number and structure of model calls per workflow are highly prompt-dependent and heterogeneous across models and workloads.

SwarmX Architecture

SwarmX introduces agentic scheduling via scheduler-agent abstraction. Routers and scalers are promoted to lightweight agents, each equipped with neural predictors that integrate prompt, device, runtime, and target-model features. Decisions are made through distribution-aware composition, preserving tail risk visibility and uncertainty throughout the scheduling pipeline.

Figure 4: SwarmX design overview, highlighting the interplay between neural predictors, distribution-aware scheduling, and integration with cluster infrastructure.

SwarmX's neural predictors are decoupled into two components:

Figure 5: Neural predictor architecture, comprising a parameter-reduced isomorphic semantic model and an MLP for feature combination and distribution prediction.

• Semantic model: Extracts prompt-level semantic vectors using small isomorphic LLMs.
• MLP: Fuses semantic vectors with device, runtime, and model characteristics, outputting quantile sketches of latency and model-call structure.

This approach yields robust prompt-awareness without incurring the prohibitive overhead of full-scale model inference.

Distribution-Aware Scheduling

A central principle is preserving uncertainty—distributional predictions—until the final scheduling action, avoiding the information collapse inherent to point estimates. Scheduling state and predictions are maintained as quantile sketches, enabling incremental composition and sophisticated tail-aware cost evaluation.

Figure 6: Distribution-based scheduling preserves tail awareness, while point-estimate composition obscures anomalous behavior crucial to tail latency minimization.

Router and scaler agents compose predicted distributional states (e.g., per-queue or per-demand quantile sketches) and select actions via distributional cost functions (e.g., P99 latency, throughput loss) sampled from these sketches.

Figure 7: Scheduler-agent framework that encapsulates predictors, coordinator logic, memory for adaptation, and actionable cluster interfaces.

Predictor Training and Online Adaptation

SwarmX employs specialized objectives for semantic and MLP components, emphasizing weighted pinball loss for quantile prediction, constructed from operational trace logs. Adaptation is triggered locally on out-of-distribution (OOD) detection—when observed tail deviations exceed thresholds—limiting retraining to lightweight MLPs for swift recovery without interrupting the scheduling path. Only when the target-model undergoes structural change is the semantic model retrained.

Integration and Implementation

SwarmX is implemented as a Ray/Kubernetes-compatible scheduler plug-in. Predictors are deployed per worker server, scaling naturally with cluster size and ensuring low latency access to prompt/runtime state. Prediction overhead remains negligible (milliseconds per request) relative to served model inference (seconds–minutes). Predictor weights are managed with eventual consistency and robust checkpointing. High prediction traffic is mitigated by delegating semantic feature extraction upstream.

Empirical Results

Comprehensive evaluation in both testbed (128-GPU) and production (nearly 1k GPUs, 1M+ CPU cores) environments demonstrates:

Router Microbenchmarks:

• P95 latency reduction of 18% (Text-to-Video) and 28.3% (Deep Research) over Ray; Murakkab fails to generalize across heavy-tailed diffusion workloads.

  Figure 8: Router-only microbenchmark (P50, P95 latency) highlighting SwarmX's advantage over Ray and Murakkab in Text-to-Video and Deep Research workloads.

Scaler Microbenchmarks:

• Scaler-driven latency reduction by 7.8–13.7% (Text-to-Video) and ~50% (Deep Research), outperforming static provisioning.

  Figure 9: Scaler-only microbenchmark (P50, P95 latency) reflecting gains from distribution-based scaling decisions.

End-to-End Performance:

• Coordinated router+scaler deployment reduces P95 latency by up to 61.5% (Deep Research).
• In agentic coding (OpenClaw, Coding Agent), SwarmX achieves 11–32% latency reduction over Murakkab and Ray baselines.

  Figure 10: End-to-end latency for structured pipelines, showing synergy between router and scaler agents.

  

  Figure 11: End-to-end latency in OpenClaw (dual and single-model setups), confirming SwarmX's generalization to open-ended agentic workloads.

  

  Figure 12: End-to-end latency in Coding Agent (dual and single-model setups), reinforcing tail-risk mitigation.

Production Deployment:

• On massive CPU clusters, Video OCR sees a 59.6% P50 and 48.38% P99 latency decrease.
• Entity Semantic Analysis delivers up to 2× sustainable throughput and 50% P99 latency reduction, learning hardware priority (H20/L20) via data-driven device features.

  Figure 13: SwarmX's priority-aware routing adapts load between H20 and L20 pools, outperforming production scheduling.

Robustness and Overhead:

SwarmX maintains prediction cost (35M-parameter predictor: ~30ms CPU/4ms GPU) orders of magnitude lower than target inference (e.g., Qwen3-8B at 0.7–100+ s). Under severe resource drift (71% capacity loss), OOD adaptation restores latency within 100s; stale predictors produce >40s P90 latency, retrained agents recover below 20s.

Figure 14: P90 latency through capacity loss; OOD adaptation recovers tail latency rapidly.

Implications and Future Directions

SwarmX reveals that agentic scheduling, grounded in neural-predictor and distribution-aware uncertainty management, is necessary for high-performance AI serving in cluster environments where prompt-induced dynamism predominates. The agentic framework enables compatibility and modular extensibility with existing infrastructure, permitting seamless adoption and subsequent evolution. SwarmX's methodology suggests future agent-driven systems will recast cluster scheduling as a learning problem, leveraging semantic signal, operational telemetry, and compositional distribution management to guarantee tail latency while maximizing resource utilization.

Potential future directions include:

• Self-improving scheduler agents: Online learning at finer granularity, with semantic feature extraction directly from target models.
• Cross-workload coordination: Federated scheduler-agents for multi-application environments with shared resource pools.
• Integration with cluster-level LLM inference optimizations: Combining SwarmX routing/scaling with replica-internal scheduling (e.g., DistServe, vLLM) for layered optimization.
• Automatic hardware adaptation: Directly learning device/model interaction for heterogeneous GPU/CPU clusters at scale.

Conclusion

SwarmX establishes agentic scheduling as a principled foundation for serving agentic AI at scale. It systematically addresses prompt-, device-, runtime-, and target-model awareness, distributional uncertainty, predictor adaptation, and framework compatibility. SwarmX achieves substantial reductions in tail latency and throughput improvements across diverse agentic workloads, outperforming state-of-the-art and production schedulers. The results advocate neural-driven, distribution-aware, modular scheduling as the emergent paradigm for intelligent cluster resource management.