INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration

Abstract: Existing multi-agent LLM orchestration methods, ranging from brute-force ensembles to learned routers, select models and topologies based on task and model features. However, these methods do not consider the runtime state of the serving infrastructure. On shared GPU clusters under concurrent load, this infrastructure blindness causes systematic resource underutilization: preferred models accumulate deep request queues while equally capable alternatives sit idle. In multi-agent pipelines, where each query triggers multiple sequential model calls, these delays then compound across every downstream step. Closing this gap is challenging because the relevant infrastructure signals (queue depths, KV-cache pressure, latencies) are dynamic and noisy, and they must drive three different decisions: planning, per-step routing, and scheduling. We introduce INFRAMIND, a framework that makes the entire multi-agent stack infrastructure-aware. An infra-aware planner conditions topology and role selection on real-time system load and remaining budget, biasing toward simpler graphs under congestion and richer ones at low load. An infra-aware executor then observes per-model queue depths, cache utilization, and response latencies at each agent step to decide which model to call and how deeply to reason; a budget-aware scheduler further reorders each model's queue so that urgent requests are served first. Cast as a hierarchical constrained MDP and solved end-to-end via reinforcement learning, the system learns to balance quality against latency automatically. Across five benchmarks, INFRAMIND delivers up to +7.6 pp accuracy over the prior baseline at low load with up to 7x lower latency, and sustains up to 99.9% SLO compliance under high load where every baseline drops below 50%.

• The paper introduces a hierarchical constrained-MDP policy that jointly adapts agent planning, model and strategy selection, and deadline-aware scheduling to live queues, latency, KV-cache pressure, and query budgets.
• InfraMind achieved up to 99.9% SLO compliance under heavy load, improved low-load MATH accuracy by 7.6 percentage points, and ran up to 14 times faster than MasRouter on HumanEval.
• Ablations show that infrastructure-aware routing, earliest-deadline-first scheduling, and adaptive reasoning depth address distinct bottlenecks, while client-side latency and congestion signals extend the approach to API-only model pools.

Motivation: infrastructure blindness

InfraMind addresses a failure mode the authors term infrastructure blindness: existing multi-agent LLM orchestrators select models, topologies, and roles from static task features alone, without observing the runtime state of the serving infrastructure. On shared GPU clusters under concurrent load, this produces systematic resource misallocation. Profiling MasRouter on a shared pool of five models under Poisson load on MATH, the paper documents three recurring symptoms: preferred small models accumulate queues exceeding 130 requests while equally capable large models sit idle; congested models incur over 30 s end-to-end delays on queries an idle alternative could answer in under 10 s; and at low load, 67% of large-model GPU capacity goes unused, forfeiting quality that deeper reasoning could harvest. Because multi-agent pipelines trigger multiple sequential model calls per query, a bottleneck at any step delays every downstream agent.

The problem is difficult for three coupled reasons. Planning topology and roles from runtime state is hard because infrastructure state changes during execution; per-step model selection must condition on noisy, fast-changing signals (queue depths, KV-cache pressure, latencies); and prioritizing among concurrent agent steps requires propagating per-request urgency into serving-layer queues. The three decisions interact—a choice at any layer reshapes the state the others observe—so heuristics tuned layer-by-layer leave cross-layer gains unrealized.

Formulation as a hierarchical constrained MDP

The system is cast as a hierarchical Constrained Markov Decision Process over $N$ LLM services and three prompting strategies (Flash, Concise, DeepThink). At each agent step $k$, the executor observes query and role embeddings ($\mathbf{e}_q$, $\mathbf{e}_{r_k}$), normalized remaining budget $b_k$, and per-model vectors of queue depth, end-to-end latency, and KV cache utilization polled from vLLM's /metrics endpoint. The action space is the product of models and strategies. The objective maximizes expected solve reward subject to cumulative step latency not exceeding the query budget, converted to an unconstrained objective via a single Lagrange multiplier $\lambda$ whose dual update rises on overspend and falls on slack.

The policy decomposes across two timescales: a planner $\pi_{\text{plan}}$ commits once per query to topology, agent count, and roles from query semantics plus a low-dimensional system summary; an executor $\pi_{\text{exec}}$ selects the (model, strategy) pair at each step from the full live state.

Method

Infrastructure-aware planner. InfraMind adopts MasRouter's cascaded controller (task classifier → collaboration structure → agent count → roles) but conditions every head on the system summary via FiLM modulation of the query embedding. This lets all four heads inherit one coherent view of load and budget, biasing toward simpler chains under congestion or tight budgets and richer debate topologies under slack.

Infrastructure-aware executor. A dual-pathway policy merges a semantic pathway (query plus role embeddings) with a resource pathway (remaining budget plus live metrics), producing a softmax policy and value head over $N \times 3$ joint actions. Strategies modulate reasoning depth atop chain-of-thought prompting: Flash emits a direct answer, Concise takes 2–3 steps, DeepThink reasons thoroughly with verification. A representative learned behavior: when a large model's queue is saturated, the executor redirects to an idle smaller model and invests in DeepThink, trading queue time for reasoning depth to recover accuracy at a fraction of the latency.

Budget-aware scheduling. Each query carries a deadline (arrival time plus budget); every LLM call inherits it, and each model's queue is served Earliest-Deadline-First rather than FCFS. Routing handles cross-model load balancing while EDF handles within-model deadline ordering—orthogonal axes of waiting delay.

Joint training. The executor trains with PPO's clipped surrogate on per-step trajectories; the planner trains with baseline-normalized REINFORCE on episode return, augmented by task-classification and VAE losses inherited from the cascaded controller. Both rewards share the same quality-minus-latency-penalty structure, and a single dual-updated $\lambda$ enforces the budget constraint across both levels. Training episodes sweep budget tiers and Poisson arrival rates with inter-sweep queue draining so the executor sees the full congestion distribution. The full policy is only ~471K parameters.

Experimental results

Evaluation covers five benchmarks (MBPP, HumanEval, GSM-Hard, MATH, MMLU-Pro), a heterogeneous five-model pool spanning a 10× parameter range (DeepSeek-R1-Distill-Qwen-32B, Mistral-Small-24B, Qwen2.5-Coder-14B, Llama-3.1-8B, Llama-3.2-3B) served by vLLM on two B200 GPUs, and baselines MoA, GPTSwarm, and MasRouter under Poisson arrivals at 10, 50, and 100 req/min with a 300 s SLO.

Setting	Result
Low-load accuracy	Best on all 5 benchmarks; +7.6 pp on MATH (82.0 vs. MoA 74.4), +7.4 pp on GSM-Hard
Low-load latency	Up to 14× faster than MasRouter on HumanEval (5 s vs. 70 s), 6.3× on MBPP
High-load SLO compliance	Up to 99.9% (HumanEval) vs. <50% for MasRouter on four of five benchmarks; MoA/GPTSwarm below 12% on most benchmarks
High-load latency	Stays under 300 s while MoA/GPTSwarm exceed 1000 s on several benchmarks

Two claims deserve emphasis. First, the SLO result is stark: at 100 req/min, every baseline drops below 50% compliance on most benchmarks while InfraMind sustains up to 99.9%. Second, accuracy degrades gracefully rather than collapsing with latency—at high load InfraMind remains most accurate on four of five benchmarks, narrowly trailing MasRouter by 0.4 pp on MBPP, but that margin costs MasRouter 985 s mean latency and 26% SLO compliance.

Ablations isolate each mechanism's contribution. Removing infra-aware routing inflates step latency 2.3–3.6× and raises mean queue depth from 25.1 to 40.6; replacing EDF with FCFS doubles mean latency (68→134 s) and increases P90 latency 1.8× (207→366 s); forcing Flash everywhere drops MMLU-Pro accuracy 9.5 pp (59.5%→50.0%), showing adaptive depth is a genuine quality lever, not merely a latency knob. Each disabled mechanism degrades a distinct axis, indicating the components fix non-overlapping failure modes. Additionally, budget sensitivity is emergent: on MATH at low load, accuracy rises monotonically from 62.6% to 82.0% (+19.4 pp) as the time budget grows, with no hand-coded budget rules—the RL policy learns to shift toward larger models and DeepThink as slack increases.

Blackbox and hybrid pools

The principle extends to settings where serving internals are unobservable using two client-side proxies: an exponential moving average of observed end-to-end latency, and a synthetic congestion signal equal to recent request count divided by each provider's RPM limit. On GSM-Hard, in a hybrid pool (three whitebox + two OpenRouter API models) InfraMind prefers whitebox models at low load and overflows to APIs as queues build; in a pure-API pool with heterogeneous RPM limits it redistributes across providers. Baselines collapse under load in both settings while InfraMind sustains accuracy and SLO, supporting the claim that infrastructure awareness requires only a signal correlated with current responsiveness, not privileged server access. Hybrid pools additionally use dual Lagrange multipliers for time and monetary budgets, reducing exactly to the whitebox formulation when no cost constraint applies.

Limitations and open questions

The paper concedes two structural assumptions. First, the planner commits to a topology once at query arrival; mid-workflow revision of the collaboration structure as load or partial outputs evolve is left open. Second, the model pool is fixed; adaptation to elastic configurations (autoscaling, hot model swaps) is not addressed, though the authors note both extensions fit within the existing state and action spaces. It is also worth noting that the evaluation uses a single five-model pool on two GPUs and a fixed 300 s SLO; generalization to larger pools, heterogeneous hardware, or tighter SLOs is not established. The blackbox proxies are heuristic (EMA latency, RPM ratio), and their fidelity under bursty provider-side throttling is not characterized.

Conclusion

InfraMind demonstrates that routing decisions made without observing queues, cache pressure, or latencies leave substantial performance unclaimed on shared GPU infrastructure, and that threading infrastructure awareness through planning, per-step execution, and queue scheduling—as a jointly trained hierarchical constrained-MDP policy—yields large gains precisely where prior orchestrators fail: high-concurrency SLO compliance. The empirical evidence supports the central claim that these gains require no privileged server access, extending to API-only deployments via client-side signals.