DS-Serve: Software-Defined Agentic Serving

• DS-Serve is a programmable serving architecture for multi-agent LLM pipelines that adapts communication strategies based on runtime state.
• It features a three-plane design—control, metrics, and configurable data—modeled on SDN to separate policy, telemetry, and data mediation.
• Dynamic adaptation in DS-Serve can improve throughput by up to 3.6× and enhance load balancing with deeper control over communication modes.

Searching arXiv for the target paper and closely related serving systems to ground the article. DS-Serve is the natural shorthand for Software-Defined Agentic Serving, an SDN-inspired serving architecture for multi-agent LLM pipelines in which communication behavior and serving decisions are made programmable and adaptive to runtime state rather than statically encoded in application logic. In the source paper, the exact shorthand “DS-Serve” is not explicitly used, but the system idea is introduced under the title “Software-Defined Agentic Serving” and is framed as a response to the mismatch between dynamic agentic workloads and static serving stacks (Agarwal et al., 6 Jan 2026).

1. Definition, scope, and naming

DS-Serve targets multi-agent LLM pipelines that combine multiple LLM agents, tools, services, heterogeneous runtimes, and changing communication patterns. The motivating examples include software-development workflows with a developer agent and a tester agent, where the same logical interaction can be served through batching, function-by-function pipelining, or token-level streaming. The central claim is that no single communication strategy is uniformly optimal, because the best choice depends simultaneously on application goals—such as responsiveness, throughput, interactivity, and SLOs—and on current system state, including load, queue lengths, model latency, token rates, and contention (Agarwal et al., 6 Jan 2026).

The paper identifies three deficiencies in prevailing agentic serving practice. First, existing protocols such as A2A, MCP, and ACP impose early binding: design-time decisions about streaming versus batching, synchronous versus asynchronous communication, and routing policy become embedded in application code. Second, current frameworks lack global visibility, so communication logic cannot react coherently to pipeline-wide telemetry. Third, they offer only limited programmability and end-to-end control, functioning primarily as message-exchange APIs rather than as programmable serving substrates with scheduling, routing, priority handling, speculative execution, or state-movement support. The paper reports that a poor communication choice can degrade throughput by up to 3.6× (Agarwal et al., 6 Jan 2026).

A terminological ambiguity is worth noting. The name DS-Serve has also appeared in an unrelated retrieval context, namely “DS SERVE: A Framework for Efficient and Scalable Neural Retrieval,” which concerns single-node billion-scale neural retrieval rather than agentic serving (Liu et al., 17 Dec 2025). In the present usage, DS-Serve denotes the Software-Defined Agentic Serving architecture for multi-agent LLM systems.

2. Three-plane architecture

The architecture is organized into three planes—control plane, metrics plane, and configurable data plane—explicitly modeled on Software-Defined Networking (SDN), but adapted from network forwarding to agentic serving. The design separates policy, observability, and communication mediation so that runtime state can influence both message transport and agent/tool behavior (Agarwal et al., 6 Jan 2026).

Plane	Role	Illustrative functions
Control plane	Logically centralized controller	Communication mode, routing, admission, scheduling, model selection
Metrics plane	Telemetry collection and aggregation	CPU/GPU utilization, memory pressure, queue lengths, TTFT, TPT
Configurable data plane	Runtime communication shim	Streaming, batching, priorities, pacing

The data plane is the runtime communication substrate between agents and tools. Rather than replacing existing protocols, it is implemented as a shim layer between high-level protocols such as A2A, MCP, and ACP and the lower-level socket or transport mechanisms. This shim can be reconfigured at runtime and mediates communication attributes such as granularity, pacing, and priority. The design therefore preserves compatibility with established agent protocols while introducing a programmable serving layer beneath them (Agarwal et al., 6 Jan 2026).

The control plane is a logically centralized controller that gathers telemetry from the metrics plane, maintains a logical view of system state, interprets operator goals and policies, pushes rules into the data plane, and can also invoke agent/tool control APIs directly. This is more expansive than classical SDN, because the controller does not merely influence routing of messages; it also influences agent runtime behavior itself. The paper distinguishes two rule classes. Agent-level rules define default behavior for an agent pipeline, such as choosing batch versus stream or admitting only high-priority requests under load. Request-level rules apply finer control, such as routing an individual request to a different instance, blocking speculative work until resources free up, or triggering state transfer (Agarwal et al., 6 Jan 2026).

The metrics plane supplies the observability substrate for closed-loop control. It collects both system-level telemetry—including CPU/GPU utilization, memory pressure, and queue lengths—and application-level telemetry, including per-request latency, token timings, dependencies between agents, and tool-usage behavior. Its implementation is described as a two-tier design: local collectors at each node write telemetry into lightweight shared-memory structures, and the controller retrieves metrics through centralized polling on demand rather than continuous metric streaming (Agarwal et al., 6 Jan 2026).

3. Programmability and control abstractions

In DS-Serve, an agentic serving system is programmable if operators can specify policies and goals that govern runtime serving behavior rather than hard-coding those choices into application code. It is system-aware if those decisions are driven by actual runtime state, including queue buildup, CPU/GPU utilization, latency, per-token behavior, contention across agents, and pipeline dependencies. The conceptual shift is from a static, developer-chosen communication strategy to a dynamic, controller-driven one (Agarwal et al., 6 Jan 2026).

The paper does not present a complete formal policy language, but it does outline the intended programming model. Operators should be able to express high-level intents such as “minimize end-to-end latency,” “batch unless load is low,” “stream only for interactive or high-priority tasks,” “maximize throughput under latency bounds,” or “ensure 90% of interactive requests meet an SLO.” The controller would then compile these intents into concrete policies and runtime actions. This suggests an intent-driven control layer above heterogeneous agent runtimes, though the paper stops short of specifying a full declarative language (Agarwal et al., 6 Jan 2026).

At the low level, DS-Serve proposes a minimal unified control interface with two functions:

set(parameter name, value) reset(parameter name)

This interface is intended to hide heterogeneous internals behind a uniform control surface. The paper’s example is setting vLLM batch size through set('max_num_seqs', 4). Each agent or tool is expected to provide a shim that translates these generic calls into its internal runtime APIs (Agarwal et al., 6 Jan 2026).

The explicitly mentioned control knobs span a broad serving surface: communication granularity such as token streaming, pipelining, and batching; routing to specific agent instances; admission policy; scheduling and prioritization; model selection; batch size; communication frequency; pacing; resource allocation such as GPU slots; state transfer such as KV-cache movement; speculative execution; and hooks for throttling, pausing, and reprioritizing. The framework is therefore intended to manipulate both transport-level behavior and deeper serving internals through one controller-mediated interface (Agarwal et al., 6 Jan 2026).

4. Runtime adaptation and execution model

DS-Serve is fundamentally a closed-loop control system. The paper describes a feedback process in which local collectors gather metrics, the controller polls them, populates a logical state store, compares observed state with target intent, installs rules or invokes agent/tool APIs, and thereby changes serving behavior so that the system evolves toward a target state. The authors explicitly avoid presenting a formal control-theoretic law; the policy engine remains architectural and conceptual rather than fully algorithmized (Agarwal et al., 6 Jan 2026).

The runtime signals named in the paper include queue lengths, load, model latency, token generation rates, GPU utilization, CPU utilization, memory pressure, time per token (TPT), time to first token (TTFT), per-request latency, agent dependencies, conditional execution, and, for tools, file access latency, throughput, and disk utilization. These signals are used to choose among communication and serving strategies. Under high load or queue buildup, the controller may prefer batching or message-level sends to amortize cost. Under low load or more interactive conditions, it may prefer token streaming. If an agent instance is overloaded, it may reroute requests to another instance; if state is already resident on the original instance, the controller may trigger KV cache transfer to avoid recomputation (Agarwal et al., 6 Jan 2026).

The execution model is explicitly multi-agent rather than single-endpoint. Application-level telemetry includes agent dependencies and conditional execution, so the controller is intended to reason over the full pipeline rather than isolated model calls. The paper discusses support for branching or conditional execution, parallelism and pipelining, adaptive routing, speculative execution, tool use, streaming, and batching. It also mentions regeneration-related adaptation, such as increasing priority or rerouting when a verifier triggers regeneration, although loops are not formalized as part of the present execution model (Agarwal et al., 6 Jan 2026).

A distinctive mechanism is state-aware rerouting. If a request must move from one LLM instance to another and the original instance holds the relevant KV cache, DS-Serve envisions controller-mediated state transfer so that the destination can resume without full recomputation. More generally, the controller can demote background agents to synchronous mode, reallocate GPU slots, or transfer session context when interactive requests risk SLO violations. This places state mobility alongside communication granularity as a first-class control variable (Agarwal et al., 6 Jan 2026).

5. Prototype system and empirical results

The implementation described in the paper is a strawman prototype rather than a production-grade deployment. It is built for a workflow using Google A2A, modified to invoke a custom communication library built on gRPC. That communication library supports synchronous, asynchronous, and streaming modes. The serving engine integrates with vLLM, which was modified to enable KV cache transfer between instances. The demonstrated workload is a MetaGPT-style software-engineering setting with one developer agent and one or more testing agents (Agarwal et al., 6 Jan 2026).

The evaluation centers on two demonstrations. The first is dynamic communication control, in which communication granularity is adapted based on load and different mechanisms—batching, pipelining, and token-level streaming—are compared. The second extends the control surface through deeper serving integration, introducing two testing-agent instances and load balancing across them, including variants with and without hints and KV-cache movement. The measurements focus primarily on serving throughput/request rate, together with qualitative responsiveness implications and load-balancing efficacy; hardware details and isolated control-plane overheads are not reported (Agarwal et al., 6 Jan 2026).

The main empirical finding is that no single communication configuration consistently dominates. Under changing load, batching and streaming can each become suboptimal. The paper reports that a suboptimal communication strategy can degrade throughput by up to 3.6×, and that dynamic control of communication granularity can improve throughput by up to the same factor relative to mismatched static choices. In the communication-control experiment, the controller is described as quickly converging on the most effective mechanism (Agarwal et al., 6 Jan 2026).

A second layer of gains appears when the controller can manipulate deeper serving internals. In the experiment with multiple tester-agent instances, rerouting and state-related controls yield 2.3× better throughput than a baseline with no load balancing. Within that setting, hint-based integration—where the controller preemptively signals that a request will arrive so that state movement or setup can occur ahead of time—achieves 1.8× better performance than a comparison that does not use hints. The paper presents these results as evidence that communication control alone is insufficient; substantial gains come from exposing deeper runtime controls such as state transfer and anticipatory preparation (Agarwal et al., 6 Jan 2026).

6. Position in the literature, limitations, and open problems

DS-Serve is positioned against three adjacent lines of work. Relative to conventional LLM serving systems, which usually optimize batching, scheduling, KV-cache management, and GPU utilization for a single model service, it elevates those concerns to the multi-agent pipeline level, where communication style itself becomes a control variable. Relative to workflow orchestration, it seeks tighter integration with token-level communication modes, LLM-specific metrics such as TTFT and TPT, and real-time serving adaptation. Relative to agent communication protocols such as A2A, MCP, and ACP, it does not replace protocol standardization but inserts a programmable shim under or around those protocols to add observability and control (Agarwal et al., 6 Jan 2026).

This framing also distinguishes DS-Serve from nearby systems papers. P/D-Serve addresses disaggregated LLM serving at very large NPU-cluster scale through fine-grained prefill/decoding organization, on-demand forwarding, and optimized KVCache transfer (Jin et al., 2024). ShadowServe targets the narrower bottleneck of interference-free remote KV-cache fetching for distributed prefix caching through a host control plane and SmartNIC-offloaded data plane (Xiang et al., 21 Sep 2025). DynaServe studies unified and elastic execution for dynamic disaggregated LLM serving by splitting requests into cooperating sub-requests across unified GPU instances (Ruan et al., 12 Apr 2025). DS-Serve differs in treating agent-to-agent communication and serving policy as the primary programmable object: its novelty lies in the combination of a logically centralized controller, a metrics plane, a configurable communication shim, and a unified agent/tool control API for multi-agent pipelines (Agarwal et al., 6 Jan 2026).

The paper is explicit that the present system is a starting point rather than a completed ecosystem. Several limitations are named directly. Heterogeneity remains difficult because agents and tools expose different APIs and runtime capabilities, so even with a standardized set/reset interface each still requires a translation shim. Metrics collection at scale is challenging across backends such as vLLM, Triton, and custom runtimes, especially for long-lived pipelines and high-volume, low-latency control loops. The work does not provide a complete policy language, and the two-function control API is intentionally minimal; richer interfaces such as interrupting planning loops, cloning context, or suppressing subcalls are left open. Hybrid deployments with external or cloud-hosted agents may expose only partial telemetry and partial control, limiting achievable benefits (Agarwal et al., 6 Jan 2026).

The future directions identified by the authors follow directly from those limitations: languages for agentic control, richer control-agent interfaces, scalable metrics infrastructure, support for external and hybrid agents, online policy adaptation, capability classes and standard APIs for heterogeneous agents and tools, and mechanisms for conflict resolution and policy correctness. Taken together, these indicate that DS-Serve is best understood not as a fixed serving kernel, but as an architectural program for making agentic serving programmable, system-aware, and closed-loop at the scale of entire multi-agent pipelines (Agarwal et al., 6 Jan 2026).