---
title: 'DS-Serve: Software-Defined Agentic Serving'
url: https://www.emergentmind.com/topics/ds-serve
type: topic
---

# DS-Serve: Software-Defined Agentic Serving

Searching arXiv for the target paper and closely related serving systems to ground the article.
DS-Serve is the natural shorthand for **Software-Defined Agentic Serving**, an **SDN-inspired serving architecture for multi-agent LLM pipelines** in which communication behavior and serving decisions are made **programmable** and **adaptive to runtime state** rather than statically encoded in application logic. In the source paper, the exact shorthand “DS-Serve” is not explicitly used, but the system idea is introduced under the title “Software-Defined Agentic Serving” and is framed as a response to the mismatch between dynamic agentic workloads and static serving stacks [2601.03197].

## 1. Definition, scope, and naming

DS-Serve targets **multi-agent LLM pipelines** that combine multiple LLM agents, tools, services, heterogeneous runtimes, and changing communication patterns. The motivating examples include software-development workflows with a **developer agent** and a **tester agent**, where the same logical interaction can be served through **batching**, **function-by-function pipelining**, or **token-level streaming**. The central claim is that no single communication strategy is uniformly optimal, because the best choice depends simultaneously on **application goals**—such as responsiveness, throughput, interactivity, and SLOs—and on **current system state**, including load, queue lengths, model latency, token rates, and contention [2601.03197].

The paper identifies three deficiencies in prevailing agentic serving practice. First, existing protocols such as **A2A**, **MCP**, and **ACP** impose **early binding**: design-time decisions about streaming versus batching, synchronous versus asynchronous communication, and routing policy become embedded in application code. Second, current frameworks lack **global visibility**, so communication logic cannot react coherently to pipeline-wide telemetry. Third, they offer only limited **programmability and end-to-end control**, functioning primarily as message-exchange APIs rather than as programmable serving substrates with scheduling, routing, priority handling, speculative execution, or state-movement support. The paper reports that a poor communication choice can degrade throughput by up to **3.6×** [2601.03197].

A terminological ambiguity is worth noting. The name **DS-Serve** has also appeared in an unrelated retrieval context, namely “DS SERVE: A Framework for Efficient and Scalable Neural Retrieval,” which concerns single-node billion-scale neural retrieval rather than agentic serving [2602.22224]. In the present usage, DS-Serve denotes the **Software-Defined Agentic Serving** architecture for multi-agent LLM systems.

## 2. Three-plane architecture

The architecture is organized into three planes—**control plane**, **metrics plane**, and **configurable data plane**—explicitly modeled on **Software-Defined Networking (SDN)**, but adapted from network forwarding to agentic serving. The design separates policy, observability, and communication mediation so that runtime state can influence both message transport and agent/tool behavior [2601.03197].

| Plane | Role | Illustrative functions |
|---|---|---|
| Control plane | Logically centralized controller | Communication mode, routing, admission, scheduling, model selection |
| Metrics plane | Telemetry collection and aggregation | CPU/GPU utilization, memory pressure, queue lengths, TTFT, TPT |
| Configurable data plane | Runtime communication shim | Streaming, batching, priorities, pacing |

The **data plane** is the runtime communication substrate between agents and tools. Rather than replacing existing protocols, it is implemented as a **shim layer** between high-level protocols such as A2A, MCP, and ACP and the lower-level socket or transport mechanisms. This shim can be reconfigured at runtime and mediates communication attributes such as **granularity**, **pacing**, and **priority**. The design therefore preserves compatibility with established agent protocols while introducing a programmable serving layer beneath them [2601.03197].

The **control plane** is a logically centralized controller that gathers telemetry from the metrics plane, maintains a logical view of system state, interprets operator goals and policies, pushes rules into the data plane, and can also invoke **agent/tool control APIs** directly. This is more expansive than classical SDN, because the controller does not merely influence routing of messages; it also influences **agent runtime behavior** itself. The paper distinguishes two rule classes. **Agent-level rules** define default behavior for an agent pipeline, such as choosing batch versus stream or admitting only high-priority requests under load. **Request-level rules** apply finer control, such as routing an individual request to a different instance, blocking speculative work until resources free up, or triggering state transfer [2601.03197].

The **metrics plane** supplies the observability substrate for closed-loop control. It collects both **system-level telemetry**—including CPU/GPU utilization, memory pressure, and queue lengths—and **application-level telemetry**, including per-request latency, token timings, dependencies between agents, and tool-usage behavior. Its implementation is described as a **two-tier design**: local collectors at each node write telemetry into lightweight shared-memory structures, and the controller retrieves metrics through **centralized polling** on demand rather than continuous metric streaming [2601.03197].

## 3. Programmability and control abstractions

In DS-Serve, an agentic serving system is **programmable** if operators can specify policies and goals that govern runtime serving behavior rather than hard-coding those choices into application code. It is **system-aware** if those decisions are driven by actual runtime state, including queue buildup, CPU/GPU utilization, latency, per-token behavior, contention across agents, and pipeline dependencies. The conceptual shift is from a static, developer-chosen communication strategy to a dynamic, controller-driven one [2601.03197].

The paper does not present a complete formal policy language, but it does outline the intended programming model. Operators should be able to express high-level intents such as **“minimize end-to-end latency,” “batch unless load is low,” “stream only for interactive or high-priority tasks,” “maximize throughput under latency bounds,”** or **“ensure 90% of interactive requests meet an SLO.”** The controller would then compile these intents into concrete policies and runtime actions. This suggests an intent-driven control layer above heterogeneous agent runtimes, though the paper stops short of specifying a full declarative language [2601.03197].

At the low level, DS-Serve proposes a minimal unified control interface with two functions:

```text
set(parameter name, value)
reset(parameter name)
```

This interface is intended to hide heterogeneous internals behind a uniform control surface. The paper’s example is setting vLLM batch size through `set('max_num_seqs', 4)`. Each agent or tool is expected to provide a shim that translates these generic calls into its internal runtime APIs [2601.03197].

The explicitly mentioned control knobs span a broad serving surface: **communication granularity** such as token streaming, pipelining, and batching; **routing** to specific agent instances; **admission policy**; **scheduling and prioritization**; **model selection**; **batch size**; **communication frequency**; **pacing**; **resource allocation** such as GPU slots; **state transfer** such as KV-cache movement; **speculative execution**; and hooks for **throttling, pausing, and reprioritizing**. The framework is therefore intended to manipulate both transport-level behavior and deeper serving internals through one controller-mediated interface [2601.03197].

## 4. Runtime adaptation and execution model

DS-Serve is fundamentally a **closed-loop control** system. The paper describes a feedback process in which local collectors gather metrics, the controller polls them, populates a **logical state store**, compares observed state with target intent, installs rules or invokes agent/tool APIs, and thereby changes serving behavior so that the system evolves toward a target state. The authors explicitly avoid presenting a formal control-theoretic law; the policy engine remains architectural and conceptual rather than fully algorithmized [2601.03197].

The runtime signals named in the paper include **queue lengths**, **load**, **model latency**, **token generation rates**, **GPU utilization**, **CPU utilization**, **memory pressure**, **time per token (TPT)**, **time to first token (TTFT)**, **per-request latency**, **agent dependencies**, **conditional execution**, and, for tools, **file access latency**, **throughput**, and **disk utilization**. These signals are used to choose among communication and serving strategies. Under **high load** or **queue buildup**, the controller may prefer batching or message-level sends to amortize cost. Under **low load** or more interactive conditions, it may prefer token streaming. If an agent instance is overloaded, it may reroute requests to another instance; if state is already resident on the original instance, the controller may trigger **KV cache transfer** to avoid recomputation [2601.03197].

The execution model is explicitly multi-agent rather than single-endpoint. Application-level telemetry includes **agent dependencies** and conditional execution, so the controller is intended to reason over the full pipeline rather than isolated model calls. The paper discusses support for **branching or conditional execution**, **parallelism and pipelining**, **adaptive routing**, **speculative execution**, **tool use**, **streaming**, and **batching**. It also mentions regeneration-related adaptation, such as increasing priority or rerouting when a verifier triggers regeneration, although loops are not formalized as part of the present execution model [2601.03197].

A distinctive mechanism is **state-aware rerouting**. If a request must move from one LLM instance to another and the original instance holds the relevant KV cache, DS-Serve envisions controller-mediated **state transfer** so that the destination can resume without full recomputation. More generally, the controller can demote background agents to synchronous mode, reallocate GPU slots, or transfer session context when interactive requests risk SLO violations. This places state mobility alongside communication granularity as a first-class control variable [2601.03197].

## 5. Prototype system and empirical results

The implementation described in the paper is a **strawman prototype** rather than a production-grade deployment. It is built for a workflow using **Google A2A**, modified to invoke a **custom communication library** built on **gRPC**. That communication library supports **synchronous**, **asynchronous**, and **streaming** modes. The serving engine integrates with **vLLM**, which was modified to enable **KV cache transfer** between instances. The demonstrated workload is a **MetaGPT-style** software-engineering setting with one **developer agent** and one or more **testing agents** [2601.03197].

The evaluation centers on two demonstrations. The first is **dynamic communication control**, in which communication granularity is adapted based on load and different mechanisms—batching, pipelining, and token-level streaming—are compared. The second extends the control surface through **deeper serving integration**, introducing two testing-agent instances and load balancing across them, including variants with and without hints and KV-cache movement. The measurements focus primarily on **serving throughput/request rate**, together with qualitative responsiveness implications and load-balancing efficacy; hardware details and isolated control-plane overheads are not reported [2601.03197].

The main empirical finding is that **no single communication configuration consistently dominates**. Under changing load, batching and streaming can each become suboptimal. The paper reports that a suboptimal communication strategy can degrade throughput by up to **3.6×**, and that dynamic control of communication granularity can improve throughput by up to the same factor relative to mismatched static choices. In the communication-control experiment, the controller is described as quickly converging on the most effective mechanism [2601.03197].

A second layer of gains appears when the controller can manipulate deeper serving internals. In the experiment with multiple tester-agent instances, rerouting and state-related controls yield **2.3×** better throughput than a baseline with **no load balancing**. Within that setting, **hint-based** integration—where the controller preemptively signals that a request will arrive so that state movement or setup can occur ahead of time—achieves **1.8×** better performance than a comparison that does not use hints. The paper presents these results as evidence that communication control alone is insufficient; substantial gains come from exposing deeper runtime controls such as state transfer and anticipatory preparation [2601.03197].

## 6. Position in the literature, limitations, and open problems

DS-Serve is positioned against three adjacent lines of work. Relative to conventional **LLM serving systems**, which usually optimize batching, scheduling, KV-cache management, and GPU utilization for a single model service, it elevates those concerns to the **multi-agent pipeline level**, where communication style itself becomes a control variable. Relative to **workflow orchestration**, it seeks tighter integration with token-level communication modes, LLM-specific metrics such as TTFT and TPT, and real-time serving adaptation. Relative to agent communication protocols such as **A2A**, **MCP**, and **ACP**, it does not replace protocol standardization but inserts a programmable shim under or around those protocols to add observability and control [2601.03197].

This framing also distinguishes DS-Serve from nearby systems papers. **P/D-Serve** addresses disaggregated LLM serving at very large NPU-cluster scale through fine-grained prefill/decoding organization, on-demand forwarding, and optimized KVCache transfer [2408.08147]. **ShadowServe** targets the narrower bottleneck of interference-free remote KV-cache fetching for distributed prefix caching through a host control plane and SmartNIC-offloaded data plane [2509.16857]. **DynaServe** studies unified and elastic execution for dynamic disaggregated LLM serving by splitting requests into cooperating sub-requests across unified GPU instances [2504.09285]. DS-Serve differs in treating **agent-to-agent communication and serving policy** as the primary programmable object: its novelty lies in the combination of a logically centralized controller, a metrics plane, a configurable communication shim, and a unified agent/tool control API for multi-agent pipelines [2601.03197].

The paper is explicit that the present system is a **starting point** rather than a completed ecosystem. Several limitations are named directly. Heterogeneity remains difficult because agents and tools expose different APIs and runtime capabilities, so even with a standardized `set/reset` interface each still requires a translation shim. Metrics collection at scale is challenging across backends such as **vLLM**, **Triton**, and custom runtimes, especially for long-lived pipelines and high-volume, low-latency control loops. The work does not provide a complete policy language, and the two-function control API is intentionally minimal; richer interfaces such as interrupting planning loops, cloning context, or suppressing subcalls are left open. Hybrid deployments with external or cloud-hosted agents may expose only partial telemetry and partial control, limiting achievable benefits [2601.03197].

The future directions identified by the authors follow directly from those limitations: **languages for agentic control**, richer **control-agent interfaces**, scalable **metrics infrastructure**, support for **external and hybrid agents**, **online policy adaptation**, capability classes and standard APIs for heterogeneous agents and tools, and mechanisms for **conflict resolution** and **policy correctness**. Taken together, these indicate that DS-Serve is best understood not as a fixed serving kernel, but as an architectural program for making agentic serving **programmable, system-aware, and closed-loop** at the scale of entire multi-agent pipelines [2601.03197].

Source: https://www.emergentmind.com/topics/ds-serve