VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?

Abstract: For years, we have built LLM serving systems like any other critical infrastructure: a single general-purpose stack, hand-tuned over many engineer-years, meant to support every model and workload. In this paper, we take the opposite bet: a multi-agent loop that automatically synthesizes bespoke serving systems for different usage scenarios. We propose VibeServe, the first agentic loop that generates entire LLM serving stacks end-to-end. VibeServe uses an outer loop to plan and track the search over system designs, and an inner loop to implement candidates, check correctness, and measure performance on the target benchmark. In the standard deployment setting, where existing stacks are highly optimized, VibeServe remains competitive with vLLM, showing that generation-time specialization need not come at the cost of performance. More interestingly, in non-standard scenarios, VibeServe outperforms existing systems by exploiting opportunities that generic systems miss in six scenarios involving non-standard model architectures, workload knowledge, and hardware-specific optimizations. Together, these results suggest a different point in the design space for infrastructure software: generation-time specialization rather than runtime generality. Code is available at https://github.com/uw-syfi/vibe-serve.

• The paper introduces VibeServe, a novel multi-agent framework that automatically synthesizes customized LLM serving stacks.
• The approach employs an outer planning loop and inner implementation loop to optimize system performance across diverse scenarios.
• Experimental results show parity with state-of-the-art systems and significant gains in non-standard deployments.

VibeServe: Agentic Synthesis of Bespoke LLM Serving Systems

Overview

The paper "VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?" (2605.06068) introduces VibeServe, an agentic framework that automatically synthesizes end-to-end LLM serving systems, tailored for specific model–hardware–workload targets. Contrasting with prevailing approaches that rely on highly engineered, general-purpose serving stacks, VibeServe pursues generation-time specialization, leveraging coding agents organized in a multi-role, multi-level loop to construct novel, optimized serving stacks automatically from minimal user inputs.

Motivation and Problem Setting

Current LLM serving frameworks such as vLLM, SGLang, and TensorRT-LLM are designed to maximize efficiency for the mainstream case—standard decoder-only Transformer models on NVIDIA GPUs, handling typical chat workloads. However, as new model architectures (hybrid, multimodal), hardware (Apple Silicon, data-center GPUs with varying memory architectures), and custom workloads (e.g., code editing with predicted outputs, large-scale prompt sharing, ASR streaming) proliferate, maintaining optimality with a single, generic stack is increasingly unsustainable. Genericity imposes a "portability tax"—manifesting as performance loss or the inability to support novel deployment axes without significant engineering investments.

The feasibility of per-deployment specialization has been traditionally constrained by engineering cost. Recent advances in coding agent capabilities—which have shown sustained progress on isolated kernel and algorithm optimization—suggest that generation-time specialization, previously impractical at scale, is becoming tractable even for complex, multi-component systems.

Figure 1: VibeServe's motivation: while general-purpose frameworks target common cases, VibeServe specializes to each model–hardware–workload combination.

VibeServe Architecture

VibeServe’s architecture comprises an outer planning loop and an inner implementation loop. The outer loop manages structured, persistent state (issues, long-term memory, git checkpoints) and dispatches optimization tasks to the inner loop. The inner loop is enacted by three specialized agent roles:

• Implementer: Proposes and revises code changes grounded in a serving-systems skills library distilled from mature engine code and research literature.
• Accuracy Judge: Validates correctness, ensuring output matches a reference implementation and that reward-hacking or degenerate solutions are blocked.
• Performance Evaluator: Profiles system performance, generates targeted hints for future iterations, and guides exploration via platform-aware profiling.

Together, these agents collaborate within a controlled workspace; only generated serving-system code may be modified, never the reference implementation or benchmarks.

External extensibility is attained via a skills library, formatted in the Agent Skills standard, defining model architectures, serving algorithms, backend libraries, hardware-specific details, and optimization strategies contextualized with compatibility metadata.

Figure 2: VibeServe architecture: user artifacts define the deployment; outer/inner multi-agent loops plan, implement, judge, and evaluate, leveraging a serving-systems skills library.

Experimental Results

VibeServe is evaluated on six scenarios spanning diverse workload patterns, architectural features, and hardware targets. Results substantiate three claims: (i) parity with state-of-the-art human-engineered stacks in standard conditions, (ii) concrete performance wins in non-standard deployments, and (iii) successful synthesis for previously unsupported settings. Each scenario was evaluated against strong baselines with correctness assured through reference-based checks.

Scenario A: Standard LLM Serving

On Llama-3.1-8B-Instruct with H100, after 60 agentic iterations, VibeServe achieves parity with vLLM on throughput and time-per-output-token (TPOT), and beats SGLang by 5% (throughput) and 3% (TTFT). The agent adaptively escalates workload stress as plateau is detected, evidencing effective outer-loop planning.

Figure 3: On Llama-3.1-8B-Instruct (H100), VibeServe matches vLLM and outperforms SGLang, adapting request rates as optimization proceeds.

Scenario B: Code Editing with Predicted Outputs

For speculative decoding on code-edit workloads (Qwen3-32B on CodeEditorBench), VibeServe deploys a predicted-output verifier, reaching $5.95\times$ speedup over a standard baseline and doubling vLLM’s draft-model speculative decoder throughput. This optimization outstrips what generalized stacks can achieve since it leverages workload specifics at code-generation time.

Figure 4: Code-editing scenario: VibeServe exploits user predictions for aggressive speculative decoding, substantially exceeding generic baselines.

Scenario C: Hybrid Model Prompt Caching

On Olmo-Hybrid-7B (NVIDIA L4, 32k-shared prefix RAG workload), VibeServe coordinates dual cache management for attention and SSM-style DeltaNet layers, attaining $3.45\times$ the throughput of vLLM, which lacks first-class hybrid cache management and cannot efficiently share state.

Scenario D: Streaming ASR

Moonshine Streaming with sliding-window encoder attention benefits from bespoke encoder-layer cache management, cutting time-to-first-token latency to $1.69\times$ lower than vLLM plugin integration.

Scenario E: Local Constrained Decoding

For Llama-3.1-8B-Instruct (MLX, MacBook, JSONSchemaBench), VibeServe composes XGrammar-based constraint enforcement, lightweight draft-model speculative decoding, and MLX-specific prefill tuning for a $2.6\times$ p50 latency reduction, integrating and optimizing stack components beyond the reach of existing generalized solutions.

Figure 5: Constrained decoding on a MacBook: VibeServe layering XGrammar, speculative decoding, and prefill chunking for substantial latency reduction.

Scenario F: Unified Multimodal Inference

For Show-o2 (text-to-image, both H100 and MacBook), VibeServe synthesizes pipelines combining AR and diffusion modules, improving latency by $21.4\%$ on H100 and $6.27\times$ on MacBook, approaching backend-specific performance ceilings.

Theoretical and Practical Implications

VibeServe’s results validate that multi-agent agentic loops, when equipped with system-specific context (via skills libraries) and structured planning, can traverse the substantial design space required to build end-to-end, custom serving stacks for LLMs. Importantly:

• Bespoke serving unlocks nontrivial optimizations: Knowledge of model, hardware, and workload at generation-time reveals system-level optimizations (e.g., dual cache management, workload-aware speculative decoding, streaming cache alignment) infeasible for general-purpose stacks.
• Structured multi-agent orchestration outperforms flat agentic or population-based scripts: The separation of planning, implementation, correctness, and profiling—persisted over repository state—avoids drift, compaction failures, and the myopic optimization trajectories found in many prior agentic systems.
• Compute budget and per-target correctness remain limitations: The agentic loop currently consumes significant compute and is limited by the quality of user-provided checkers.

Bold claim: "Generation-time specialization will beat runtime generality in more domains where generic abstractions cost performance," repositioning agentic system synthesis as a viable, eventually preferable alternative to single-stack engineering.

Future Prospects

As coding agents continue to scale, the value of generation-time system specialization is expected to further increase, particularly in domains with fast-evolving model semantics, hardware architectures, and workload types. Immediate research questions include curriculum-based bootstrapping (leveraging prior specialized systems), branching outer-loop exploration, and improving verification mechanisms for correctness and performance.

Conclusion

VibeServe demonstrates that long-horizon, multi-agent coding agents can build bespoke, performant LLM serving stacks in a fully automated manner and that such systems can match or exceed state-of-the-art hand-engineered serving solutions, especially in emerging, non-standard settings. This work represents a significant step in practical automated systems design, laying a pathway for AI-driven infrastructure specialization across the broader systems landscape.