Scepsy: Agentic Serving for LLM Workflows

• Scepsy is an agentic serving system designed for orchestrating complex, data-dependent multi-LLM workflows on GPU clusters.
• It employs profiling, aggregate pipeline modeling, and hierarchical allocation search to optimize resource allocation and balance latency-throughput trade-offs.
• The system achieves significant performance gains, including up to 27× lower 99th-percentile latency and 2.4× higher throughput versus traditional methods.

Scepsy is an agentic serving system designed for the efficient orchestration of complex, multi-LLM agentic workflows on GPU clusters. These workflows integrate multiple LLMs and tools, frequently exhibiting data-dependent execution paths such as dynamic branching, fan-outs, and recurrences. Scepsy addresses the central challenges of unpredictable workload dynamics, conflicting latency-throughput trade-offs, and severe GPU oversubscription by modeling and optimizing aggregate LLM resource usage, achieving cross-model, workload-aware scheduling with high empirical efficiency (Wagenländer et al., 16 Apr 2026).

1. Motivation and Challenges in Agentic Workflow Serving

Agentic workflows comprise pipelines where multiple LLMs and external tools are orchestrated via dynamic, data-dependent constructs—loops, branches, fan-outs, and fan-ins. Their execution is characterized by the following factors:

• Unpredictable branching and dynamic parallelism: The number, sequence, and concurrency of LLM invocations can vary dramatically per request. For example, in beam search, the number of beams may lead to hundreds of LLM calls with large variance in end-to-end latency.
• Throughput versus latency trade-off: Increasing data-parallel replicas of a model improves throughput but reduces available GPU slices for tensor parallelism, increasing latency for individual requests. These trade-offs propagate and interact across the multi-stage workflow, shifting bottlenecks non-trivially.
• GPU oversubscription and fragmentation: The number of LLMs in a workflow often exceeds the available GPU pool. Naïvely assigning whole GPUs per model results in pronounced underutilization, particularly when models have heterogeneous computational requirements.

Traditional serving and batch-oriented systems are not equipped to efficiently schedule such workloads at scale, due to their inability to model or react to these dynamic resource demands (Wagenländer et al., 16 Apr 2026).

2. System Architecture and Components

Scepsy decomposes the agentic workflow serving problem into four primary components:

1. Profiler: Replays traced LLM calls under a variety of arrival rates and tensor-parallel (TP) degrees, gathering per-model throughput and latency curves.
2. Aggregate LLM Pipeline Constructor: Processes workflow-level traces to extract per-model average invocation count ($n_m$) and intra-request parallelism ($p_m$), combining them with model profiles to construct a structured abstraction.
3. Allocation Search (GPU Scheduler): Explores allocations over fractional GPU shares ($g_i$), TP degrees ($t_i$), and replica counts ($r_i$) to minimize end-to-end latency subject to throughput targets and the available GPU budget.
4. Placement Heuristic: Hierarchically maps optimized allocations onto the physical cluster, minimizing fragmentation and observing network topology (NVLink) constraints.

A representative architecture workflow is as follows:

Component	Input	Output
Profiler	Traced LLM workflow requests	Throughput/latency curves per model
Aggregate LLM Pipeline	Per-model stats ($n_m$, $p_m$), profiles	Pipeline performance model
Allocation Search	Pipeline model, GPU inventory	Optimized allocation ($g⃗$, $t⃗$, $r⃗$)
Placement Heuristic	Allocation, cluster topology	Concrete deployment assignments, Kubernetes files

3. Aggregate LLM Pipeline Modeling

Scepsy's key insight is that while end-to-end latencies for workflows vary widely, the aggregate share of total execution time contributed by each LLM stage is statistically stable across inputs. Leveraging this, Scepsy constructs an "Aggregate LLM Pipeline" model, reducing dynamic workflows to a compact sequence of LLM stages for performance prediction.

The pipeline model operates as follows:

• Per Model Statistics: For each model $p_m$0, compute average invocations $p_m$1 and average parallelism $p_m$2 from workflow traces.
• Performance Curves: Profile each model’s average latency $p_m$3 and max throughput $p_m$4 under different arrival rates and TP degrees.
• Workflow Latency and Throughput: For workflow $p_m$5 and vector of settings $p_m$6:
  • Latency:

  $p_m$7

  where $p_m$8 is the profiled per-model latency for the given effective arrival rate and parallelism. - Throughput:

  $p_m$9

• GPU Share Abstraction: Each LLM’s TP degree and replica count define its fractional GPU allocation ($g_i$0), supporting joint optimization of resource partitioning and task parallelism.

This modeling approach allows accurate prediction of achievable latency and throughput for diverse allocation configurations (Wagenländer et al., 16 Apr 2026).

4. Allocation Search and Optimization

Given a user-specified throughput requirement $g_i$1, Scepsy optimizes GPU allocation by solving: $g_i$2 where:

• $g_i$3: fractional GPU share for model $g_i$4
• $g_i$5: tensor parallelism degree for model $g_i$6
• $g_i$7: replica count for model $g_i$8
• $g_i$9: total GPUs available

The search is performed hierarchically:

1. Fractional Share Enumeration: Enumerate possible $t_i$0, pruning infeasible or suboptimal splits by aggregate latency ratios ($t_i$1) and symmetry constraints (e.g., contiguous fractions).
2. Packing into Physical GPUs: Map fractional allocations to whole GPUs and sub-GPU slices, reconciling physical tiling requirements.
3. Configuration Filtering: For each candidate ($t_i$2), enumerate ($t_i$3) combinations that tile exactly onto available hardware, prune allocations exceeding NVLink bandwidth, evaluate pipeline model predictions, and select optimal configuration.

This hierarchical approach enables efficient exploration of millions of possible deployments, balancing throughput scaling (via data-parallel replicas) against latency reduction (via increased tensor parallelism per replica) (Wagenländer et al., 16 Apr 2026).

5. Placement and Cluster Deployment Strategy

The chosen allocation must be mapped onto the physical cluster amidst hardware constraints:

• TP groups and NVLink domains: Tensor-parallel replicas for a single model must reside in the same NVLink domain to avoid excessive PCIe latency.
• Sub-GPU Fraction Isolation: Co-located models using fractions of a GPU must be scheduled without resource contention.

Because these constraints make packing NP-hard, Scepsy implements a two-stage heuristic:

• Inter-node Packing: Place largest TP groups first onto nodes where the remaining NVLink domain matches the required footprint, balancing utilization.
• Intra-node Fractional Packing: Within each node, assign model fractions to the least-loaded GPUs.

Scepsy generates Kubernetes deployment files, patches the NVIDIA device plugin to provision fractional SM and memory allocations using MPS (Multi-Process Service), and leverages orchestrator health/liveness semantics for operational reliability (Wagenländer et al., 16 Apr 2026).

6. Empirical Validation and Comparative Performance

On a 16-GPU on-premises cluster (4×RTX A6000, dual-socket AMD EPYC, 100 Gbps IB, and NVLink domains of 2 GPUs), Scepsy was benchmarked against:

• Kubernetes HPA autoscaling for independent per-LLM vLLM services
• Aegaeon’s spatial/temporal multiplexing
• Ayo’s workflow-aware batching and pipelining

Workloads comprised RAG + reranker pipelines and beam-search-based multi-LLM pipelines with Llama-family models. Scepsy demonstrated:

• Up to 2.4× higher throughput at equal latency versus Kubernetes HPA
• Up to 27× lower 99th-percentile latency at moderate throughput versus Kubernetes HPA
• 6.8–7.3× throughput gain and 12.9–14.1× latency reduction versus Aegaeon
• 2–8× throughput improvement versus Ayo (Ayo outperformed Scepsy only at very low load)

Ablation studies revealed that both fractional co-location and tensor parallelism are necessary to realize Scepsy’s performance gains; disabling either reduced throughput by half and doubled latency. Allocation search time remained under 30 seconds for 16 GPUs, 10 fractions per GPU, and 5 LLMs, demonstrating feasibility for interactive deployment.

7. Significance and Broader Implications

Scepsy establishes a methodology for serving dynamic, multi-LLM agentic workflows at scale by (a) reducing dynamic task graphs to aggregate-stage pipelines via workflow profiling, and (b) performing cross-model, structured allocation search over fractional GPUs, tensor parallelism, and replicas. This approach obviates the need for custom integration with specific agentic frameworks and enables efficient, automatic resource partitioning even on clusters with heterogeneous workloads.

Compared to existing systems (multi-LLM multiplexers, user-specified resource partitioners, workflow-aware batched serving frameworks), Scepsy achieves higher throughput and lower latency without specialized code for individual workflows, supporting arbitrary agentic code that leverages large-scale LLM orchestration (Wagenländer et al., 16 Apr 2026). This suggests broader applicability for serving infrastructure in high-throughput, low-latency, multi-model AI services where workflow dynamics preclude static resource allocation strategies.