Autellix: Program-Aware LLM Serving

• Autellix is a program-aware LLM serving system that treats entire workflows as atomic scheduling units to orchestrate interdependent calls.
• It employs novel scheduling algorithms (PLAS and ATLAS) that use execution context and dependency graphs to dynamically prioritize and route tasks.
• Real-world evaluations demonstrate 4–15× throughput improvements and reduced tail latency, making it ideal for interactive agents and high-throughput processing.

Autellix is a LLM serving system designed to efficiently execute general-purpose agentic programs—workflows in which numerous interdependent LLM calls, tool invocations, and human actions are dynamically orchestrated to solve complex tasks. Distinct from traditional serving engines, Autellix treats entire programs as first-class entities, enabling program-aware scheduling and drastic improvements in throughput and latency. By leveraging program-level context and dependencies, Autellix systematically intercepts, prioritizes, and routes LLM calls to mitigate both call-level and program-level head-of-line blocking.

1. Motivation and Conceptual Framework

The emergence of agentic LLM applications marks a fundamental shift from static, single-turn chatbots to interactive systems comprising intricate workflows. In such agentic programs, LLM calls are executed not in isolation but as steps within an intertwined logic that may span multiple sequential and parallel operations. Traditional serving systems such as vLLM schedule each LLM call independently, applying a first-come, first-served order. This approach fails to account for dependencies between program calls, resulting in excessive cumulative wait times and pronounced head-of-line blocking—where lengthy operations impede the scheduling of dependent or shorter tasks. Autellix reverses this paradigm: program entities, rather than individual calls, are the atomic units for scheduling decisions, enabling optimizations that reduce the end-to-end latency experienced by users and AI agents.

2. Architectural Design

The architecture of Autellix consists of a persistent session interface, a global process table, and an augmented backend scheduling and load-balancing layer. Upon program initialization, a session record is established containing metadata including cumulative execution time, waiting times, engine assignments, and thread-level statistics (particularly relevant for multi-threaded workflows). The scheduler, deeply embedded within the LLM engine pipeline, assigns priorities to incoming calls according to this session-level information. The load balancer exploits data locality—such as shared system prompts or KV-cache states—to decide whether to preserve engine affinity for subsequent calls or to distribute call load among minimally loaded engines. By minimizing redundant prefill computations and exploiting KV-cache co-location, Autellix manages GPU resources efficiently and adapts resource assignment to the evolving state of agentic programs.

3. Scheduling Mechanisms

Autellix implements two non-clairvoyant scheduling algorithms designed to optimize resource allocation for both single-threaded and multi-threaded program workflows:

• PLAS (Program-Level Attained Service): In single-threaded settings, the priority $p(c_j)$ of an LLM call $c_j$ is calculated as the sum of execution times $t_k$ for previously completed calls in the same program:

$p(c_j) = \sum_{k < j} t_k$

Programs that have already consumed significant computational resources see their subsequent calls deprioritized, ensuring fair resource distribution and minimizing aggregate wait times. This methodology is inspired by the Least-Attained-Service scheduling discipline.

• ATLAS (Adaptive Thread-Level Attained Service): For multi-threaded programs represented as dynamic directed acyclic graphs (DAGs), ATLAS recursively assigns priorities, favoring dependency chains with the lowest cumulative service. For an incoming call $c_j$ with parent calls $\mathrm{Parents}(c_j)$:

$$p(c_j) = \max_{c_k \in \mathrm{Parents}(c_j)} (p(c_k) + t_k)$$

This results in prioritization along the program’s critical path, improving the responsiveness of logically interconnected tasks.

Both algorithms discretize priority values into multiple queues conforming to a multi-level feedback queue (MLFQ) schema. This structure supports preemption: calls exceeding their quantum are demoted to lower-priority queues. An anti-starvation mechanism—a boosting rule based on the ratio of waiting time to service time exceeding a threshold $\beta$—prevents indefinite postponement of lower-priority calls.

4. Performance Evaluation

Empirical analysis demonstrates significant performance gains with Autellix in real-world agentic workloads, including multi-turn chatbot conversations, ReAct frameworks, and multi-threaded planners such as Monte Carlo Tree Search (MCTS):

System	Throughput Improvement	Tail Latency Impact
vLLM	Baseline	Higher 95th/99th percent.
Autellix	4–15× (vs. vLLM)	Lower 95th/99th percent.

Reducing program-level and call-level blocking allows short or low-service programs to complete rapidly, increasing the initiation rate for new calls. In scenarios with multiple engines, locality-aware load balancing ensures calls with high KV-cache hit rates remain engine-affined, optimizing context reuse; quick, cache-light calls are distributed across GPUs to balance load. Autellix’s program-aware scheduling delivers measurable improvements in both mean and tail (95th, 99th percentile) latency compared to preemptive but program-oblivious schemes.

5. Principal Use Cases

Autellix excels in scenarios characterized by agentic program workflows, where aggregation and synchronization of multiple LLM calls are essential:

• Conversational Agents: Enables cumulative multi-turn interactions with efficient scheduling across evolving dialogue contexts.
• ReAct Agents: Supports dynamic control flows requiring interleaved reasoning and actions.
• Parallel Planning Algorithms (e.g., MCTS): Manages concurrent LLM evaluations and coordination for search and planning tasks.

By multiplexing a large number of concurrent requests and reducing cumulative wait times, Autellix supports both interactive user-facing agents and high-throughput batch processing—for reinforcement learning pipelines, distributed post-training of large models, and complex agent-based systems.

6. Limitations and Prospects

Autellix’s approach, while demonstrating substantial system-level efficiency, has explicit limitations:

• The dynamic internal program DAG is constructed during execution, without speculative prefetching or branch prediction; integration of compiler-style optimizations could further reduce scheduling delays.
• The stateful API currently lacks rigorous tamper-resistance, indicating future development needs for security and robustness in production contexts.
• Existing scheduling algorithms operate in a non-clairvoyant regime, absent foreknowledge of a program’s runtime graph. Incorporating predictive feedback mechanisms or partial trace estimation may approach optimal scheduling.
• Integration with distributed reinforcement learning or post-training workflows presents opportunities for throughput enhancement in large-scale LLM deployments.

This suggests that further investigation into predictive scheduling and robust stateful session management could extend Autellix’s impact beyond current benchmarks.

7. Significance in LLM Systems

Autellix introduces a shift in the design of LLM serving engines by centering optimization at the program rather than call level. Its explicit modeling of program dependencies, combined with program-aware scheduling (PLAS, ATLAS) and locality-sensitive resource allocation, positions it as a reference architecture for interactive agentic AI systems. The measured improvements in throughput (4–15×) and tail latency advance the practicality of deploying responsive, scalable LLM agents across academic, industrial, and real-time settings.