Online Linear Programming for Multi-Objective Routing in LLM Serving

Abstract: We study the online routing problem in LLM serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and KV-cache constraints. Unlike widely used routing heuristics that are not tied to explicit service-level objectives (SLOs) and offer limited control over latency-throughput trade-offs, we introduce a multi-objective optimization framework that formulates routing as an online linear programming with interpretable decision rewards. We apply an efficient bid-price control policy based on the online linear programming that admits requests when their SLO-weighted benefit exceeds their shadow prices. To meet millisecond decision requirements, we develop a warm-started, projected first-order updates that track the evolving dual shadow prices online with predictable runtime. We integrate our router into the Vidur simulator and demonstrate substantial improvements over standard baselines across multiple SLO regimes, including end-to-end latency, time-to-first-token, throughput, and tail performance. A big picture from our result: a science-based approach outperforms others based on heuristics.

• The paper introduces a multi-objective online LP framework that optimizes LLM routing by balancing throughput, latency, TTFT, and tail SLOs via tunable reward weights.
• It employs bid-price control with LP duality to efficiently allocate resources under time-coupled constraints, significantly outperforming traditional heuristics.
• Empirical results from the Vidur simulator demonstrate up to 45.75% lower average latency and scalable adaptation to dynamic, heavy-tailed workloads.

Online Linear Programming for Multi-Objective Routing in LLM Serving

Problem Overview and Motivation

The increasing scale and diversity of LLM services has elevated the complexity of routing requests in distributed LLM inference contexts. Each user request submitted to an LLM undergoes distinct prefill and decode stages, with unpredictable sequence lengths and variable memory and compute requirements. Production LLM serving stacks must simultaneously accommodate stringent resource, batch, and KV-cache constraints while meeting heterogeneous service-level objectives (SLOs), such as end-to-end latency, time-to-first-token (TTFT), throughput, and tail latency. Existing SLO-agnostic routing heuristics (e.g., round-robin, join-shortest-queue) lack transparency, offer limited control over metric trade-offs, and handle resource contention without explicit optimization.

The paper "Online Linear Programming for Multi-Objective Routing in LLM Serving" (2607.03948) introduces a rigorous multi-objective online linear programming framework for LLM request routing. It formulates online inference serving as a resource allocation problem under time-coupled constraints and decomposes aggregate SLOs into per-request rewards suitable for tractable, interpretable policy design.

Multi-Objective LP Formulation for LLM Routing

The proposed framework recasts LLM routing as an online packing-style linear program over a finite time horizon. At each period, the system must allocate requests (arriving sequentially with unpredictable decode lengths) to a set of parallel decode devices, constrained by per-device batch sizes and KV-cache memory budgets. Each request, once routed, induces time-coupled occupancy over its decode lifetime on a target device, with explicit accounting of its dynamic resource usage.

The key modeling insight is the construction of a reward function for each candidate assignment $(j,g,k)$ (request $j$, device $g$, start time $k$), formulated as a linear combination of:

• Throughput contribution: number of tokens the assignment will process within the window;
• End-to-end latency penalty: negative cost proportional to expected completion time minus arrival time;
• TTFT penalty: negative cost for time until first token;
• Queries-per-second contribution: positive additive term for admitting a request;
• Tail-latency satisfaction bonuses: indicator rewards for meeting tail threshold constraints for the assignment.

By exposing the reward weights $$(\alpha,\,\beta,\,\gamma,\,\sigma,\,\zeta_1,\,\zeta_2)$$ as tuning knobs, the framework allows explicit trade-off management among competing objectives and enables the Pareto-efficient exploration of metric frontiers.

Concretely, packings must respect, at each device-time-constraint coordinate, a summary over admitted requests of batch occupancy and memory usage. The sequential, time-coupled nature of requests leads to complex, forward-looking resource coupling.

Figure 1: Illustration of Prefill/Decode (P/D) disaggregation with continuous batching. Active requests are processed token-wise in an always-updating batch as completed jobs make space for new admissions.

Algorithmic Contributions: Online Bid-Price Control

Directly resolving a large linear program at every online decision point is computationally prohibitive given tight millisecond-scale latency requirements. The authors leverage LP duality to derive shadow prices (bid prices) for each temporal-local resource constraint. Routing decisions are then cast as bid-price control: for each candidate assignment, the router computes the difference between its SLO-weighted reward and the dual-valued resource cost. Admission proceeds greedily in descending order of positive margins, subject to resource and assignment feasibility. If no assignment achieves positive net benefit, the router opts to defer or buffer.

To ensure both adaptivity and efficiency, the authors propose a warm-started, projected online subgradient update procedure for dual variables. The optimization leverages historical action-reward data, incrementally tracks dual congestion signals, and requires no full re-solving of the primal at inference time. This bid-price adaptation grants interpretable, robust, and computationally predictable policy updates, with scaling linear in system and horizon size.

Integration, Evaluation, and Empirical Results

The proposed routing policy was integrated into the Vidur simulator, which models realistic LLM serving with continuous batching and resource management. The experimental protocol encompasses both real data (drawn from the lmsys-chat-1m dataset) and synthetic heavy-tailed workloads, under varying Poisson arrival rates and decode-length prediction accuracy. Performance is benchmarked against industry-standard policies: Round Robin, Least Outstanding Request, Random, and Power-of-2 Choices.

Strong empirical results are achieved in all SLO regimes:

• Substantial numerical improvement: The LP-based router outperforms all baselines by a significant margin—e.g., it achieves 45.75% lower average EEL and 42.49% lower P95 EEL relative to Round Robin under noisy decode-length prediction (see main text Table 1).
• Robust tail control: High weights on tail metrics directly translate into marked improvements at high quantile latency thresholds, with clear explicit degradation/improvement as weights are varied.
• Pareto trade-off control: Systematic adjustment of reward weights produces interpretable Pareto-efficient shifts between throughput and the full suite of latency objectives (see Figure 2 for TTFT satisfaction as a function of tail weight).
• Adaptivity to workload drift: The rolling-window dual update supports dynamic adaptation to shifting arrival rates and workload shocks, outperforming static baselines under non-stationary conditions.

Figure 2: TTFT satisfaction rate as a function of the tail weight $\zeta_2$. Increase in tail metric weight systematically improves the fraction of requests meeting strict TTFT SLOs.

Structural Insights and Implications

The LP dual variables (shadow prices) serve as explicit, interpretable congestion signals for system operators—diagnosing which resource (batch or KV memory) is globally or temporally bottlenecked under prevailing demand and system configuration. Practitioners gain:

• Unified capacity planning: Objective weights provide a systematic lever for exploring workload–throughput–latency frontiers, enabling proactive allocation and tuning as opposed to reactive, heuristic per-stack re-engineering.
• Admission control and overload resilience: The framework provides a natural SLO-aware mechanism for request prioritization or deferral when demands exceed available resources.
• Scalable deployment: The algorithm’s compute profile and state requirements are commensurate with real system constraints and remain efficient even at high device and batch scale.

Limitations and Future Directions

The current validation remains simulation-based, with anticipated engineering and systems integration challenges for direct deployment in production (e.g., vLLM, handling preemption, distributed coordination). The framework presently assumes homogeneous request priorities and does not natively encode fairness or class-based isolation, but is extensible to such objectives via the LP machinery.

Several future research directions are immediate: productionization with in-the-wild telemetry, extension to heterogeneous hardware, explicit fairness and multi-tenant SLOs, and joint optimization with upstream prefill batching.

Conclusion

This work formalizes LLM serving routing as a multi-objective, time-coupled linear program and demonstrates efficient, interpretable bid-price-based online control. The framework achieves strong improvements in all principal SLO regimes over classical heuristics, exposes the fundamental trade-offs of LLM serving, and supplies a robust foundation for future scalable, optimization-based serving architectures. This methodology constitutes a paradigm shift toward systematic, science-driven system design and metrics-driven capacity planning in LLM inference infrastructure.