Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs

Abstract: Serverless multi-model LLM systems multiplex popularity-skewed model catalogs over shared GPU pools, yet typically schedule each request independently. Tool-using agents break this abstraction: a session repeatedly calls an LLM across short tool gaps, carries a long reusable KV prefix, and is judged by session completion time (SCT). Load-only routing can separate a continuation from both its model and KV state, while round-based model multiplexing can delay even a correctly placed continuation until the target model's next slot. Both failures are especially costly for hundred-billion-parameter models: their weights constrain residency, while long-context KV is expensive to reconstruct or move. We present Talaria, a session-aware serverless multi-model serving system that makes session continuity a joint placement-and-admission decision. Its router ranks placements by model residency, KV locality, and instance pressure, while soft reservations account for likely returns in the last serving instance's admission budget. Session-prefill (SP) admits budget-eligible continuations before the active model slot closes. An instance-local substrate keeps HBM addresses stable, preserves host-restorable KV, and stages weights across model switches. On a single TP=8 server, we replay 30 SWE-Bench model-sessions (960 calls) over three models, each with more than 100B total parameters. Against an otherwise identical round scheduler with SP, host-KV restoration, and D2D staging disabled, Talaria cuts p50 SCT from 1000 s to 189 s and p95 from 2296 s to 867 s, speedups of 5.3x and 2.6x.

• The paper introduces Talaria, a session-aware serverless serving system that combines soft-reservation routing, session-prefill admission, KV restoration, and D2D weight staging to preserve agent-session locality.
• Talaria reduces median session completion time from 1,000 to 189 seconds, cuts p95 completion time 2.6×, lowers median TTFT from 13.44 to 0.55 seconds, and improves 29 of 30 evaluated sessions.
• The evaluation shows that session-prefill delivers the largest performance gain, while host KV restoration and D2D staging reduce recovery and switching costs, though cluster-scale deployment, dynamic pool management, and tail latency remain open challenges.

Motivation: agent sessions break request-level scheduling

Talaria addresses a scheduling gap in serverless multi-model LLM serving. Existing systems multiplex popularity-skewed model catalogs over shared GPU pools but schedule each request independently. Tool-using agents violate this abstraction in three ways: a session issues many model calls separated by short tool gaps, each call carries a long and largely reusable KV prefix, and the user-visible latency metric is session completion time (SCT) rather than per-request TTFT.

The paper grounds this claim in a characterization of 445 completed SWE-Bench Verified sessions comprising 7,386 LLM calls. The median session depth is 11 model calls (p95: 49, max 187); the median tool-use gap is 0.39 s (p95: 1.32 s); the median prompt is 25K tokens (p95: 105K); and 96.8% of continuations reuse more than half of the prior call's prefix. The critical asymmetry is temporal: recomputing a median-length prefix on Qwen3-235B at TP=8 costs 4.3 s TTFT (10.9 s at 100K tokens), more than an order of magnitude longer than the median return gap. A misplaced or deferred continuation therefore pays a recovery cost that dominates its queueing savings.

The paper identifies two coupled failure modes. Spatial mismatch: load-only routing can send a continuation to an instance holding neither the target weights nor the session KV, so KV movement or prefix recomputation exceeds the queueing delay avoided. Temporal mismatch: even with correct placement, round-based model multiplexing (e.g., Aegaeon-style token-level time-sharing) defers a returning session until the target model's next slot, squandering preserved locality on the time axis. Under prefill/decode disaggregation, an analogous cross-tier mismatch arises; Talaria deliberately evaluates a co-located design to avoid per-turn KV handoff.

System design

Talaria separates a control plane (a stateful router) from a data plane (hot and cold inference instances), built on SGLang as a Go gateway (~5.5K lines) plus ~15K lines of engine extensions.

Hot/cold pool organization. Hot-pool instances pin one model and never switch, eliminating switching tail latency for stable traffic. Cold-pool instances token-level time-share long-tail models via round-based scheduling. Pool membership is configured at deployment time; dynamic promotion/demotion is explicitly out of scope for the prototype.

Session-aware residency routing. After a call completes, the router installs a soft reservation—an admission lease $(s, m, i, k, t_{exp})$ recording session, model, serving instance, prefix handle, and expiry ($\tau = 1$ s, near the p90 tool gap). The lease charges one admission unit against the instance's next eligible model-slot budget without pinning device KV. A matching return consumes the lease and routes back to that instance; otherwise placement falls back to a cost-ranking policy that charges incremental queue/prefill/decode work, KV recovery (zero for device-resident, measured H2D restore for host-restorable, profiled recomputation otherwise), and model activation exactly once. Replica opening is gated by a gain test comparing predicted demand benefit against switch latency plus an HBM-pressure-priced footprint term. The paper is careful that the lease provides bounded affinity, not a same-slot admission guarantee or TTFT deadline, since the next suffix length and concurrent returns are unknown at reservation time.

Session-prefill (SP) admission. Cold-pool slots follow the structure $$P \rightarrow D_1 \rightarrow (SP \rightarrow D)^* \rightarrow D_{\text{fin}}$$. At decode boundaries, SP batch-admits eligible returns—those targeting the active model with device-resident or host-restorable prefixes—whose predicted prefill, restore, and HBM costs fit the budget remaining after protected decode. The scheduler reserves decode iterations $d_m$ satisfying a TPOT-target inequality before allocating slot budget to prefill and SP, and bounds rounds below $\gamma T_{\text{TTFT}}$. Overload is surfaced through staging-slack and deadline-risk telemetry rather than silently rolled into later rounds.

Memory substrate. Three mechanisms make the scheduling contract executable. First, a bump-managed HBM layout places weights/KV growing upward and runtime buffers anchored high, with SGLang's torch_memory_saver (VMM-based) parking CUDA-graph and runtime allocations per model while preserving virtual addresses—so captured kernels remain valid across switches. Second, a host KV registry (HKVR) checkpoints aligned KV blocks asynchronously into a unified pinned-memory pool shared across MHA, MLA, and Mamba-style layouts, restoring via layer-first/page-first transfer kernels and recomputing only unconfirmed tails. Correctness rests on three invariants: separate DMA pins from radix ownership, publish prefixes only when all TP ranks agree (truncating disagreement to the longest common block range), and drain outstanding D2H events before freeing a model's KV region. Third, opportunistic D2D weight staging pre-stages the next model's weights into the clean high-address tail of KV layer blocks during service; dirty chunks fall back to H2D at switch time.

Evaluation

The primary evidence is a fixed replay of ten Astropy SWE-Bench issues across Qwen3-235B, GLM5-nvfp4, and Qwen3.5-122B-A10B on one TP=8 server (eight 183 GiB GPUs), yielding 30 model-sessions and 960 calls. Fixed replay holds request bodies, IDs, call order, completion-token counts, and gaps constant, preventing agent-path divergence; global interleaving may still differ. Router experiments use a separate two-worker TP=4 testbed with smaller mapped models to isolate placement decisions—an intentional split the paper acknowledges means cluster-scale behavior with hundred-billion-parameter models is not directly measured.

End-to-end results. Against the Round-only ablation (identical round scheduler with SP, HKVR, and D2D staging disabled), Talaria cuts p50 SCT from 1000 s to 189 s (5.3×) and p95 from 2296 s to 867 s (2.6×), improving 29 of 30 paired model-sessions. p50 TTFT falls from 13.44 s to 0.55 s. Reverse ablations show non-additive marginal effects: SP contributes the largest p50 reduction (623 s → 189 s); HKVR reduces p50 SCT from 486 s to 189 s and TTFT p95 from 26.33 s to 14.17 s; D2D staging leaves p50 nearly unchanged (194 s → 189 s) but reduces p95 from 933 s to 867 s.

Mechanism	Marginal effect (p50 SCT)	Notable secondary effect
SP	623 s → 189 s	p50 TTFT 4.60 s → 0.55 s
HKVR	486 s → 189 s	TTFT p95 26.33 s → 14.17 s
D2D staging	194 s → 189 s	p95 933 s → 867 s

Routing. On the 120-call two-worker trace, the full router eliminates avoidable model opens at low and medium load and leaves one at high load (versus 37 for least-pressure), retains 97.8% of returning calls on the same worker under high load, and cuts TTFT p95 from 8.07 s to 5.28 s. The paper reports a candid negative result: at low load, eliminating avoidable opens yields no SCT gain (full-router p50/p95 of 7.39/15.81 s versus 5.62/13.68 s for session-sticky), and at high load it improves p95 but not p50. Residency-aware placement thus pays off mainly at medium load.

Reservation sensitivity. Sweeping $\tau \in \{0, 0.5, 1, 2, 5\}$ s shows a genuine tradeoff: $\tau=0$ gives 56.2% same-worker returns and 72.5% model-residency hits; $\tau=1$ s raises returns to 67.7% with the lowest returning-call TTFT p95 (1.91 s) but drops residency hits to 60.8% and increases opens from 33 to 47; $\tau=5$ s reaches 79.2% returns but only 53.3% residency hits. The unused-lease proxy at $\tau=1$ s is 50%, indicating substantial reserved capacity goes unconsumed—a cost the proxy does not fully quantify.

SP admission. SP admits 47.8% of recorded return-handling events within the active slot (316 admitted, 345 parked over 200 rounds). The paper notes these are schedule-dependent diagnostics—a parked request may appear in multiple rounds—and uses TTFT over the common call set as the direct cross-policy result.

Substrate. D2D staging reduces aggregate switch time by 38% (907 s → 565 s) and switch share of round wall time from 51.7% to 34.6%. On logical switches (max across TP ranks), p50 falls from 1497 ms to 984 ms and mean from 1521 ms to 1071 ms—but p95 rises from 2133 ms to 2679 ms, a tail tradeoff caused by active writes dirtying staged chunks. Full-D2D switches (27.7% of the mix) complete at 106 ms p50; partial coverage (36.3%) lands near 1000 ms; H2D fallback (36.0%) sits at 2011 ms. Peak observed HBM use is 177.1 GiB, below the 183 GiB cap. A component stress run attributes the switch path almost entirely to weight opening (Phase 2): full D2D reduces Phase 2 p50 from 2.57 s to 110 ms.

Limitations and open questions

The paper concedes several boundaries plainly. Evidence combines single-server large-model execution with isolated two-worker placement experiments; broader workloads and cluster-scale evaluation remain open. Pool membership is static, and dynamic hot-cold reallocation is unimplemented. The soft reservation cannot guarantee same-slot admission because suffix lengths and concurrent returns are unknown at lease creation, and the 50% unused-lease rate suggests headroom for workload-adaptive lease policies. D2D staging's p95 regression under high KV pressure persists because dirty-chunk fallbacks reload from host; reducing those fallbacks is identified as the path to tail improvements. Extending SP admission across P/D-disaggregated tiers would require coordinated cross-tier KV ownership, which the co-located design sidesteps rather than solves. Finally, the replay caps inter-call gaps at 10 s (affecting 20 of 960 calls), and global interleaving differences mean policy comparisons are paired but not perfectly controlled.

Conclusion

Talaria argues that session continuity—not the individual request—is the missing scheduling unit in serverless multi-model LLM serving for agents, and realizes it through complementary mechanisms: soft reservations at the router preserve return affinity without pinning device KV, while SP admission exposes mid-slot execution opportunities inside cold-pool slots. A bump-managed HBM layout, host-restorable KV registry, and D2D-staged switching make these decisions executable within round budgets. The headline result—5.3× p50 and 2.6× p95 SCT improvement over an internally controlled round baseline, with 29 of 30 sessions improved—is strong, though the evaluation scope (one server for end-to-end results, two workers for placement) leaves cluster-scale generality as the principal open empirical question.