Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving

Abstract: Mainstream LLM serving systems reuse prefix work mainly through paged or radix key-value (KV) caches. This is highly effective for high-throughput, high-concurrency serving, but it manages only one positional fragment of execution state: the KV cache. We study the opposite regime: low-latency, small-batch, on-device physical-AI serving, where interactive LLM agents, speech systems, and robot policies repeatedly branch, reset, interrupt, and re-enter under tight responsiveness budgets. We introduce execution-state capsules, a graph-bound checkpoint and restore mechanism for the complete restorable state at a committed boundary. FlashRT is a white-box, backend-facing kernel runtime whose evaluated NVIDIA CUDA backend runs captured graph plans over contiguous static buffers with no block-table indirection. Because the live state is a closed set of named buffers, a capsule can snapshot, restore, fork, or roll back the whole execution boundary, including KV, recurrent state, convolution state, MTP state, and metadata. This moves reuse from token-addressed KV fragments to graph-bound execution-state boundaries. On an RTX 5090, capsule restore is byte-exact at the stored-state level and token-identical under greedy decode. A KV-only ablation diverges, showing that recurrent state is load-bearing. GPU-resident snapshot and restore are sub-millisecond, and TTFT speedup over cold prefill grows from 3.9x at 2k tokens to 27x at 16k tokens. On Jetson AGX Thor and DGX Spark, the same correctness and structural properties hold. Capsules are not a replacement for high-throughput KV-cache serving; they define a complementary latency-first serving point for explicit execution-state reuse.

• The paper’s main contribution is the execution-state capsule abstraction that checkpoint, restore, fork, and rollback LLM and embodied AI states.
• It demonstrates a methodology achieving flat time-to-first-token with up to 26.94x speedup over traditional KV-cache techniques like vLLM.
• The approach enables byte-exact state recovery and efficient multi-session branching, ensuring reliable low-latency physical-AI serving.

Execution-State Capsules: System-Level Checkpointing for Low-Latency Physical-AI Serving

Conceptual Framework and System Design

The paper introduces a new paradigm for LLM and physical-AI serving centered around the "execution-state capsule" abstraction (2606.20537). The principal innovation is explicit system-level state management for LLMs, embodied agents, and multimodal pipelines in interactive, low-latency, small-batch, and on-device deployment contexts.

Prior art such as vLLM's PagedAttention and SGLang's RadixAttention optimize for aggregate throughput via token-indexed KV-cache management. These mechanisms rely on block-table or radix-tree indirection for positional reuse, which precludes freezing the execution state as a self-contained object. Capsules move the unit of reuse from token-addressed KV fragments to graph-bound execution-state boundaries: a fixed set of contiguous static buffers representing KV, recurrent, convolutional, and metadata states, bound to the captured CUDA graph(s) and checkpointed as a binary blob.

FlashRT, the kernel-level runtime evaluated here, establishes a latency-focused substrate—a backend-facing pipeline with aggressive graph-capture, static buffer allocation, and hand-written CUDA kernels. Its cold time-to-first-token (TTFT) is $2.6$--$2.8\times$ lower than vLLM’s on identical hardware/model, defining a single-stream latency floor.

Capsule Mechanism and Operational Semantics

Capsules support four verbs: snapshot, restore, fork, and rollback. Snapshot freezes the entire graph-bound buffer set at a meaningful boundary (e.g., after a long prefix). Restore re-copies the buffer set, avoiding expensive recompute. Fork clones a boundary into $N$ independent sessions with token-exact equivalence. Rollback restores a previous boundary, enabling state reversal for retry or undo.

The capsule abstraction turns prefix reuse from an expensive compute-bottleneck into a bandwidth-limited copy-and-rebuild operation, scaling linearly with capsule size rather than prefix length. The graph executable is strictly decoupled from the state blob; it's the combination that renders restore operations both cheap and byte-exact.

(Figure 1)

Figure 1: Runtime floor and state-reuse floor, identical model/GPU, single-stream; capsule preserves the latency floor across explicit boundaries.

Correctness Envelope and State Restoration

Correctness is multilayered: byte-wise buffer restoration, completeness (every buffer the forward pass depends on is checkpointed), and end-to-end token/action equivalence. The hybrid LLM model relies on non-positional recurrent and convolutional state (folds with no addressable block slices); ablation shows KV-only restoration diverges immediately ($97.9\%$ token mismatch). Exact replay requires chunk-alignment: boundary positions must be multiples of the chunk size, matching model-specific constraints for deterministic state recovery.

(Figure 2)

Figure 2: Embodied loop, single-stream, under comparable reported device residency; capsule TTFT remains flat as working set grows.

Numerical Results and Comparative Analysis

Single-stream TTFT/TTFA: On a 5090 GPU, capsule TTFT remains flat ($51$--$57$ ms) as prefix grows from $2$k to $16$k tokens (max speedup $26.94\times$), while cold prefill grows linearly ($200$--$2.8\times$0 ms). Capsule restore and snapshot are sub-millisecond bandwidth-bound copies; append is a flat $2.8\times$1--$2.8\times$2 ms.

vLLM Comparison: Capsule TTFT consistently beats vLLM APC under reuse ($2.8\times$3--$2.8\times$4 lower), and holds explicit pinning rather than ephemeral cache residency. Under typical physical-AI workloads (skill cycling, interrupts, branch/re-entry), vLLM falls back to cold latency when cache eviction occurs or session mismatches arise, while capsules preserve their performance envelope via explicit buffer management.

(Figure 3)

Figure 3: LLM+TTS barge-in, composed latency; capsule restores pinned persona, yielding $2.8\times$5 faster pipeline entry vs naive re-prefill.

Embodied Working Set: Capsules hold flat TTFT across large context sets (up to $2.8\times$6k tokens), leveraging explicit device residency. vLLM APC collapses under VRAM exhaustion or hybrid block-retention limits.

On-Device Replication: Results on Jetson AGX Thor and DGX Spark confirm the mechanism's architectural independence. Thor’s cold-to-capsule speedup ranges $2.8\times$7--$2.8\times$8, Spark $2.8\times$9--$N$0, both maintaining flat capsule TTFT ($N$10.2 s).

Figure 4: Cross-device, single-stream TTFT scaling; capsule speedup increases with prefix length, flat across hardware.

Implications and Structural Distinctions

This work shifts the serving design axis, privileging explicit execution-state retention/pinning/forking/rollback over automatic cache policy. Capsules enable scenarios—multi-turn coding agents with pinned long prefixes, embodied episode resets, tree-of-thought branching, planner-actor hand-offs—that block/radix KV caches cannot express as first-class operations without significant architecture augmentation.

Persistent, device-tiered capsules unify warm start, multi-session branching, and hybrid recurrent state recovery in one object, robust to cache eviction, process churn, or session interruption. Fork and rollback become byte-exact primitives, not dependent on positional-token heuristics.

Figure 5: Embodied working set across hardware; capsule TTFT remains flat, APC retention collapses under VRAM constraints.

Limitations and Future Directions

Capsules are deployment-specific binary blobs, tightly bound to weights, quantization, and graph bucketing; portability is limited to identical deployments. Static buffer allocation and graph capture require bounded shape variants and fixed sequence lengths, unsuited to high-concurrency variable batching. Distributed cluster serving and decode throughput optimization remain out of scope.

Future research includes production-grade agent serving with dynamic prefix matching, broader serving-layer pinning policies, and full on-device/robotic evaluation, especially for safety-critical or irreversible physical-world settings.

Conclusion

By reifying the execution-state buffer set as a checkpointable capsule, the serving regime for interactive, low-latency, small-batch, and on-device AI gains explicit control of state persistence, recovery, and multi-session branching. The capsule mechanism is validated structurally and numerically, preserving hard responsiveness guarantees (TTFT/TTFA) across diverse domains under rigorous correctness criteria. The abstraction unifies LLM and embodied AI serving systems around explicit state management, providing a formal foundation for future low-latency, high-resilience interactive applications.