Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference

Abstract: Long-running LLM agents keep valuable state resident on GPUs: KV caches, request schedulers, communication state, and sometimes online adapters. Losing this state after a GPU or communicator failure can discard minutes to hours of work, yet existing recovery mechanisms either restart the whole serving stack or require application-specific checkpoint logic inside every attention and runtime component. This paper argues that fault tolerance for such workloads needs a GPU-resident execution context: checkpoint hooks must run at device synchronization points, observe binary kernels that frameworks and libraries actually execute, and recover without putting the host CPU on the critical path. We present Concordia, a runtime that uses a device-resident persistent kernel as the substrate for fault-tolerant LLM inference. Concordia interposes on GPU module loading and supports PTX- and SASS-level instrumentation, allowing checkpoint and pause hooks to be inserted below framework code and library boundaries. For each registered LLM state region, Concordia JIT-compiles a specialized delta-checkpoint handler -- for example, a KV-block scanner, adapter-page scanner, or recovery applier -- and hot-swaps it into the persistent kernel's operator table. The persistent kernel consumes a lock-free ring buffer of compute, checkpoint, append-log, and recovery tasks, so the same always-on executor triggers dirty-page detection, stages deltas, and appends committed records to a CPU-visible log in CXL memory or host DRAM.

• The paper presents a GPU-native checkpointing system using JIT-compiled handlers, achieving up to 219× speedup over traditional CPU-side methods.
• The paper details a persistent kernel design that minimizes SM overhead (<1 block) and decouples recovery from host mediation, ensuring sub-microsecond trigger submission.
• The paper demonstrates cross-architecture portability and minimal inference overhead with second-scale recovery through an efficient append-only recovery log.

Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference

Problem Motivation and System Requirements

Stateful LLM inference workloads—characterized by persistent GPU-resident KV-caches, request schedulers, and online adapters—expose vulnerability to device or communication failures that lead to irrecoverable loss of execution state. Existing recovery mechanisms either rely on full-stack restarts or require complex, application-specific checkpoint logic dispersed across attention kernels, allocators, and framework paths. This paper claims that robust, low-latency fault tolerance requires GPU-native checkpoint hooks operating at device synchronization points, with the ability to observe the actual binary kernels being executed and implement recovery without involving the CPU in the critical path.

Concordia Design: Persistent Kernel Substrate and Instrumentation

Concordia implements a device-resident persistent kernel as a foundational runtime substrate. This kernel continuously executes for the lifetime of an LLM agent, consumes a lock-free ring buffer of compute, checkpoint, append-log, and recovery tasks, and maintains operator tables for dynamic function dispatch. Key mechanisms include:

• PTX/SASS-level instrumentation: Concordia interposes on GPU module loads, inserting pause and checkpoint hooks at kernel/collective boundaries, enabling binary-level coverage that transcends framework and vendor library boundaries.
• JIT-compiled checkpoint handlers: For every registered LLM state region (e.g., KV-cache, adapter pages), Concordia generates specialized GPU-side delta checkpoint handlers—block-table-aware scanners, shadow-compare diff engines, or recovery appliers—hot-swapped into the persistent kernel operator table.
• Append-only recovery log: The persistent kernel appends committed checkpoint records (region ID, version, dirty page list, payload) to a CPU-visible log in host DRAM or CXL memory, analogously to Redis AOF, with periodic compaction for efficient replay.

Accelerating Transparent GPU-Side Delta Checkpointing

The motivating experiment demonstrates Concordia’s delta checkpointing capability on RTX PRO 6000 Blackwell, contrasting CPU-side and GPU-side approaches for a typical sparse KV-cache update. CPU-side methods incur significant latency due to device-to-host transfer and host-side comparison, whereas GPU-side detection leverages HBM bandwidth for rapid diffing and only transfers dirty pages.

Figure 1: On RTX PRO 6000, GPU-side delta checkpointing achieves up to $219\times$ speedup versus CPU-side transparent page scanning, with host comparison dominating CPU-side cost and GPU diffing bounded by HBM bandwidth.

This result exposes a fundamental scaling advantage: CPU-transparent checkpointing incurs $O(\text{region\_size})$ bandwidth cost irrespective of mutation rate, while Concordia’s GPU-side diffing scales with $$O(\text{region\_size}/\text{HBM\_BW}+ \text{dirty\_size}/\text{PCIe\_BW})$$, matching LLM's sparse state mutation patterns.

Persistent Executor Microarchitecture and Dispatch Efficiency

Concordia’s persistent executor incurs minimal ($<1$ block) SM overhead and enables sub-microsecond checkpoint trigger submission, eliminating the need for fresh kernel launches at checkpoint boundaries.

Figure 2: Heatmap of dispatch latency across operators and tensor sizes, showing uniform latency in the dispatch-dominated regime for small N due to Concordia's persistent kernel design.

This makes checkpointing and recovery triggers merely ring-buffer task submissions, decoupling launch overhead from recovery latency.

Inference and Checkpointing Overhead

Evaluations on Qwen3-0.6B demonstrate that Concordia supports regular inference throughput with modest checkpointing overhead. Concordia exploits the sparse mutation structure: after initial KV-cache warmup, subsequent checkpoints incur negligible cost as only dirty pages are appended.

Figure 3: Qwen3-0.6B inference: per-boundary checkpoint overhead is minor, and throughput remains stable; initial prompt is slower due to KV-cache allocation.

Fault Tolerance and Recovery Timelines

Concordia integrates with NCCL collectives for rapid per-boundary checkpointing and fast recovery. Recovery proceeds via detection, isolation, state restoration from append-only logs, and reintegration.

Figure 4: Timeline of Concordia GPU fault recovery: detection (10 ms), isolation (300 ms), state restoration (800 ms), reintegration (400 ms), — total $\sim$1.5 s.

This prototype demonstrates service continuity and second-scale recovery without full communicator restart, in contrast to traditional approaches that lead to >47s outages.

Resource Overhead and Throughput

Concordia’s persistent kernel is resource-frugal, occupying only 0.5% of SMs, compared to 7.2% for CLC and 4.8% for Green Contexts. Throughput impact under overlapped delta checkpointing is minimal, validating practical deployment.

Figure 5: (a) SM overhead: Concordia (0.5%) vs. CLC (7.2%), Green Contexts (4.8%). (b) Throughput impact with overlapped delta checkpointing is negligible.

Cross-Architecture Portability and Migration

Concordia’s optional CTX portable IR enables migration across NVIDIA, AMD, Intel, and Tenstorrent architectures. Microbenchmarks show close-to-native performance overhead for compute intensive kernels, with JIT compilation amortized across runs.

Figure 6: Concordia microbenchmark and migration across H100, RX~9070~XT, Iris Xe, BlackHole; bottom: JIT compilation and live migration timeline across heterogeneous GPUs.

Real-World Workloads

On real-world SWE-Bench workloads, Concordia sustains efficient tokens/sec, with overheads dependent on model and task complexity, supporting heterogeneous fleet deployments.

Figure 7: Throughput analysis for real-world LLM workloads under Concordia fault-tolerant execution.

Implications and Future Directions

Concordia establishes that persistent-kernel checkpointing, device-native delta detection, and append-only recovery logs are a unified substrate for cross-framework, low-latency fault tolerance in LLM serving. The persistent executor decouples checkpointing and recovery from host mediation and enables device-resident recovery in heterogeneous environments. Practically, this allows migration, rapid recovery, and improved availability in production LLM deployments, particularly as agentic and online-adaptive workloads proliferate.

Theoretically, Concordia’s approach suggests future AI runtimes should move toward device-autonomous checkpointing and recovery, leveraging opaque binary instrumentation, composable persistent kernels, and host-agnostic recovery protocols. Hardware support for dirty-page tracking, AOT-generation for frequent state layouts, and extensions for in-kernel operator scheduling will further strengthen this substrate.

Conclusion

Concordia offers a GPU-resident, JIT-compiled persistent-kernel approach to transparent, fault-tolerant LLM inference. By combining binary instrumentation, region registration, GPU-side delta checkpointing, and append-only recovery logs, Concordia achieves up to $219\times$ checkpointing speedup, sub-microsecond triggers, and second-scale recovery, all with minimal SM footprint and cross-architecture portability (2606.23521). The persistent kernel transforms fault tolerance from a peripheral concern to a core runtime service for long-running agentic inference, scaling across frameworks and device ecosystems.