LUMEN: Coordinated Failure Recovery for Distributed LLM Serving

Abstract: Modern LLM serving clusters distribute inference requests across multiple worker processes on different GPUs, but failures are prevalent at scale. When a worker fails, the cluster simultaneously loses the failed worker's GPU-resident key-value (KV) caches and serving capacity, leaving surviving workers to absorb the redirected traffic while re-running interrupted requests from scratch. Existing fault-tolerant systems either restart interrupted requests from scratch or restore KV caches from checkpoints stored on a fixed neighboring worker, but both approaches route recovery work without considering current cluster load and leave the recovering worker idle during model reload. We present LUMEN, a fault-tolerant LLM serving system that treats recovery as a load-aware coordination problem across three decision points: checkpoint placement before failures, interrupted-request distribution at failure time, and serving capacity restoration during model reload. We evaluate LUMEN using both prototype experiments and large-scale simulations and demonstrate significant improvements in serving and recovery times.

• The paper presents a coordinated recovery mechanism that reduces time-to-first-token by up to 51% and recovery time by up to 64% compared to traditional methods.
• It leverages load-aware KV checkpointing and locality-aware scheduling to balance memory usage and minimize replay overhead during worker failures.
• The study demonstrates that speculation-assisted progressive recovery enables immediate throughput without incurring measurable steady-state overhead.

LUMEN: Coordinated Failure Recovery for Distributed LLM Serving

Introduction and Motivation

As LLM deployments scale to thousands of GPUs, failures—whether originating from hardware, node, or network issues—become routine, directly impacting availability, tail latency, and overall reliability of interactive LLM-based services. When a worker fails, the loss of both its GPU-resident KV cache and serving capacity causes request interruption and amplifies system-wide queuing and replay costs. Traditional recovery approaches, such as Stop-and-Restart and Fixed-Checkpointing, present clear limitations: they induce expensive replay on long-context requests, unbalanced recovery, and extended idle periods for recovering workers.

Figure 1: Single-worker failure in a simulated four-worker cluster illustrates significant spikes in TTFT and TPOT across the cluster.

Figure 2: Worker failures versus cluster size demonstrates that Stop-and-Restart exacerbates latency at scale.

Analysis reveals that failures degrade both time-to-first-token (TTFT) and time-per-output-token (TPOT) for virtually all requests, not just those directly interrupted. Queueing delay dominates TTFT for uninterrupted requests, while interrupted requests endure expensive replay due to lost KV cache, with replay costs that scale poorly with cluster size.

LUMEN Design and Architecture

LUMEN reframes failure recovery as a three-dimensional, load-aware coordination problem, leveraging real-time per-worker observations to achieve minimal end-to-end recovery cost:

• Load-aware KV Checkpointing: Distributes in-flight KV cache checkpoints across workers—based on updated load and restore pressure estimates—in order to mitigate recovery hotspots and balance memory usage.
• Locality-aware Recovery Scheduling: On failure, interrupted requests are routed primarily to their designated KV checkpoint holders, but heavily loaded holders trigger dynamic migration of requests (prioritizing those with less checkpointed prefix) to underutilized survivors, optimizing for low additional recomputation.
• Speculation-assisted Progressive Recovery: During model reload, the idle recovering worker loads a lightweight draft model and assists the most congested surviving worker by generating speculative tokens for parallel verification, thus contributing capacity instantly, even before its full model reload completes.

  Figure 3: LUMEN overview showing the gateway, the centralized controller, request and checkpoint placement, and the decoupling of checkpoint data transfers from the control path.

This architectural decomposition enables both fine-grained checkpoint placement and dynamic rebalancing without centralized data movement. LUMEN's progressive recovery pipeline avoids the typical stall in serving capacity by exploiting parallel speculative decoding, contributing immediate throughput via draft assistance.

Key Mechanisms

Load-aware KV Checkpointing

Checkpoint holders are assigned using a metric that integrates queueing delay and prospective restore pressure (estimated via the expected host-GPU restore latency under recovery conditions), ensuring each request's checkpoint is placed to minimize future adverse effects on latency and capacity.

Figure 4: Load-aware KV checkpointing distributes checkpoints based on worker load and memory constraints, with direct peer-to-peer transfer of KV pages and asynchronous streaming off the critical GPU path.

Incremental checkpointing, backed by atomic page tags and event-driven updates, allows for highly robust recovery, as in-flight token sequences can be deterministically mapped to checkpointed KV cache states.

Locality-aware Recovery Scheduling

Upon worker failure, the controller dispatches all interrupted requests to their checkpoint holders, but if a holder is overloaded, it migrates requests starting with those that incur the least additional recomputation cost—essentially those with shorter checkpointed prefixes—to the most underutilized workers. This tightly integrates load balancing with efficient reuse of cached states.

Speculation-assisted Progressive Recovery

The recovering worker immediately loads a draft model, pairing with the most congested surviving worker for speculative assistance. Batches from these assisted requests are fused with unassisted ones for a single target-model forward pass per step. The transition to full-capacity serving is facilitated by a background target-model load, allowing seamless resumption upon model availability and minimizing additional disk stalls.

Figure 5: Speculation-assisted progressive recovery shows the recovering worker assisting the most congested survivor while concurrently loading the target model.

Experimental Results

LUMEN's design is evaluated through both prototype clusters and large-scale simulation, spanning a range of failures, request loads, and cluster sizes. Notable outcomes include:

• On a four-worker prototype (Qwen3-32B), mean TTFT reduction by 44.4% over Stop-and-Restart, and 50.0% reduction in recovery time.
• In an eight-worker cluster (Qwen3-14B), mean TTFT reduction of 29.6% over Stop-and-Restart, and 64.1% recovery time reduction.
• In larger-scale simulation (up to 64 workers, 25% failure rate), LUMEN maintains mean TTFT 46.8%–51.2% lower than Stop-and-Restart and 12.8%–18.8% below Fixed-Checkpointing, with similar advantages in TPOT and recovery time.
• LUMEN shows no measurable steady-state overhead, as all asynchronous and control-plane operations are off the core inference path.

Figure 6: End-to-end prototype recovery shows LUMEN returning to baseline TTFT and TPOT considerably faster than prior methods.

Figure 7: Recovery-path breakdown quantifies the contributions of KV-reuse and speculation assistance in LUMEN’s overall performance.

Performance benefits are robust to variation in checkpoint placement parameters, speculative depth ($K$), and persist across increasing cluster size and higher failure rates.

Figure 8: Impact of request rate underlines LUMEN’s gains under both low and high concurrency workloads.

Figure 9: As the number of failures grows, LUMEN’s advantage over traditional recovery approaches amplifies further.

Figure 10: Simulator studies confirm LUMEN’s scalability and latency gains across a range of request rates.

Figure 11: Impact of number of failed workers highlights cluster-wide resilience and latency containment under multi-worker failures.

Figure 12: Cluster scaling study shows stable recovery latency gaps and highlights LUMEN’s orchestration at scale.

Figure 13: With single-worker failures in large clusters, LUMEN maintains low recovery overhead and swiftly returns to steady-state.

Implications and Future Directions

LUMEN effectively decouples recovery resilience from static checkpoint placement or rigid recovery paths, instead leveraging dynamic, load-aware coordination to deliver low recovery latencies, balanced resource utilization, and rapid restoration of service capacity. The theoretical advances—specifically, tightly integrated speculative assistance and flexible, load-proportional checkpoint placement—indicate important future directions for managing failures in ever-larger distributed LLM deployments.

Open research areas include: replication for multi-checkpoint fault tolerance (enabling resilience against correlated or cascading holder failures); extensions to dynamic scaling and heterogeneous hardware contexts; and integration with advanced scheduling primitives and request migration strategies. Deployments targeting mission-critical inference workloads can leverage LUMEN’s mechanisms for stronger SLAs under arbitrary failure patterns.

Conclusion

LUMEN establishes a principled framework for minimizing recovery cost in distributed LLM serving. By fusing load-aware checkpoint placement, adaptive recovery scheduling, and overlapping speculative assistance, LUMEN achieves substantial reductions in TTFT, TPOT, and recovery time—without incurring measurable steady-state overhead—relative to canonical failure recovery strategies, across a wide regime of practical failure and workload scenarios (2606.17787).