PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint

Abstract: State-of-the-art LLM training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware stack. Existing fault-tolerance mechanisms either impose non-trivial overhead during failure-free execution or suffer from prolonged recovery latency, particularly under scenarios where a small subset of compute nodes experience permanent failures. %The tradeoff between failure-free overhead and recovery latency forms a space forms a Pareto frontier We present PHOENIX to simultaneously address both optimization objectives. PHOENIX incorporates a fault-tolerance mechanism that restores LLM training via hot-swapping, namely by replacing failed nodes with spare nodes without terminating the complete job. The hot-swapping of PHOENIX is enabled by two ideas: First, it exploits an off-critical-path in-memory checkpointing mechanism for spatial redundancy. Second, it introduces a communicator reconstruction protocol that replaces failed nodes with spare nodes at runtime. PHOENIX efficiently overlaps the in-memory checkpointing with computation, thus introducing zero overhead during error-free execution. Upon permanent node failures, PHOENIX can rebuild memory states with minimal recomputation by leveraging in-memory checkpoints. We evaluate PHOENIX across scales (up to 512 NVIDIA A100 GPUs) and LLMs (up to 65B parameters), and observe zero checkpoint overhead with hot-swapping recovery completing in under 40 seconds. These results show that PHOENIX simultaneously achieves both zero-overhead error-free execution and extremely low recovery cost.

• The paper introduces a hot-swapping fault tolerance mechanism that eliminates full-job restarts through zero-overhead, in-memory checkpointing.
• It integrates asynchronous optimizer shard replication with online communicator reconfiguration to achieve recovery latencies under 40 seconds.
• Empirical results on supercomputers demonstrate scalability and an order of magnitude reduction in time-to-solution versus traditional checkpoint-restart methods.

PHOENIX: Zero-Overhead, Hot-Swapping Fault Tolerance for LLM Training

Motivation and Background

Distributed training of LLMs at scale—spanning tens of thousands of GPUs and running for weeks—faces significant fault-tolerance challenges due to the inevitability of node failures. Empirical data from 32K-GPU deployments shows frequent hardware and system-level interruptions, rendering failures an expected operational condition. Classical fault tolerance schemes, predominantly checkpoint-restart paradigms, impose either considerable runtime overhead from frequent checkpointing or high disruption (latency and replayed work) when failures occur, especially with persistent node loss. These schemes are further complicated by complex parallelism strategies (3D parallelism) used in state-of-the-art LLM training.

The "PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint" (2607.01646) paper introduces PHOENIX, a new fault-tolerance system for large-scale LLM training. The key contributions are a hot-swapping recovery mechanism that eliminates full-job restarts, zero-overhead in-memory per-iteration checkpointing, and a fast online communicator reconfiguration protocol. PHOENIX achieves recovery latency under 40 seconds, independent of model size, while introducing no measurable runtime overhead during failure-free execution.

Design Principles and System Architecture

PHOENIX is predicated on two central insights:

1. Selective State Replication: The only irreplaceable state in distributed LLM training is the optimizer shard owned by a failed rank. Model parameters are recoverable via redundant data parallelism. Thus, PHOENIX employs an asynchronous, per-iteration in-memory replication protocol: optimizer shards are sent to a data-parallel (DP) ring peer at every step, while the full state is locally snapshotted to host memory for rapid transient recovery.
2. Online Topology Repair: Node failures are handled as online topology repair rather than global restarts. PHOENIX reconstructs communicator groups at runtime, inserts spare nodes into the logical topology, and reassigns logical shards as needed, preserving all invariants required for correct distributed training.

   Figure 1: PHOENIX system overview. Normal path performs two-phase asynchronous offload and replication; failure path detects faults, swaps in a replacement node, and restores state from in-memory replicas.

   

   Figure 2: Asynchronous offload pipeline alongside compute: forward/backward computation and synchronization overlap with device-to-host (D2H) and host-to-host (H2H) replication.

Fault Model and Recovery Protocol

PHOENIX's error classifier distinguishes transient and permanent failures. For the latter, recovery executes in three orchestrated stages:

• Spare Node Attachment: Spares are pre-registered; selection and insertion occurs via the external control plane.
• Communicator Reconstruction: All involved nodes undergo a barrier, group assignments and communication topologies are deterministically rebuilt.
• State Restoration: Spare nodes reconstruct missing parameter shards via collectives and retrieve optimizer shards from assigned in-memory replicas (potentially multi-peer for higher resilience). Only the currently executing step may need replay; global replay is eliminated.

Correctness is maintained via four invariants (topology consistency, shard completeness, optimizer availability, and step atomicity). Multi-peer, topology-aware replication is supported for robustness under correlated failures (e.g., rack loss).

Figure 3: Ping-pong double buffering. D2H and H2H offloads interleaved with compute, ensuring immediate recoverability with no critical path stalls.

Implementation

PHOENIX is realized as a hybrid of Megatron-LM data-plane extensions and a fault-isolated external control-plane. Core design features:

• In-process checkpoint/replication wrappers: Leverage existing 3D-parallel sharding and optimization logic; opt-in extension when enabled.
• Control plane: TCPStore-managed distributed key-value store for recovery epochs, synchronization, and node metadata; persists independently from training processes.
• Transport: Asynchronous MPI-based, chunked replication and restoration for efficient overlap of offloads and recovery.

Experimental Evaluation

PHOENIX is empirically validated on both HPE Cray EX and NVIDIA Grace-Hopper (TACC Vista) supercomputers, up to 512 GPUs and 65B model size, using FP32 for maximal state volume.

Zero-Overhead Checkpointing

PHOENIX's per-iteration in-memory checkpointing is consistently measured to add negligible (statistically indistinguishable) overhead under weak and strong scaling, across parallelism topologies, model sizes, and hardware configurations.

Figure 4: Same-topology weak scaling (TP=4, PP=4, up to 64 nodes): baseline and PHOENIX step times are indistinguishable, confirming zero-overhead per-iteration checkpointing.

Figure 5: Step time across six 3D parallel topologies on 16 GPUs. PHOENIX closely tracks the baseline for all model sizes.

Figure 6: Step times on Vista (GH200/H200) for 7B, 21B, 65B models; PHOENIX matches baseline, confirming cross-architecture effectiveness.

Recovery Latency and Scalability

PHOENIX achieves recovery latencies 3.7× faster than baseline checkpoint-restart methods. Recovery time is invariant to model size and host memory pressure (including near full-GPU utilization), and increases only gradually with cluster scale due to collective communicator re-initialization.

Figure 7: Recovery latency breakdown vs. model size (fixed 128 GPUs). Recovery time is nearly constant across 0.6B–50B parameters.

Figure 8: Recovery latency and end-to-end time as a function of cluster scale for GPT-7B (fixed memory utilization). Communicator reconfiguration dominates scaling cost.

Communication Overlap and Micro-Breakdown

Detailed timing traces show that D2H and H2H replication are successfully overlapped with compute and communication stages. Even at smaller scale with high per-rank checkpoint size, PHOENIX offloads remain well within the available compute window.

Figure 9: Per-iteration time breakdown. D2H and H2H transfer occupy a small fraction of the compute window, supporting effective overlap.

Quantified Impact: Real-World Failure Model

Using failure rates observed in production-scale deployments (Meta's 100K-GPU jobs), PHOENIX reduces effective time-to-solution by an order of magnitude relative to conventional checkpoint-restart, as both replay and re-initialization stalls are virtually eliminated.

Theoretical and Practical Implications

PHOENIX shifts the dominant cost of resilience in LLM training from job-wide restart and checkpoint replay to a lightweight online topology repair mechanism. The core assumptions—selective optimizer shard replication and runtime communicator reconfiguration—are broadly applicable across distributed deep learning frameworks adopting sharded or replicated state. Notably, PHOENIX's zero-overhead guarantee is expected to persist or improve on next-generation architectures (e.g., NVIDIA Grace-Blackwell), given continued increases in PCIe/interconnect bandwidth relative to step compute cost.

PHOENIX's design provides a framework for future research in:

• Generalized, topology-aware replication for truly large-scale correlated failure domains.
• Extension to heterogeneous or elastic systems where resource variability mandates dynamic topology repair.
• Integration with cross-framework checkpointing ecosystems (e.g., [ByteCheckpoint], Universal Checkpointing [10.5555/3768039.3768129]).

Conclusion

PHOENIX (2607.01646) represents a major step in fault-tolerant distributed LLM training, demonstrating that asynchronous, in-memory optimizer replication combined with online hot-swapping can provide zero-overhead failure-free operation and drastic reductions in recovery time under realistic failure conditions. These results have broad implications for the scalability and efficiency of future LLM training runs, particularly at O(10^4–10^5) GPU scales, and offer a blueprint for adopting similar lightweight, topology-repair-based recovery methods in the design of next-generation distributed training systems.