Fault Tolerant Hybrid-Shared Data Parallelism

• FT-HSDP is a robust method that combines hybrid data-parallel and model-parallel techniques to maintain training progress during GPU/node failures.
• It leverages layered in-memory checkpointing, adaptive collective communications, and redundant shard placement to minimize rollback and recomputation.
• FT-HSDP's fine-grained recovery protocols and fault-containment strategies enable high availability and scalable performance on systems with 100,000+ GPUs.

Fault Tolerant Hybrid-Shared Data Parallelism (FT-HSDP) is a family of system and algorithmic designs for large-scale distributed machine learning and linear algebra that enable robust, high-efficiency training in environments where GPU or node failures are frequent. Combining hybrid data-parallel and model-parallel (tensor/pipeline/etc.) parallelism with fine-grained fault-containment domains, FT-HSDP permits continued training progress with minimized rollback and recomputation, attaining high availability and notable fault resilience at scales of up to 100,000+ GPUs. Core contributions include layered in-memory checkpointing, redundant shard placement, adaptive collective communications, and protocols that reconcile stochastic and numerical equivalence in the face of process loss. The approach is extensible, with implementations spanning deep learning (PyTorch+TorchElastic, FSDP, hybrid-shard) and Krylov subspace methods in high-performance computing.

1. System Model and Topology

FT-HSDP organizes P GPUs (with $P = N \times K$ for N GPUs per node and K nodes) into a three-dimensional parallel structure. Key axes:

• Data Parallel (DP): D groups ($G_{\mathrm{dp}} = P/D$ per group), sharding samples.
• Model Parallel (MP): M groups ($G_{\mathrm{mp}} = P/M$ per group), sharding model tensors via tensor/pipeline/expert parallelism.
• Hybrid Structure: DP typically crosses nodes, MP within nodes (or vice versa), enabling flexible mapping of communication patterns to interconnect locality.

For parameter tensors $\theta_\ell$ (layer $\ell$) of total size $|\theta|$, each $\theta_\ell$ is further partitioned over the DP axis into S shards per group, $\theta_\ell^s$ (with $|\theta_\ell^s| \approx |\theta_\ell|/S$). This sharded representation is foundational for both checkpointing and redundancy.

2. Fault Containment and Collective Protocols

Fundamental to FT-HSDP is defining replica-level fault domains. Each data-parallel group (replica) acts as the smallest unit of isolation; if a GPU/node in replica $r$ fails, only that replica is paused, rebuilt, and rejoined, while others proceed without global interruption (Salpekar et al., 30 Jan 2026).

Collective synchronization—particularly for cross-replica gradient all-reduce—is achieved via the Fault Tolerant All-Reduce (FTAR) protocol. FTAR splits responsibilities: the CPU orchestrates group membership and progression (including dynamic add/removal of replicas via a consensus service), while the GPU executes data transfer and reduction. This separation allows real-time recovery from replica loss without global NCCL reinitialization, reducing reconfiguration overhead to less than one second for cluster sizes up to 98,000 GPUs (Salpekar et al., 30 Jan 2026).

The FTAR protocol is pipelined and robust to mid-step failures, with ring algorithms implemented via reduce-scatter and all-gather kernels. When a replica rejoins after recovery, it leverages a non-blocking catch-up protocol: tardy replicas contribute zero-gradient vectors, enabling synchronous state advancement without significant stalls. Average stall from a single replica failure and recovery drops to around 3 minutes (from 10 minutes or more for global synchronized recovery), boosting effective cluster utilization from 44% to ~80% (Salpekar et al., 30 Jan 2026).

3. In-Memory Checkpointing and Redundancy

FT-HSDP introduces a hierarchical, asynchronous checkpointing mechanism to enable fine-grained, low-overhead state persistence, crucial for rapid failure recovery (Wang et al., 2023):

• Level 1: Each GPU copies shards ($G_{\mathrm{dp}} = P/D$0) from device memory to a small pinned CPU buffer (bucketed to size $G_{\mathrm{dp}} = P/D$1).
• Level 2: Buffers are asynchronously flushed to node-shared memory.
• Level 3: (Periodically) Parity shards are generated via erasure coding (e.g., RAID-5 style: $G_{\mathrm{dp}} = P/D$2) and either stored locally or persisted to NFS/disk as a checkpoint.

Overlapping these stages with computation yields snapshot overhead that is often fully hidden ($G_{\mathrm{dp}} = P/D$3), and practical snapshot throughput of 70 GB/s (compared to 0.5 GB/s for baseline DP checkpointing) (Wang et al., 2023). Total in-CPU memory overhead for checkpointing is $G_{\mathrm{dp}} = P/D$4, accounting for primary, parity, and double buffering.

When a node fails, recovery proceeds via local or parity decoding without resort to global storage:

$G_{\mathrm{dp}} = P/D$5

End-to-end restart (Llama-2-34B, 136 GB, 256 MI250X nodes) completes in under 60 s, compared to over 3 minutes for standard disk-based reload (Wang et al., 2023).

4. Redundancy Layouts and Adaptive Reordering

At exascale, naive $G_{\mathrm{dp}} = P/D$6-fold replication for failure tolerance is untenable. Advanced FT-HSDP lineages (e.g., SPARe (Lee et al., 27 Feb 2026)) introduce stacked data-parallel redundancy and adaptive reordering to mask failures at near-constant compute cost.

Each group holds $G_{\mathrm{dp}} = P/D$7 data shards in a cyclic Golomb-ruler placement, guaranteeing every shard is replicated $G_{\mathrm{dp}} = P/D$8 times and overlaps are minimal, enabling tolerance to $G_{\mathrm{dp}} = P/D$9 failures per shard type (Lee et al., 27 Feb 2026). Upon node failure, the ReCtlr reorders the execution stacks via bipartite matching (Hopcroft-Karp) and min-cost-max-flow, ensuring one gradient from every data type is always available for global reduction—no global restart required unless an entire shard type is lost.

Key theoretical results:

• Max endurable failures: $G_{\mathrm{mp}} = P/M$0
• Mean compute overhead per step: $G_{\mathrm{mp}} = P/M$1–2.8 (even for large $G_{\mathrm{mp}} = P/M$2), vs $G_{\mathrm{mp}} = P/M$3 for traditional replication.
• Optimized redundancy: $G_{\mathrm{mp}} = P/M$4

SimGrid-based evaluation (600k H100 GPUs) demonstrates SPARe+CKPT delivers 40–52% lower time-to-train than replication+CKPT (J = 2.34–2.92 vs 3.88–6.07) (Lee et al., 27 Feb 2026).

5. Fine-Grained Forward Recovery and Trajectory Preservation

ReCoVer (Liu et al., 11 May 2026) generalizes FT-HSDP to enforce stochastic gradient equivalence in the presence of sustained failures, upholding the invariant $G_{\mathrm{mp}} = P/M$5 microbatches per step. This is achieved via three protocol layers:

1. Fault-tolerant collectives (ULFM-based): Process group all-reduce and consensus operations that shrink communicator membership and promote spares without indefinite blocking.
2. In-step fine-grained recovery: Snapshot and restore individual gradient buckets tagged by world-epoch, rewinding only buckets affected by group loss and classifying work as stale or safe for reduction.
3. Versatile workload policy: Dynamically adjust per-replica microbatch assignments (majors, minors, spares), redistributing quotas as faults eliminate workers while always summing to the invariant batch $G_{\mathrm{mp}} = P/M$6.

This protocol recovers from up to 256 losses (injected across a 512-GPU run) with no visible divergence in loss trajectory compared to failure-free training. Under sustained failures, effective throughput is $G_{\mathrm{mp}} = P/M$7 higher than best-tuned checkpoint-and-restart, with 74.9% more tokens processed at 234 GPU-hours. The recovery cost is constant in $G_{\mathrm{mp}} = P/M$8, whereas checkpoint costs scale linearly. Limitations include dependence on OpenMPI overheads and the lack of dynamic rejoin for new GPUs (Liu et al., 11 May 2026).

6. Early Numerical Hashing: Selective Reliability in Scientific Computing

FT-HSDP-style selective reliability strategies originated in scientific computing, notably in fault-tolerant Krylov solvers (Bridges et al., 2012). Software and kernel layers permit data regions (e.g., sparse matrix $G_{\mathrm{mp}} = P/M$9, preconditioner) to be explicitly marked as "failable." OS-level handlers intercept uncorrectable memory faults, replace pages, and delegate local repair or restoration from per-rank checkpoints.

The solver structure uses a reliable outer iteration and a fault-permissive inner loop (e.g., FGMRES). Upon detection of a corrupted chunk, the relevant region is restored before the next outer phase. This codesign enables graceful degradation: for high injected fault rates (up to 50%), outer-iteration counts increased only 1.4×, with convergence preserved. Performance overheads remain under 5%, with negligible scaling impact. The trichotomy property of FGMRES ensures either convergence, stalling, or clear failure indication, never silent error propagation (Bridges et al., 2012).

7. Limitations and Comparative Analysis

FT-HSDP schemes achieve substantial efficiency gains and failure resilience, yet several practical challenges remain:

• Timeout-based failure detection (typ. 60 s) can be further reduced via DSM-style mechanisms (Salpekar et al., 30 Jan 2026).
• Initial step after recovery ("first-step effect") can be a source of extra stall (45–100 s).
• Traditional checkpointing (NFS/disk) is a bottleneck at scale; peer-to-peer sharded fetch and erasure-coded deduplication are under active investigation.
• Existing protocols generally discard lost replicas' shards instead of reclaiming partial data.
• Dynamic join for fresh GPUs remains future work for most FT-HSDP implementations (Liu et al., 11 May 2026).
• Async collectives (FTAR, ULFM) introduce manageable but nonzero communication overheads compared to pure NCCL.

Comparatively, FT-HSDP outperforms standard checkpoint-and-restart schemes, drop-and-go/elastic frameworks, and rigid global synchronization in both sustained throughput and fidelity to the original optimization trajectory (Liu et al., 11 May 2026). Advances such as SPARe further compress redundant compute without sacrificing availability, while selective reliability approaches in linear solvers enable robust computation in environments where all-reliable hardware is cost prohibitive.

---

References

Reference Framework	arXiv id	Distinguishing Methods
Fault-Tolerant Checkpointing	(Wang et al., 2023)	Hierarchical snapshotting, intra-node RAID-5 parity, distributed in-memory restart
FTAR/Hybrid Fault Domain	(Salpekar et al., 30 Jan 2026)	Replica-level isolation, CPU-GPU split collectives, non-blocking zero-gradient catch-up
SPARe Redundancy/Stacking	(Lee et al., 27 Feb 2026)	Golomb-ruler stacked shards, adaptive matching, minimax redundancy overhead
Selective Reliability (HPC)	(Bridges et al., 2012)	"Failable" memory, fine-grained repair, FGMRES trichotomy, inexact Krylov methods
Forward Trajectory Recovery	(Liu et al., 11 May 2026)	Constant-batch, step-consistent microbatching, in-step granular rollback, ULFM collectives