Determine the cause of large-state A2 in-memory-replication deadlock

Determine whether the deadlock of FT2 in-memory replication for the A2 Ring All-Reduce architecture with approximately 2.6 GB of per-rank state is caused by interference between the peer-replication subgroup and the DDP process group or by a memory-pressure stall.

Background

The authors evaluate FT2 in-memory replication on the A2 Ring All-Reduce architecture with a large per-rank state matching the scale used by the pipeline, tensor-parallel, and hybrid workloads. All three FT2 runs terminate on the NCCL watchdog before fault injection because training deadlocks during the normal execution phase.

The paper offers two possible explanations—interference between FT2’s peer-replication subgroup and the DDP process group, or a memory-pressure stall—but does not isolate the root cause. Resolving this issue is necessary to determine whether the failure reflects a limitation of the FT2 mechanism, the process-group interaction, or resource pressure at large state sizes.

References

This remains a hypothesis, since the root cause (subgroup participation vs. memory-pressure stall) is not isolated.

— FailBench: Evaluating Fault Tolerance Across Distributed Training Architectures  (2610.07688 - Aljbab et al., 6 Oct 2026) in Section 5, RQ3, subsection “State-size control: A2 with a 2.6 GB workload”