Localization of Faulty GPU Components in Silent Data Corruption

Determine how to localize the faulty hardware components responsible for silent data corruptions in GPU-based AI computation, extending beyond checksum-based detection and recovery to identify the underlying device or component.

Background

Silent data corruptions in GPU tensor-core arithmetic can produce valid but incorrect integer results without triggering interrupts or conventional parity and ECC mechanisms. The paper explains that conventional checksum-based Algorithm Based Fault Tolerance provides only a binary detection verdict and does not identify the corrupted element, its magnitude, or the responsible device.

The paper presents SProbe as a mechanism that augments detection with error localization, exact magnitude recovery, and device-level telemetry. However, it explicitly identifies the broader problem of locating the faulty hardware component as an open problem previously recognized by the Open Compute Project, motivating SProbe’s diagnostic capabilities.