GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures

Abstract: GPU training jobs fail often, roughly two in five on large production clusters, yet the operator typically learns of a failure only by reconnecting hours later. Experiment trackers require editing the training script and maintaining a cloud connection; the scheduler's mail hook delivers a single status line with no cause and no logs. GPUAlert is a command-line wrapper that monitors any training command at the process boundary, and with no change to that command, emails a structured notification on completion carrying a classified failure cause, durable logs, and output artifacts. The tool is organized around three reliability primitives: a pre-launch log guarantee that establishes the durable destination before the child process can crash, notifier isolation that makes the wrapper's exit code a pure function of the child's status regardless of whether the email succeeds, and a non-silent artifact budget that bounds attachment size without ever dropping output silently. We release a labelled corpus of 474 GPU training logs across 15 failure classes and a reproducible evaluation harness. On the twelve hardware-reproduced classes, the ordered-rule classifier reaches 0.997 macro-F1, against 0.830 for unordered keyword matching and 0.133 for exit-code inspection. Wrapper overhead is a constant approximately 3ms per job; the pre-launch guarantee preserves a log where a shell redirect yields nothing; and across all 15 failure modes the wrapper returns the child's exit code unchanged even when the SMTP relay is unreachable.

• The paper presents GPUAlert, a process-boundary monitor that requires zero instrumentation to diagnose GPU training-job failures due to hardware and software faults.
• Its methodology employs pre-launch log guarantees, notifier isolation, and a regex-based classifier, achieving Macro-F1 scores above 0.997.
• Empirical evaluation shows GPUAlert reduces wasted GPU-hours with near-perfect accuracy and negligible overhead, enhancing operational efficiency.

GPUAlert: A Zero-Instrumentation Monitor for GPU Training-Job Failure Diagnosis

Motivation and Problem Statement

Large-scale ML workloads executed on GPU clusters exhibit failure rates where approximately 40% of training jobs terminate prematurely, predominantly due to hardware and system faults such as CUDA errors, ECC events, and NCCL failures. Latency in failure detection leads directly to significant wasted GPU-hours, as jobs may remain idle for hours after failure before detection. Existing solutions are either intrusive—requiring code instrumentation and external connectivity (e.g., MLflow, Weights & Biases)—or provide insufficient diagnostic context (e.g., scheduler mail hooks delivering only exit status).

GPUAlert is presented as a process-boundary monitoring solution, requiring zero instrumentation, fully decoupled from the workload code, for rapid, automated notification and structured diagnosis of GPU training-job failures. It targets scenarios where experiment scripts are inherited, clusters are air-gapped, or administrative privilege for daemons is unavailable.

System Design

The system wraps any user-specified command as a child process without source modification or dependency injection. The design is predicated upon three reliability primitives:

• Pre-launch Log Guarantee: Before the target process is spawned, log files for stdout, stderr, and combined output are touched on disk. This ensures diagnostics survive even if the child fails catastrophically at launch, sidestepping the classic race with shell-based redirectors.
• Notifier Isolation: Notification handling is robust to channel failures (e.g., email server unreachable) as exceptions are caught and logged locally, never polluting the exit code or process identity of the wrapped job.
• Non-silent Artifact Budget: Output artifacts are transmitted up to a strongly bounded per-file size, with any excess logged (never dropped without trace), ensuring graceful degradation under heavy output.

The wrapper enforces strict equivalence: the child runs as if unmonitored, with the return code always that of the underlying job, even on SMTP or notification failures.

Failure Mode Classification

GPUAlert uses a priority-ordered, regex-based classifier over logs, spanning 15 failure modes. Specific errors (CUDA OOM, NCCL errors, Python assertion failures, OOM-killer events) are matched before generic runtime or traceback rules. This schema supports deterministic, debuggable, dependency-free operation, providing both a structured cause and actionable remediation hints alongside attached logs and artifacts. The classifier was constructed and prioritized a priori and evaluated on an open corpus.

Empirical Evaluation

The evaluation is conducted on 474 logs across 15 failure classes, using both fault injection and public issue tracker logs. Twelve classes are hardware-reproduced; three synthetic classes remain due to limitations of the hardware environment.

Classification Performance:

• Macro-F1: 0.997 on hardware-realized classes; 0.998 including synthetic.
• Baselines: Naive grep reaches 0.830 F1, traceback parsing 0.559, and exit code inspection only 0.133.
• Wild Logs: On 24 public, held-out logs, GPUAlert achieves 95.8% accuracy, compared to 70.8% for grep.

  Figure 1: GPUAlert confusion matrix over the 474-log corpus; the only error is a boundary case between assertion and generic traceback.

The off-diagonal entry—an assertion classified only by a generic traceback—is the sole adversarial case, underscoring the effectiveness of rule prioritization.

Reliability and Overhead:

• Pre-launch Guarantee: Ensures log existence in all exec-failure scenarios, addressing a failure mode unhandled by shell-based methods (Fisher exact, $p=1.45\times 10^{-11}$).
• Notifier Isolation: SMTP failures never alter the wrapped job’s exit code in 15/15 modes tested.
• Overhead: The process adds a constant $2.6$–$3.8$ ms per job, negligible for any realistic training workload.

Artifact Handling: Artifacts >25 MB are never silently dropped; instead, their omission and location are enumerated for operator retrieval.

Comparative Positioning

GPUAlert is unique among per-job monitoring solutions, offering zero code modification with high-fidelity, structured failure cause classification, durable logging, and robust process equivalence. Cluster-level monitoring tools such as NVIDIA DCGM and Elasticsearch/Loki are complementary, providing device-level visibility but not per-job causal analysis. Decorator packages (e.g., knockknock) require code changes and do not address log durability or exit-code preservation. Experiment trackers target instrumented pipelines with persistent outbound connectivity.

Limitations and Future Work

Hardware Coverage: The evaluation is restricted to a V100 environment due to resource constraints, with three classes using synthetic signals. Generalization to A100/H100 and broader software stack diversity must be empirically established.

Surface of Failures: Regex-based classification, while auditable and robust, defaults to generic traceback matching on uncaptured/new failure signatures. Non-canonical errors (e.g., custom frameworks or novel hardware) may require future corpus extension and, potentially, learned classifiers.

Multi-Node Training: GPUAlert as presented monitors a single process boundary, and is thus insufficient for distributed job execution (e.g., torchrun, MPI-based multi-rank jobs), wherein failures may be nonlocal or silent. Extensions aggregating collective statuses or interfacing with launchers are necessary for comprehensive distributed-job diagnostics.

Live Anomaly Detection: The system currently triggers on process exit. Incorporating in-flight anomaly detection—e.g., stalled metric lines or loss plateaus—would further shrink dead-running times and reclaim wasted resources.

Delivery Channels: Email is the only notification channel supported. Integrating chat (Slack, Teams), HTTP endpoints, and scheduler-epilog invocation would increase applicability in heterogeneous environments.

Practical and Theoretical Implications

Prompt and automated failure classification with robust artifact delivery directly minimizes wasted GPU time and aids fast remediation. At scale, this contributes to operational efficiency and energy savings, underscoring the environmental footprint of ML training. The process-boundary monitoring paradigm establishes a practical baseline for reproducible, non-intrusive observability in unmodified workloads, and the open corpus fosters community-driven expansion in capabilities.

Conclusion

GPUAlert exemplifies a principled middle ground between intrusive instrumentation and low-information scheduler mail hooks for GPU job failure diagnosis. It delivers robust, structured failure notification without modifying training scripts or requiring persistent connectivity, achieving near-perfect classification accuracy with negligible overhead on representative hardware. Its process-boundary primitives—pre-launch log guarantee, notifier isolation, and artifact bounding—are individually standard but, in aggregate, guarantee strong correctness and usability at the per-job level. The limitations pertain primarily to hardware diversity and distributed executions, both of which are well-scoped targets for future development.

In environments with high job failure rates and operational latency sensitivity in ML workflows, GPUAlert provides a provably reliable, zero-instrumentation diagnostic framework that bridges a critical observability gap.