Intermittent Job Failure Detection
- Intermittent job failure detection is a process that identifies non-deterministic, transient failures caused by environmental and infrastructural factors rather than code defects.
- It leverages diverse evidence such as logs, rerun metadata, resource telemetry, and statistical monitoring to predict and diagnose failures early in the job lifecycle.
- Techniques including proactive scheduling adjustments, runtime monitors, and sequential statistical tests have shown significant reductions in failures and improved system efficiency.
Intermittent job failure detection denotes the family of methods used to identify failures that arise irregularly, transiently, or under context-dependent execution conditions rather than as stable, immediately reproducible defects. Across CI/CD pipelines, Hadoop and cloud schedulers, HPC systems, GPU training workloads, and large distributed training jobs, the target may be a failed job that would later succeed on rerun under the same revision, a task predicted to fail before dispatch, a running job entering a short-lived degraded regime, or even a nominally successful job whose intended effects did not occur. The common objective is to reduce wasted reruns, wasted compute, delayed triage, and failure propagation through dependencies by using logs, rerun sequences, scheduler metadata, resource telemetry, or sequential monitoring statistics to detect instability as early and as reliably as possible (Aïdasso et al., 5 Jul 2025, Soualhia et al., 2015, Sokolov et al., 2022, Aïdasso et al., 17 Sep 2025).
1. Definitions and operational scope
In CI/CD research, intermittent failures are usually defined as non-deterministic failures caused by factors not primarily related to the submitted code. Reported causes include flaky tests, overloaded servers, out-of-memory conditions, networking problems, authentication or token issues, pod failures, and other infrastructure faults, whereas regular failures are code-related failures such as compilation errors, legitimate test failures, static-analysis violations, or bugs (Aïdasso et al., 5 Jul 2025). A closely related industrial term is flaky job failure, defined as a non-deterministic job failure caused by irregular issues, typically related to the CI environment rather than coding errors, such that the same job can pass or fail on rerun without any source-code or build-script changes (Aïdasso et al., 9 Jan 2025).
In scheduler-centric systems, the notion is broader and more operational. Hadoop work on ATLAS defines failures through task and job outcomes under dynamic cloud conditions, including TaskTracker or DataNode failures, node slowdown, network degradation or disconnection, resource depletion, loss of job data, and repeated task-attempt failures. The intermittent character there is tied to delayed visibility: a TaskTracker can fail right after a heartbeat, yet the JobTracker may not discover the failure until the next heartbeat interval; the paper cites a default interval of 10 minutes, so tasks assigned just after a heartbeat may fail and remain undetected for about 9 minutes (Soualhia et al., 2015).
A statistical-monitoring perspective formalizes the same phenomenon as a transient change. In that view, the system is nominally governed by a baseline law , temporarily shifts to an under-change law for an unknown finite duration , and then returns to . For job systems, the observations can be failure indicators, retry counts, queue delay, latency, exception rates, or resource-error counts. This framing treats intermittent failure episodes as temporary departures from normality that must be detected during the episode rather than eventually after it has ended (Sokolov et al., 2022).
Recent CI work also broadens the scope from visible failures to silent failures, where a job exits with code 0 and is marked successful but fails to complete all or part of its intended tasks. In that setting, reruns of successful jobs become a behavioral proxy for untrustworthy outcomes rather than for conventional fail-then-pass intermittence (Aïdasso et al., 17 Sep 2025).
2. Evidence sources and labeling regimes
Intermittent job failure detection depends first on how unreliable outcomes are observed and labeled. In CI/CD, the dominant weak label is rerun-based nondeterminism: if a job is rerun on the same commit and later succeeds, the original failed execution is treated as intermittent or flaky. GitHub Actions exposes rerun metadata via run_attempt, enabling reconstruction of rerun sequences at the build and job level. For detector construction, one study started from initially failed jobs, labeled jobs as flaky if a developer-triggered rerun later succeeded, and then used CI-Rerunner to perform up to 10 additional reruns under the same code version; this process identified 83 additional flaky jobs, and a manual audit of 50 jobs that still failed in all 10 reruns found 4 more flaky jobs (Ge et al., 2 Feb 2026).
Rerun-based labeling is practical but noisy. A manual analysis of 2,125 failed jobs from 5 industrial TELUS projects and 1 open-source project showed that the heuristic mislabels on average 31.97% of jobs, with project-level error rates from 16.18% to 50.14%. The core failure mode is systematic: intermittent failures are mislabeled as regular when no confirming rerun exists, so logs with similar intermittent patterns can appear in both classes (Aïdasso et al., 5 Jul 2025). This connects directly to a broader industrial observation: at TELUS, 25% of failed jobs are flaky, but of 7,763 flaky job failures identified from rerun behavior, 2,507 lacked logs and had to be excluded from diagnosis-oriented analysis (Aïdasso et al., 9 Jan 2025).
Different settings expose different evidence modalities.
| Context | Primary evidence | Typical target |
|---|---|---|
| CI/CD pipelines | Rerun sequences, job logs, project metadata | Flaky vs regular failure; failure category |
| Cloud, Hadoop, HPC schedulers | Task history, retries, priority, waiting/service time, resource usage | Pre-dispatch failure risk; runtime failure risk |
| Running GPU or distributed jobs | stdout, stderr, artifacts, peer-relative telemetry |
Terminated failure cause; faulty machine before halt |
| Sequential monitoring | Observation streams over time | Temporary failure episode |
Scheduler and trace-based studies usually define failure more broadly than CI rerun studies. One cloud-scheduling paper groups the problematic outcomes failed, killed, evicted, lost, and unscheduled, and even treats tasks or jobs with a dependent failed task or job as failed for modeling purposes (Soualhia et al., 2015). Process-boundary monitoring is still another regime: GPUAlert determines whether a job succeeded or failed when the wrapped child process ends, then classifies the captured combined.log into one of 15 ordered failure classes and attaches durable logs and artifacts to the notification (Agarwal et al., 1 Jul 2026).
Reruns of successful jobs introduce an additional labeling regime. In industrial GitLab CI, 11% of successful jobs were rerun, and 35% of these reruns occurred after more than 24 hours. Those successful rerun-initiating jobs are treated as evidence of silent failures: outcomes that looked green to the scheduler but were not trusted by developers (Aïdasso et al., 17 Sep 2025).
3. Scheduler- and workload-level predictive detection
A major line of work treats intermittent failure detection as a scheduling problem: predict whether a task or job will fail before or during execution, then change scheduler behavior accordingly. In Hadoop, ATLAS (“AdapTive faiLure-Aware Scheduler”) is implemented as a proactive layer on top of FIFO, Fair, and Capacity. It predicts whether a map or reduce task will succeed or fail if executed, then modifies dispatch by checking TaskTracker and DataNode availability, waiting for safer execution opportunities, requeuing with penalty after timeout, speculatively launching predicted-failure tasks on many nodes with enough resources, and lowering priority for repeatedly problematic tasks. ATLAS also adapts the heartbeat interval: if more than $1/3$ of TaskTrackers were failed between two heartbeats, the interval is decreased by half each time, down to a minimum of 2 minutes in the experiments. The strongest correlates of task outcome are the number of running, finished, and failed tasks on a TaskTracker, task locality, and the number of previous finished or failed attempts of the task. In Amazon EMR experiments, ATLAS reduced failed jobs by up to 28%, failed tasks by up to 39%, and total job execution time by 10 minutes on average while also reducing CPU, memory, and HDFS I/O (Soualhia et al., 2015).
A related Google-trace study models task and job outcomes from waiting time, service time, scheduling class, priority, requested and used CPU, RAM, and Disk, dependency-history counts, and reschedule events. Random Forest was the best task-level model, reaching accuracy 97.3%, precision 98.1%, and recall 97.7% in the 10-file setting. The extracted decision rule is strongly history-based: previous dependent killed, failed, and evicted tasks, together with low priority, are leading indicators of future failure. In simulation, predictive rescheduling improved the number of finished tasks by up to 40%, and in a Hadoop case study it reduced failed jobs by up to 45% with overhead less than 5 minutes. The same study reports that some tasks were resubmitted up to 182 times, underscoring the relevance of repeated scheduling instability to intermittent failure detection (Soualhia et al., 2015).
In production HPC-style environments, the same problem has been separated into queue-time and runtime prediction. On a production cluster at Texas Tech, a queue-time Random Forest achieved maximum precision 90.61%, whereas a runtime Random Forest achieved maximum precision 97.75% and F1 95.91%. Integrating the runtime model with the scheduler reduced CPU time and integral memory usage by up to 16.7% and 14.53%, respectively. The runtime model is especially pertinent to intermittent failures because it incorporates execution-dependent features such as cpu, mem, io, iow, maxvmem, and the engineered ratios and , allowing the detector to react to runs that become failure-prone only after launch (Li et al., 2023).
A further HPC study designed for online submit-time prediction uses only submission-time fields and compares integer encoding with Sentence-BERT representations of concatenated job records. In the online setting, the best model is SB+MWD, with macro F1 0.70 and failed-class F1 0.54, representing a 20% improvement over the best offline failed-job F1. The result is notable because it links temporally adaptive retraining with semantically similar command and job-name representations, which is a plausible substrate for detecting intermittent failure risk in recurring workload families even before execution starts (Antici et al., 2023).
4. Log-centric CI/CD detection and diagnosis
CI/CD research has increasingly shifted from scheduler metadata to log text, but the central dispute concerns label quality rather than model class. The strongest prior baseline in this literature is the Ubisoft method by Olewicki et al., which uses TF-IDF features of logs, additional job metrics, and a two-stage XGBoost detector trained on rerun-based labels. Later work shows that this baseline performs well only where the heuristic labels are relatively clean (Aïdasso et al., 5 Jul 2025).
Few-shot learning addresses that label-noise problem by replacing large weakly labeled corpora with a very small number of manually labeled examples. One study preprocesses full job execution logs with seven rule-based transformations, fine-tunes the small embedding model BAAI/bge-small-en-v1.5 using SetFit, and trains a scikit-learn Logistic Regression classifier on the resulting 768-dimensional embeddings. A shot means one manually labeled example per class; with 12 shots per class, or 24 total logs, the method reaches 70–88% F1-score across six projects, whereas the state-of-the-art baseline is ineffective in four projects with F1 between 34 and 52 (Aïdasso et al., 5 Jul 2025). The same study reports that performance stabilizes around 10–12 shots and that gains from 12 to 15 shots are not statistically significant.
GitHub Actions work reaches a similar conclusion from a different angle. After excluding approval-driven reruns, 52,610 rerun builds remained across 1,960 open-source Java projects, and 35,634 of them, or 67.73%, were flaky builds. The proposed detector, FlakeDetector, combines a log module based on SentenceTransformer all-MiniLM-L6-v2 and FAISS retrieval with a structured-feature module over developer, code-change, and project-context variables. Averaged over 10 repositories, the best model, SVM, reaches precision 0.7336, recall 0.9143, F1 0.8026, and improves F1 over the state-of-the-art baseline from 0.6668 to 0.8026, a relative gain of 20.3% (Ge et al., 2 Feb 2026).
Diagnosis has then been pushed downstream of binary detection. FlaXifyer addresses only already-identified intermittent failures and performs multiclass categorization over 13 priority categories using SetFit fine-tuning and Logistic Regression on 768-dimensional embeddings. With 12 labeled examples per category, it reaches 84.3% Macro F1 and 92.0% Top-2 accuracy on 2,458 intermittent job failures. Its interpretability companion, LogSift, recursively extracts prediction-preserving log segments, runs in 0.98 s mean execution time per log on CPU, reduces review effort by 74.4%, and surfaces at least relevant failure information in 87% of sampled cases (Aïdasso et al., 29 Jan 2026).
Industrial taxonomy studies show why that downstream layer matters. An analysis of 4,511 flaky job failures at TELUS identifies 46 categories and prioritizes 14 by Recency, Frequency, and Monetary measures. The top five categories—misconfigured_env_variable, docker_daemon_connection_failure, container_platform_auth_failure, job_execution_timeout, and image_not_found—cover 40.76% of labeled flaky failures, while container_registry_server_error is the most widespread, affecting 50 of 80 projects (Aïdasso et al., 9 Jan 2025). This suggests that binary intermittent detection is operationally incomplete without category-aware triage.
5. Runtime monitors and telemetry-based detectors
Some systems detect intermittent failures not from historical logs or submission metadata but from the running process boundary or from peer-relative telemetry. GPUAlert is a zero-instrumentation command-line wrapper for GPU training commands. It initializes stdout.log, stderr.log, and combined.log before launch, classifies the final combined output using an ordered first-match-wins rule set over 15 failure classes, gathers artifacts under a 25 MB per-file cap, and sends a structured email on job completion. Its reliability model is based on three primitives: the pre-launch log guarantee, notifier isolation, and the non-silent artifact budget. On a labeled corpus of 474 GPU training logs, the ordered-rule classifier reaches Macro-F1 and Real-12 ; wrapper overhead is approximately 3 ms per job, and the wrapper exit code remains equal to the child exit code even when the SMTP relay is unreachable (Agarwal et al., 1 Jul 2026). The method is deliberately exit-time only: it detects terminated failures, not live stalls that never end.
Minder addresses the complementary case in large distributed model training: pre-halt detection of the faulty machine. It relies on the observation that synchronized 3D-parallel training jobs should show similar second-level telemetry across machines, so one persistently dissimilar machine is likely faulty. The system aligns and normalizes metrics, trains a separate LSTM-VAE per metric, ranks metrics by fault sensitivity using per-window maximum Z-scores and a decision tree, computes pairwise Euclidean distances among per-machine embeddings, and requires the same machine to remain the top outlier for a continuity threshold of about 4 minutes. Deployed for over one year in production, Minder reacts within 3.6 seconds on average, with precision 0.904, recall 0.883, and F1-score 0.893. The paper notes that most abnormal patterns last for over five minutes before the entire training task comes to a halt (Deng et al., 2024).
These runtime detectors are complementary rather than interchangeable. GPUAlert preserves high-value evidence after process termination without modifying training code, whereas Minder looks for persistent but still-running divergence in one machine relative to its peers. A plausible implication is that intermittent job failure detection increasingly spans a continuum from post-termination diagnosis to early-warning telemetry analysis rather than a single binary classification stage.
6. Sequential statistical detection, limitations, and open problems
A separate theoretical tradition models intermittent failures as transient changes or intermittent multivariate faults. In sequential change detection, the core question is not average delay after a persistent change but the probability of alerting while a finite failure episode is still active. One paper therefore proposes controlling the local conditional probability of false alarm,
$\LCPFA_m(T)=\sup_{\ell \ge 0}\Pr_\infty(T \le \ell+m \mid T>\ell),$
and maximizing the local conditional probability of detection,
0
rather than optimizing classical average-run-length or expected-delay criteria. Under maximum-likelihood treatment of unknown transient duration, standard CUSUM, window-limited CUSUM, and FMA emerge as natural stopping rules; WL CUSUM is recommended when an upper bound on burst length is known, whereas modified FMA is strongest for very short, known-ish bursts (Sokolov et al., 2022).
Multivariate statistical process monitoring makes the same point for weak, brief, recurring faults. MA-1 control charts with multiple window lengths simultaneously track different intermittent-fault durations, then use duration constraints and cross-window overlap to suppress false alarms, recover missed alarms, and infer the appearing and disappearing time instances of the fault. The theory provides necessary and sufficient detectability conditions in terms of fault magnitude, active duration, and inactive duration, and simulation studies show that single-observation charts are inadequate for these settings (Zhao et al., 2020). A related extension to stationary autocorrelated observations replaces equal or exponential smoothing by an optimally weighted moving average. The resulting OWMA-2 control chart chooses the weight vector to maximize intermittent-fault detectability, proves the symmetry of the optimal weight vector, and proves that equal weighting is optimal only when data exhibit no autocorrelation (Zhao et al., 2020).
Across application domains, several limitations recur. Rerun heuristics are useful but systematically noisy when reruns are inconsistent or absent, and periodic retraining is often the only explicit response to drift; ATLAS, for example, retrains every 10 minutes but has no formal drift detector (Aïdasso et al., 5 Jul 2025, Soualhia et al., 2015). Cross-project transfer remains unstable in CI/CD, with one-to-one cross-project F1 ranging from 18% to 70% in a few-shot study, even though within-project performance is strong (Aïdasso et al., 5 Jul 2025). Runtime detectors impose their own assumptions: GPUAlert is restricted to failures that terminate the process and emit recognizable output, while Minder assumes peer machines in the same distributed job should be telemetry-similar at second-level granularity (Agarwal et al., 1 Jul 2026, Deng et al., 2024). Finally, binary intermittent-vs-regular detection is no longer sufficient on its own. Diagnosis taxonomies, multiclass categorization, and silent-failure detection show that the operational problem includes not only whether a job failed intermittently, but also what category of failure occurred, whether the job should be rerun or rerouted, and whether a nominal success can be trusted at all (Aïdasso et al., 29 Jan 2026, Aïdasso et al., 17 Sep 2025).
