- The paper introduces ADA-ST, an adaptive fault-injection planner that models four infrastructure layers and reaches full fault-propagation edge coverage in 9–12 iterations, compared with static campaigns covering only 20–25%.
- The paper shows that severity-aware, coverage-guided testing prioritizes cross-layer incidents more efficiently, with one full-cascade test producing greater coverage gains than 233 static tests and physical validation requiring under 4% of static campaign effort.
- The paper presents FLAM, a functional abstraction model that transfers fault scenarios across hardware generations with 96–100% role-mapping fidelity while exposing blind spots such as missing telemetry, failed signal correlation, and unverified sensor trust.
The paper presents ADA-ST, an adaptive fault-injection planning methodology for validating multi-layer self-healing pipelines in GPU-accelerated AI training infrastructure, together with FLAM, a functional abstraction model that transfers fault scenarios across hardware generations (2607.16161). The work is grounded in production data from a hyperscale operator: Platform Alpha accumulated 72,550 repair tickets over four years, of which 49% involve cross-layer fault propagation. The central empirical claim is stark: static NPI test campaigns on three successive platforms cover only 20–25% of modeled fault-propagation edges, and the paper argues this is a structural plateau rather than a resource constraint.
Motivation and problem statement
The authors decompose self-healing AI infrastructure into four layers following the Avizienis et al. dependability taxonomy: L1 hardware, L2 firmware (BMC, GPU/NIC firmware), L3 management plane (health checkers), and L4 orchestration (auto-remediation, job scheduling). Static NPI validation campaigns exhibit three structural limitations: no formal coverage model, single-layer scope, and fixed scenario catalogs derived from component specifications. The consequence is quantified by the ticket corpus—cross-layer incidents are disproportionately expensive to resolve. Tickets spanning all four layers average 36.5 days to resolve, 6.6 times longer than single-layer tickets at 5.5 days. This resolution-time multiplier motivates testing cross-layer propagation paths before they manifest in production.
Methodology
Fault-propagation graph and coverage model
The infrastructure is modeled as a directed graph G=(V,E) where vertices are fault-observable subsystems across four layers and edges carry conditional propagation probability p and latency d. A six-class fault taxonomy (F1 latent hardware degradation through F6 topology-dependent failure) was derived via hierarchical clustering of repair-action co-occurrence vectors followed by expert consolidation, cross-referenced against Avizienis' taxonomy and IEC 60812 FMEA categories. Coverage is defined as the fraction of edges exercised by a campaign, complemented by a severity-weighted metric w(e)=f(e)⋅tˉ(e) combining observed ticket frequency and mean resolution time.
ADA-ST algorithm
ADA-ST operates in three phases. Historical priming initializes edge probabilities from tag co-occurrence in the incident corpus; a 72-hour temporal-window sensitivity check removes ~8% of co-occurrences without altering graph topology. Campaign planning generates $6|V|$ candidate scenarios (six fault classes per vertex) scored by w1Phist+w2Blast+w3(1−Cov) with weights 0.4/0.3/0.3, following IEC 60812 risk-priority-number philosophy. Runtime steering executes the top-scored scenario, updates coverage, adds emergent edges when observed propagation deviates from prediction, and spawns bounded exploratory variants ($2k$ per cascade). The paper proves deterministic finite-time completeness: if every edge has a covering scenario and budget suffices, coverage reaches 1.0 within ∣E∣ iterations for any weight vector with w3>0—a stronger guarantee than the probabilistic convergence of coverage-guided greybox fuzzing, owing to the finite candidate pool.
FLAM
FLAM decouples scenarios from concrete hardware via abstract functional roles R, platform-specific mappings p0, layer assignments, and fault-class associations. Scenarios transfer automatically when all roles map; unmapped roles—subsystems with no predecessor—are flagged as highest-risk test targets because they have zero historical coverage.
Evaluation design
A five-leg hybrid framework separates analytical validation from physical spot checks. Legs 1–4 are retrospective/prospective analyses on three anonymized platforms: Alpha (production, 22 vertices/34 data-derived edges), Beta (NPI, 26/29), Gamma (early NPI, 26/20). Leg 5 executes four scenarios on Gamma hardware. Effort comparisons use Scenario Effort Units (SEUs), normalizing single-layer checks against multi-layer cascading scenarios requiring integrated-stack access.
Results
Structural plateau of static campaigns
Static NPI plans achieve tightly banded edge coverage regardless of size: Alpha's 233-test plan covers 7/34 edges (20.6%), Beta's 459-test plan 7/29 (24.1%), Gamma's 384-test plan 5/20 (25.0%). All static tests concentrate on L1 hardware targets and L3 management-plane verification; zero test cases exercise L2 firmware behavior, L4 orchestration logic, or cross-layer cascades on any platform. That Alpha's four-year operational history yields no better coverage than Gamma's twelve months confirms saturation at roughly one-quarter of the graph—a structural limitation, not an under-resourcing artifact.
Adaptive coverage
ADA-ST reaches full edge coverage in 10 iterations on Alpha (all 27 blind-spot edges included), 12 on Beta, and 9 on Gamma. A single full-cascade scenario added to Alpha's static plan raises coverage from 20.6% to 41.2%—more incremental value than all 233 static tests combined. Monte Carlo comparison against random selection from the same candidate pool shows that even random cross-layer selection dramatically outperforms static testing (median 14 scenarios to full coverage on Beta vs. 459 static tests plateauing at 24.1%), while ADA-ST requires 14–18% fewer scenarios than the random median; the scoring function's principal additional value is severity-aware prioritization rather than iteration reduction. Sensitivity analysis over five weight vectors varies convergence by at most ±1 iteration, indicating robustness to tuning.
FLAM achieves 100% role-mapping from Alpha to Beta and 96% from Beta to Gamma (25 of 26 roles; one net-new Leak Sensor role plus one architectural migration). All unmapped roles reside in L1 Hardware, while L2–L4 roles remain stable across both transitions—an expected pattern given that management/orchestration software persists across GPU generation refreshes. The near-perfect transfer rates support the claim that the four-layer abstraction captures a genuine structural invariant, though the evidence base spans only two architecture lineages.
Physical spot validation
Four scenarios executed on Gamma hardware confirmed all predicted propagation edges at 14 SEU total—under 4% of the static campaign's effort—and surfaced three emergent behaviors not captured by the graph model:
- Absence-based blind spots: a CPLD fault causes downstream sensors to disappear rather than report bad values; the health checker validates only readable sensors, so telemetry removal goes undetected.
- Multi-signal correlation failure: simultaneous leak and thermal events produce two independent urgent-maintenance recommendations instead of one correlated cooling-system diagnosis, potentially doubling remediation effort on liquid-cooled platforms.
- Trust-without-verification: Sensor Monitor accepts corrupted BMC telemetry at face value without cross-referencing configured thresholds or plausibility, allowing a single fabricated register write to trigger a remediation cascade.
Limitations and open questions
The paper concedes several constraints candidly. The Leg 3 simulation uses production incidents as ground truth, introducing hindsight bias; mitigation rests on the prospective Beta/Gamma legs. Edge probabilities derive from tag co-occurrence, capturing correlation rather than causation, and represent lower bounds due to missing or delayed tags. Failure modes never observed in production—GPU HBM soft errors manifesting as silent data corruption, firmware race conditions during concurrent updates, correlated manufacturing defects—are absent from the graph; the emergent-discovery mechanism addresses this only during physical execution, not analytical simulation. Propagation timescales are not yet used in scenario design despite the latency attribute existing in the model. Physical execution cost scales linearly (~300 hours for a hypothetical 500-edge graph), which the authors argue remains feasible within multi-month NPI campaigns but acknowledge as an order-of-magnitude estimate. Finally, the four-layer decomposition is an engineering choice rather than a mathematical constraint, and the evaluation covers only three platforms across two lineages.
Conclusion
The paper demonstrates that static NPI validation structurally saturates at 20–25% cross-layer edge coverage regardless of test-plan size, and that coverage-guided adaptive steering closes this gap deterministically within 9–12 iterations while transferring across platform generations at 96–100% fidelity. The strongest evidentiary contribution is the combination of a large production corpus (72,550 tickets) with physical confirmation of predicted blind-spot edges and discovery of qualitative failure behaviors—telemetry absence, correlation gaps, unverified trust—that motivate taxonomy extensions. Open questions include whether the methodology retains its advantages on substantially larger graphs, whether absence-based and correlation-gap failure classes generalize beyond liquid-cooled platforms, and how the approach performs when integrated into live NPI tooling rather than evaluated retrospectively.