- The paper introduces triangulated monitoring across quality, suitability, efficiency, and within-run, cross-run, and structural scopes, using variance and z-score thresholds to diagnose immature agentic systems.
- The experiments show that deterministic integration defects can completely mask task-level errors, while monitor-scope coefficients of variation ranging from 0.00 to 1.25 distinguish failure patterns without ground-truth labels.
- The paper demonstrates that severity-based FMEA triage reduced analyst review volume by 43×, or up to 445× after excluding a simulator artifact, and proposes maturity stages from structural diagnosis to reliability tracking.
Motivation and problem statement
This paper addresses a gap between how agentic AI systems are validated before deployment and what actually fails once they operate in production. Pre-deployment validation relies on capability benchmarks such as SWE-bench and AgentBench, which answer whether a model can act as an agent, but not whether a deployed system remains reliable. The authors argue that most agentic systems entering production—particularly in regulated industries such as audit, finance, healthcare, and legal services—are partially integrated assemblies whose failure landscape is dominated by structural integration defects rather than task-level errors. Under these conditions, they contend, task-level error detection may be infeasible because structural defects mask the very signal that task-level monitors are designed to detect.
The paper also identifies a systematic bias in conventional mean-based monitoring: aggregate metrics disproportionately reflect performance on frequent, easy cases while obscuring rare, high-severity failures where operational risk concentrates. A system reporting 87% mean accuracy may be reliable on routine tasks yet fail unpredictably on the complex cases that carry the highest risk. The work builds on the observation that compressing agent behavior into a single success metric obscures operational properties across consistency, robustness, predictability, and safety dimensions (Rabanser et al., 18 Feb 2026).
Triangulated monitoring methodology
The methodology decomposes evaluation along two axes. The first axis comprises three evaluative dimensions: quality (correctness, completeness, grounding), suitability (instruction adherence, contextual fit, workflow alignment), and efficiency (completion time, computational cost, human review burden). These dimensions are deliberately non-redundant—an agent can be accurate but unsuitable, suitable but inefficient, or efficient but low-quality—and each is operationalized through rule-based validators, statistical measures, and LLM-based evaluators.
The second axis comprises three monitoring scopes: within-run (documents within a single execution as the reference cohort), cross-run (the fleet of completed runs, using leave-one-out z-score estimation to prevent an anomalous run from inflating its own reference distribution), and structural (invariants of the integration architecture itself). Flagging uses z-score thresholds of τ=2.0 for most evaluators and τ=3.0 for right-skewed timing metrics, with evaluator-specific flagging direction: quality monitors flag only downward outliers, efficiency monitors only upward outliers.
The central methodological device is treating within-category variance as a first-class signal alongside means. In mature systems, variance detects unreliable behavior that means obscure; in immature systems—the regime studied here—it characterizes the monitoring architecture itself, distinguishing monitor scope classes without ground truth labels. This dual role connects to reliability engineering's maturity concept, where immature systems exhibit high variance that decreases as corrective actions accumulate [Rausand & Høyland 2004].
FMEA-based automated triage
Detection alone consumes oversight resources indiscriminately. The paper adapts Failure Mode and Effects Analysis from safety-critical industries, applying it not as a design-time risk identification tool—as prior ML-FMEA work does—but as a runtime triage mechanism over monitoring output. Triage proceeds through four phases: failure classification from structured monitoring metadata; risk classification across severity, probability, and exposure; prioritization; and response determination.
Severity follows the four-level MIL-STD-882E scheme: L1 (Catastrophic) halts processing; L2 (Critical) requires human review before downstream reliance; L3 (Marginal) receives periodic sampling; L4 (Negligible) is handled automatically. This calibration aligns with NIST's AI Risk Management Framework's requirement for impact-proportionate oversight. Notably, routing in this experiment operates on Severity alone; the full Severity–Probability–Exposure cube is deferred because Probability and Exposure require deployment context—a multi-version fleet and component-attributable findings—that a single-system synthetic testbed cannot provide.
Experimental setup
Validation uses a synthetic testbed generated with Simthetic for the Subsequent Events and Unrecorded Liabilities (SURL) audit procedure under US GAAP: 120 document bundles of ten interrelated financial documents each, split into 20 clean baselines and 100 error bundles following a factorial design of 5 error subtypes × 4 difficulty levels × 5 document sets per cell. Ground truth is documented per bundle and independently validated. Synthetic generation is a prerequisite for the paper's central masking claim, which is only demonstrable with known ground truth.
The agentic system under test is deliberately early-stage, carrying two known integration defects: document type identified at ingestion but not propagated downstream (so a hardcoded invoice schema is applied to all documents), and a cross-document validator receiving zero related-document context. The system consequently produces no actionable error-detection signal at all—the zero-signal baseline the methodology targets. All evaluators used are rule-based or statistical; no LLM judges are deployed, a deliberate choice to avoid confounding evaluator variance with the monitor firing variance under measurement, though this departs from typical production configurations.
Results
Three findings emerge from 220 runs producing 10,453 total findings.
Monitor scope determines failure type. Within-run monitors fire on every run with high volume and near-zero variance (DM-095: 45.7 findings/run ±1.0, CV = 0.02, all L3), diagnosing the deterministic schema-misapplication defect. Cross-run monitors fire on only 41% of runs with high variability (CV = 1.25) and 24% of findings at severity L2, surfacing stochastic consequences of the integration gaps. The structural monitor fires exactly once per run on every run with CV = 0.00, all L2, identifying a manifest-to-coverage integrity gap. These CV values (0.00–1.25) characterize scope class without ground truth labels. A scope-unaware system would bury the 23 cross-run L2 findings under more than 10,000 within-run L3 findings.
Structural defects mask task-level errors. The hypothesis that variance would distinguish error-containing runs from clean baselines was not supported: across 100 paired companies, clean and error conditions are statistically indistinguishable. When a stage applies the wrong schema to every document, its findings are identical whether or not the data contains an anomaly—an instance of evaluation entanglement. The authors state this masking result plainly as a null result and scope it carefully: it holds for deterministic, content-independent structural defects; conditionally firing defects would produce partial masking instead. Characterizing the full taxonomy of defect types and their masking profiles remains open.
Deterministic triage achieves substantial review reduction. All 10,210 L3 findings route to automated monitoring; all 243 L2 findings route to human investigation—a 43× reduction in analyst volume. The authors note a caveat that strengthens rather than weakens the claim: 220 of those 243 L2 findings originate from DM-100, which fires on a simulator artifact (macOS resource fork files inflating the expected document count) rather than a genuine system defect. Excluding it, the reduction is 445×, carried entirely by cross-run findings, so the triage value does not depend on the structural monitor.
| Scope |
Firing |
Findings/run |
CV |
Severity |
| Within-run |
100% |
45.7 ± 1.0 |
0.02 |
L3 |
| Cross-run |
41% |
0.44 ± 0.55 |
1.25 |
76% L3 / 24% L2 |
| Structural |
100% |
1.0 ± 0.0 |
0.00 |
L2 |
Maturity staging model
From these Stage-1 results the authors propose a three-stage maturity model: Stage 1, system characterization, where monitoring produces a diagnostic map of broken stages and missing connections (validated here); Stage 2, error detection, once integration defects are repaired and monitors transition to detecting output errors; and Stage 3, reliability tracking via failure frequency, MTTR, and fix effectiveness metrics. The staging is explicitly presented as a hypothesis: Stages 2 and 3 are theoretical extrapolations requiring longitudinal study of a system as defects are repaired, not observations. The practical implication is that monitoring infrastructure should be deployed before systems mature, since Stage-1 diagnosis identifies which integration points to fix first—organizations that defer monitoring miss the phase where it delivers the most immediate engineering value.
Limitations and open questions
The authors concede several constraints directly. The synthetic testbed cannot fully replicate real-world agent interactions. The error taxonomy spans a single audit procedure (SURL under US GAAP); broader taxonomies would strengthen generalization. Results reflect one agentic system with two specific integration defects, so replication across different defect profiles is needed. The maturity-staging model rests on Stage-1 data alone. Generalizability claims are limited to architectural transfer within document-driven, multi-stage regulated-industry workflows; specific firing profiles, CV ranges, and severity distributions are explicitly not claimed to generalize. The contribution is characterized as a design methodology rather than a framework—no reusable artifact is released, and evaluators, severity calibrations, and thresholds must be rebuilt per domain. Finally, the interaction between LLM-evaluator variance and monitor signal interpretation, deferred here to keep the experiment clean, is left unexamined.
Conclusion
This paper presents a triangulated monitoring and FMEA-based triage methodology for agentic systems operating below the maturity threshold at which error detection is viable. Its empirical contributions are a demonstrated mapping from monitor scope to failure type, quantified by coefficient of variation without ground truth; evidence that deterministic structural defects completely mask injected task-level errors; and a 43× (conservatively, up to 445×) reduction in human review volume through deterministic severity-based routing. The proposed maturity-staging model offers a hypothesis for how monitoring's role evolves from structural characterization to error detection to reliability tracking, with the transitions themselves awaiting longitudinal validation.
