- The paper derives a five-class, mechanism-oriented taxonomy from 22 production incidents over eight weeks, showing that failures often arise at system seams rather than within individual components.
- The study finds that human observation detected about 70% of incidents while 4,286 tests, 827 governance checks, and preflight monitoring remained largely green, highlighting the limits of automated controls against novel failures.
- The paper identifies fail-plausible failure, in which polluted inputs become fluent false narratives, and recommends provenance labeling, context hygiene, sabotage validation, read-only observers, and promoting fixes into mechanized scanners.
This paper presents a longitudinal empirical study of silent failures in a continuously operating LLM agent runtime, deriving a five-class, mechanism-oriented taxonomy from 22 fully documented production incidents collected over eight weeks (2606.14589). Its central contribution is the identification and characterization of fail-plausible failure: a mode in which an LLM does not merely fail to report an error but actively converts it into fluent, confident, false narrative delivered to the user. The study is positioned as the methodological complement of horizontal failure taxonomies such as MAST (Cemri et al., 17 Mar 2025) and provider-side inference incident analyses (2511.07424), trading breadth for the longitudinal texture—latency, discovery channels, defense evolution, recurrence—that only a continuously running system can exhibit.
System context and methodology
The subject system is a personal-assistant agent runtime in production since March 2026, connecting eight LLM providers to messaging channels via a three-plane architecture: a control plane (tool-governance proxy, declarative governance engine with 90 invariants and 827 checks, SLO monitoring, and a declared-state convergence engine), a capability plane (adapter with capability-scored fallback chains), and a memory plane (RAG-indexed knowledge base and daily LLM synthesis jobs that push digests to the user). The observation band beneath these planes—4,286 unit tests, daily governance audits, preflight checks, watchdogs, and a weekly human observation ritual—defines the study's selection criterion: an incident qualifies only if it crossed all planes and the observation band without raising an actionable signal.
The corpus comprises every incident between April 9 and June 2, 2026 that reached production, exhibited a silent phase, and was closed with a postmortem under a mandatory protocol requiring a full causal-chain diagram, a three-layer root cause (trigger, amplifier, concealer), timeline reconstruction, condition-combination analysis, and feeding of the governance ontology. Classification is by mechanism rather than location, on the empirical grounds that the same mechanisms (e.g., positional parsing of LLM output, copy-pasted error suppression) recur across unrelated components, so one mechanism-level defense immunizes an entire class. The authors flag two limitations at the outset: classification was performed by the two system operators without independent annotators, and the corpus excludes failures still silent at study end—a surviving-silence bias they revisit in their threats section.
A five-class taxonomy of silent failure
The taxonomy distributes 22 incidents across five mechanism classes. Class A (environment and platform quirks) covers cases where correct logic is defeated by runtime-environment behavior—bash 3.2 not propagating ERR traps, BSD awk aborting on invalid UTF-8, missing GNU utilities—with the signature of green development environments and silent production. Class B (design-assumption mismatches) covers code consistent with an assumption that production violates; its sharpest generalization is that any positional parse of LLM output is a latent failure, since instruction-following is a distribution, not a contract. Class C (error swallowing and dilution) includes an incident in which three individually reasonable abstraction hops stripped an upstream quota-exhaustion cause down to a content-free "HTTP 502" alert, and a governance executor that executed 67 checks from an empty string for months. Class E (operational omission and forensic blind spots) is the largest class at 8 incidents and contains the corpus's longest silence: a 60-day SSD backup failure ultimately traced to macOS TCC sandbox denials, during which the forensic collectors themselves were silently denied and returned empty output recorded as "normal"—leading to the rule that an instrument that cannot distinguish "nothing there" from "I was not allowed to look" manufactures false reassurance.
Class D (chained hallucination and fabrication) is the paper's distinctive contribution. In four documented incidents, a Class A/B/C failure deposited polluted content where a downstream LLM expected signal, and the model performed its trained function—fluent completion—producing output that inherited the form of health with the content of failure. The flagship incident is instructive in its ordinariness: a Unicode surrogate in scraped content caused a mid-write JSON serialization error; a logging function printed diagnostics to stdout; command substitution captured the error text as a cache payload; and the reduce-step LLM, finding error-code vocabulary throughout its context, composed a confident analysis of a fabricated "Hugging Face platform crisis" pushed to the user as a routine digest. The load-bearing fix was a single >&2 redirection. A companion finding is that fabrication does not require a model: a fallback path that manufactures plausible-shaped output is, in the authors' phrase, a hallucination implemented in shell. The defense is consequently system-side—stderr discipline, alert stripping before context assembly, provenance labeling—rather than model-side, a reframing of hallucination research as a systems property.
Cross-cutting quantitative findings
Three quantified results carry most of the paper's argumentative weight. First, on discovery: roughly 70% of silent failures were caught by human observation of pushed output, while the automated stack of 4,286 tests, 827 checks, and a 19-point preflight stayed green through most incidents. The authors acknowledge this share is partly selection-driven and may reflect an unusually attentive single operator, but the magnitude supports their institutionalization of a weekly 30-minute user-view observation ritual as a first-class observability signal. Their attempt to mechanize this—an LLM "observer" critiquing prior-day outputs—found real regressions but shipped with a Class B path bug and a sampling artifact, illustrating that a judge LLM is itself a governed component inheriting every class in the taxonomy.
Second, a retrospective audit of 15 incidents scored the declarative governance layer at 0% ex-ante prevention but 87% ex-post regression blocking, with 80% of misses attributable to "blank categories"—dimensions no invariant had ever contemplated. The authors draw the pointed conclusion that audit is a regression engine, not a prediction engine: it encodes the past, and prevention of novel mechanisms must come from user-view observation, adversarial review, and target-environment exposure. A 16-scenario adversarial injection exercise scored 16/16 only after its first run drove a round of guard additions.
Third, incident latency spans 13 hours to 60 days and correlates with failure mechanism rather than code complexity. The long tail lives in seams—deployment topology, OS policy, observer–observed coupling, declared-vs-runtime state—where, by construction, no test runs. The authors propose silence-latency percentiles as a reportable reliability metric, observing that time-to-detect dominates time-to-repair by one to two orders of magnitude for this failure class.
Nearly every postmortem also decomposed into a trigger–amplifier–concealer structure, which yields the paper's most prescriptive claim: a fix that addresses only the trigger is cosmetic, because triggers are unbounded while amplifiers and concealers are finite and architecture-owned. Every recurrence in the corpus traces to a trigger-level fix. Relatedly, a meta-rule manifestation counter reached at least 28 across the 22 incidents, appearing at a roughly constant rate in new forms—supporting the treatment of silent failure as a bug class to be governed rather than a finite list of bugs to be fixed.
The defense framework and its scorecard
The emergent defense framework rests on five pillars: declarative governance with mandatory verification depth; sabotage validation (deliberately breaking the system to prove each guard fires, without which an unvalidated detector is indistinguishable from a vacuous one); a declared-state convergence engine that machine-closes the registry-to-crontab gap with staged escalation and a standing rule that audit observes and never mutates; layered context hygiene and a six-level cumulative anti-fabrication guard ladder, with post-deployment measurement showing targeted fabrication patterns dropping 53–92% in the affected synthesis job; and self-monitoring watchdogs with alert routing that never depends on the failing subject. Defense effectiveness follows a three-step maturation path—point fix, named meta-rule, mechanized scanner—with a clean empirical separation: every meta-rule reaching the scanner step has zero recorded recurrences, while every recurrence involves a lesson that stopped at step one.
Discussion: seams, not components
The discussion reframes complexity as a seam problem. No individual failing part was complex—a symlink, an abspath call, a boolean default; every incident lived in a combination, and combinations grow superlinearly while tests cover only imagined ones. This inverts the instinctive incident response of adding guards, since each guard adds a part and therefore seams (the convergence engine itself produced three incidents before being caged to read-only). The codified posture is a "Sunset Law"—retiring complexity outranks adding protection, one logical entity must have one physical representation, and observers must not mutate the observed. The authors also examine their own arrangement—one human operating an AI system with an AI engineering collaborator—concluding, with appropriate hedging, that AI collaboration shifts the binding constraint from implementation bandwidth to judgment discipline, and that the governance framework functions as the collaboration's contract.
Limitations
The paper is candid about its threats to validity. Construct validity is strained at the boundary between silent failure and loud-but-cause-free alerts, with two to three borderline incidents admitting reasonable disagreement. Internal validity is limited by operator-performed classification with no reported inter-annotator agreement and acknowledged confirmation-bias risk, partially mitigated by classes being load-bearing (each drives a scanner with objective findings). External validity is the sharpest constraint: one system, one host OS, one operator pair, eight weeks, roughly 40 jobs—the frequencies and the 70% discovery share are case-study statistics, not population estimates, and Class D frequencies may not transfer to systems without synthesis-and-push pipelines. The corpus is right-censored by construction: failures silent at study end are absent, so the true latency tail is unknown—a caveat that, as the authors note, strengthens rather than weakens their thesis. The AI collaborator co-wrote both postmortems and manuscript; the offered check is mechanical traceability of all numbers to the public repository.
Conclusion
This paper converts a single system's incident record into a transferable discipline: complete causal-chain postmortems, mechanism-oriented classification, lessons promoted from diffs to named meta-rules to sabotage-validated scanners, declared state converged by machine, observers kept read-only, and complexity actively retired. Its most uncomfortable quantified claims—that thousands of tests and hundreds of checks prevented zero novel incidents ex ante while the human eye caught 70%, and that the longest failures lived in seams between simple, correct parts—constitute a substantive empirical challenge to prevailing assumptions about agent reliability engineering. The fail-plausible class, specific to systems that generate language, is the paper's most durable conceptual addition: the failure mode in which the observer is not merely blind but convincingly deceived by the failure itself.
