- The paper presents a novel deterministic regression testing framework that isolates component-level failures in production LLM agents.
- It employs pure-mode, locked baseline tests enforcing coverage honesty and enabling rapid per-commit gating within CI pipelines.
- Quantitative results reveal that aggregate metrics mask specific regressions while targeted slice metrics expose significant performance drops.
Deterministic Layer-Isolated Evaluation for Production LLM Agents
Motivation and Context
Current LLM agent evaluation paradigms overwhelmingly rely on aggregate end-to-end task success metrics, such as those computed by AgentBench, TauBench, SWE-bench, WebArena, and GAIA. While essential as global measures, such metrics fail to provide actionable information on the locus of performance regressions or failures. Specifically, when an agent's task-success rate drops, these methods lack the granularity to identify which internal subsystem (e.g., ontology resolution, intent detection, routing, escalation, safety, or memory) is responsible or even affected. This shortcoming complicates diagnosis, mandates time-consuming and stochastic reruns, and is exacerbated by subsystems' varying degrees of determinism and interpretability.
Layer-isolated evaluation, as introduced in "Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness" (2606.11686), presents a systematic solution: a deterministic, component-level regression testing framework for production LLM agents, grounded in a layer taxonomy and enforced via a coverage-honest locked baseline.
Architectural Decomposition and Pure-Mode Harness
The methodology instantiates a fixed architectural taxonomy for a deployed multi-turn, food-and-beverage ordering agent, partitioning logic into discrete, contract-defined layers: L0 ontological pre-resolution, intent signals, routing, decomposition, escalation, safety, memory, and a slate of cross-cutting concerns (envelope, defense, reformulation, OOD rejection, etc.). Each architectural layer is associated with a deterministic assertion slice—an executable pure-mode test that verifies the layer's output without invoking any LLM calls or relying on stochasticity. This makes test execution strictly deterministic, highly efficient (approximately $2.4s$ wall time for 225 cases), and fully reproducible, enabling per-commit (per-PR) gating in CI rather than expensive post-hoc nightly jobs.
Coverage-honesty is a fundamental tenet of this harness: a slice with no corresponding tests is reported as "null" (uncovered), not "green." Thus, the aggregate metric cannot mask a lack of coverage in any layer; the test suite's weaknesses are explicitly surfaced on each run.
Quantitative Validation via Controlled Regression Injection
The core empirical validation is realized by controlled single-layer regression injection. Each experiment selectively degrades one non-safety layer (via monkeypatch) and quantifies the resultant effect across all slices by comparing the post-injection pass-rate deltas to the locked baseline. The main metric is the drop in assertion pass rate: both globally (aggregate) and per-slice.
Critically, the majority of induced regressions are dramatically masked by the aggregate: for six of seven non-foundational regressions, the full-suite pass-rate drop is between −1.7 and −5.9 percentage points—within typical dashboard noise—whereas the targeted slice craters by −25 to −91 percentage points. In foundational layers (e.g., ontology resolution), where broad dependency exists, the aggregate drop is higher (−26.5pp) with a wider blast radius, yet even then the targeted slice is among the three hardest hit.
(Figure 1)
Figure 1: Fault-localization heatmap—each row represents a single injected regression, columns are slices, and cells are pass-rate drops; darkest cells correspond to highest impact, largely boxed on the diagonal, indicating effective localization.
These results quantitatively demonstrate both masking—focal regressions are hidden in aggregate in all but foundational failures—and localization—the responsible slice is cleanly identifiable. The per-slice gate thus operationalizes fine-grained fault localization lost in end-to-end metrics.
Cross-Tenant Replication
To establish generalizability beyond a single tenant or catalog, the entire injection experiment is replicated on a structurally distinct tenant (Starbucks SG, a Western menu with divergent ontology and modifiers). The authored pure slice-set for each injected layer exhibits the same signature: all seven injections cause slice-specific craters for their intended layer, with foundational regressions presenting the expected broad impact. The aggregate pass-rate moves more for the smaller test suite, but the localization property holds robustly.
Coverage-Honest Gating and Real Regression Case Study
A salient methodological contribution is the enforcement of per-layer coverage-honesty—unexercised layers remain visible as such until explicitly tested. This was directly actionable: a historical real-world regression in the order-confirmation guardrail (over-restricting confirmed orders) was only caught via exhaustive human QA; the relevant routing slice was uncovered, which the framework would have flagged. Unlike aggregate pass rates, which would have remained green, the coverage-honest harness would have surfaced the risk. The ability to replay actual shipped regressions, and their lack of effect on the current slice suite (where not covered), reinforces the honesty and limitations of the approach.
Comparative Positioning and Cost Efficiency
Layer-isolated evaluation is not a replacement for stochastic end-to-end testing and outcome-focused benchmarking, but complements these by making regression diagnosis and gating rigorous, cheap, and actionable during development. In direct comparison with stochastic mutation-based whole-workflow methods such as AgentAssay (Bhardwaj, 3 Mar 2026), the presented framework is exact, oracle-backed, deterministic, and near-zero cost, supporting per-commit gating at constant scale.
The table below summarizes this comparative perspective:
|
Pure Layer-Isolated |
Whole-Workflow Mutation |
| Unit |
One architectural layer |
Entire agent episode |
| Verdict |
Deterministic |
Stochastic |
| LLM Calls |
None |
Yes |
| Cost |
∼10ms |
Minutes, token cost |
| Baseline |
Locked pass-rate |
Behavioral fingerprint |
| Localization |
Slice crater |
Fingerprint delta |
| Placement |
Every PR |
Nightly / release |
Practical and Theoretical Implications
Practically, this approach enables fine-grained, deterministic, highly sensitive gating over compositional agent architectures, minimizing regression latency and accelerating development cycles in environments with rapid evolution and complex code/data dependencies. Theoretically, it provides a concrete instantiation of the component-level, intermediate-artifact evaluation model advocated but unrealized in agent operations literature (Xia et al., 2024).
Its limitations are acknowledged: pure-mode assertions cannot replace stochastic/generative capability evaluation and only measure compliance relative to a locked, potentially non-normative baseline (i.e., they are drift detectors, not oracles of correctness). The scope of organic, non-author-injected regressions, and generalizability to non-F&B domains or agents with a higher proportion of non-deterministic generative behavior, remains an area for future empirical work.
Conclusion
Layer-isolated evaluation, as realized in this work, demonstrates that deterministic component-level regression gates, paired with coverage honesty and a locked baseline, can efficiently and rigorously localize regressions in production LLM agents—regressions that traditional aggregate task success metrics routinely mask. While this is not a panacea for all evaluation needs, it operationalizes a crucial missing link in agent CI: diagnostics that are exact, fast, and traceable to the layer of failure. Its future utility depends on continued expansion of slice coverage, broader domain replication, and systematic mining of organic fault signatures.
