Layer-Isolated Evaluation for LLM Agents

• Layer-Isolated Evaluation is a component-level assessment technique that decomposes LLM agents into distinct architectural layers (e.g., ontology, routing, safety) for targeted regression diagnostics.
• It replaces aggregate end-to-end metrics with deterministic assertion slices, enabling precise localization of failures and facilitating continuous integration checks.
• The method demonstrates that layer-specific evaluation can reveal hidden regressions that standard scores obscure, ensuring more reliable and actionable insights.

Layer-Isolated Evaluation is a component-level evaluation methodology in which an LLM agent is decomposed into explicit architectural layers, each exercised by its own deterministic assertion slice rather than being judged only by a single end-to-end success number. In the formulation introduced for a deployed ordering agent, the layers are evaluated in a no-LLM “pure” mode against locked per-slice baselines, so that a regression can be localized to ontology, intent, routing, decomposition, escalation, safety, memory, or cross-cutting envelope and defense logic instead of being submerged in an aggregate outcome metric (Zhang et al., 10 Jun 2026). Related work in other AI settings uses analogous layered decompositions—separating gatekeeping from fidelity analysis, or run data from benchmark and model metadata—because different failure classes become visible at different levels of the evaluation stack (Coleman et al., 16 Jun 2026, Ghosh et al., 8 Jun 2026).

1. Definition and motivating problem

The immediate target of Layer-Isolated Evaluation is the deterministic scaffold of a production LLM agent, not the stochastic model call as such. The motivating diagnosis is that end-to-end task-success is the dominant way to evaluate agents, but one aggregate number reports that an agent regressed without indicating where the regression occurred. This is especially problematic when the system contains multiple internal contracts—ontology resolution, intent extraction, routing, decomposition, escalation, safety, memory, and prompt-envelope logic—that can fail independently or interact asymmetrically (Zhang et al., 10 Jun 2026).

The method therefore replaces a purely black-box view with a fixed internal taxonomy. Each layer is given its own assertion slice, and each slice is executed in a deterministic no-LLM mode. This changes the object of evaluation from whole-task success to contract preservation within a known architecture. A plausible implication is that the method is best understood not as a benchmark for overall capability, but as a regression-localization discipline for systems whose behavior depends heavily on non-generative orchestration logic.

The paper is explicit that this does not replace end-to-end evaluation. End-to-end evaluation remains the user-facing measure of whether the agent succeeds overall. Layer-Isolated Evaluation instead answers different questions: which internal contract broke, whether the break is local or foundational, whether the damage smeared across unrelated components, and whether the system even has coverage for the affected behavior (Zhang et al., 10 Jun 2026).

2. Layer taxonomy and assertion slices

The production agent is decomposed into a fixed taxonomy of layers named in the abstract and body as ontology pre-resolution, intent signals, routing, decomposition, escalation, safety, memory, and cross-cutting envelope/defense, with additional cross-cutting slices such as reformulator, OOD reject, locale fidelity, session init, recommendation rules, and read tools. The slice table reports 23 slices in total: L0_speech_act, L0_ontology, L0_intent, L2_routing, L3_escalate, L3_decompose, L3_csp, L4_safety, L4_safety_kg, L4_knowledge, L4_health, L4_memory, L4_personalization, L4_reflexion, defense, envelope, reformulator, ood_reject, recommend_rules, read_tools, session_init, locale_fidelity, and L1_legacy (Zhang et al., 10 Jun 2026).

The baseline contains 238 total cases across those 23 slices, with 19 covered slices. The per-slice counts range from 43 cases in L0_speech_act and 21 in L0_ontology to 1 in locale_fidelity, while L2_routing, L4_memory, L4_personalization, and L4_reflexion are listed with zero cases and are therefore uncovered rather than treated as passing (Zhang et al., 10 Jun 2026).

“Pure mode” denotes the crucial engineering restriction. In this mode no model call occurs; outputs are computed from deterministic code paths such as ontology resolver canonical IDs, rule-based escalation decisions, deterministic reformulation, OOD short-circuit predicates, server-side repricing, and rendered prompt-envelope blocks. Because the harness bypasses the LLM, the execution is fully reproducible and cheap enough for CI. The paper reports that the pure suite runs 225 cases in 2.39 seconds, about 10 ms per case, with 30 live-only variants skipped in pure mode (Zhang et al., 10 Jun 2026).

This slice-based design also formalizes the meaning of a “layer.” A layer is not merely a conceptual stage in a pipeline; it is a testable contract with a dedicated assertion space. The methodology is therefore architectural as much as evaluative.

3. Baseline locking, coverage honesty, and gating logic

A central feature of the method is the locked per-slice baseline. For each slice, the baseline records total cases, passed cases, pass rate, and failed IDs. Current runs are compared against this locked contract on every pull request, turning the suite into a deterministic change detector rather than a free-form test collection (Zhang et al., 10 Jun 2026).

The paper’s second major contribution is the “coverage-honesty” criterion. If a slice has zero cases, its reported rate is null, not 100%. The method therefore refuses to score an unexercised layer as healthy. The uncovered slices are L2_routing, L4_memory, L4_personalization, and L4_reflexion; the low-$N$ slices are L4_health and locale_fidelity (Zhang et al., 10 Jun 2026). This choice is methodologically important because an aggregate green status would otherwise conceal missing coverage.

The gate logic is intentionally simple. If a slice was previously uncovered and remains uncovered, it continues to be tracked without being falsely credited. If a slice had a baseline rate and the current rate drops below that baseline, the merge is blocked. The paper characterizes this as a drift detector rather than a correctness oracle: it detects that behavior changed relative to a locked baseline, but it does not prove that the baseline itself is ideal (Zhang et al., 10 Jun 2026).

This baseline-locked design also gives the methodology a specific operational role. It is optimized for per-commit regression detection, not for scoring frontier capability. That orientation explains both the deterministic harness and the emphasis on exact per-slice comparisons.

4. Controlled regression injection and the masking phenomenon

The empirical core of the paper is a controlled regression-injection study. Exactly one non-safety layer is degraded at a time by monkeypatching a single entry point, and the full pure suite is rerun. The injected regressions are: escalation $\rightarrow$ never escalate, intent signals $\rightarrow \varnothing$, defense scan $\rightarrow$ allow-all, OOD gate $\rightarrow$ never reject, reformulator $\rightarrow$ identity, decomposer $\rightarrow$ no sub-goals, and ontology resolve $\rightarrow \varnothing$ (Zhang et al., 10 Jun 2026).

For the six local regressions, the aggregate pure-suite pass rate moves only from $-1.68$ pp to $-5.88$ pp, while the matching slice collapses by $\rightarrow$0 pp to $\rightarrow$1 pp. For ontology, the slice drop is $\rightarrow$2 pp. This disparity is the paper’s “masking” result: an aggregate metric can appear nearly stable while the directly affected contract craters (Zhang et al., 10 Jun 2026).

Injected regression	Matching slice delta	#moved slices
escalation $\rightarrow$3 never escalate	$\rightarrow$4 pp	2
intent signals $\rightarrow$5	$\rightarrow$6 pp	2
defense scan $\rightarrow$7 allow-all	$\rightarrow$8 pp	1
OOD gate $\rightarrow$9 never reject	$\rightarrow \varnothing$0 pp	1
reformulator $\rightarrow \varnothing$1 identity	$\rightarrow \varnothing$2 pp	3
decomposer $\rightarrow \varnothing$3 no sub-goals	$\rightarrow \varnothing$4 pp	2
ontology resolve $\rightarrow \varnothing$5	$\rightarrow \varnothing$6 pp	9

The localization statistics are equally important. The injected layer’s slice is the single worst-hit in 5 of 7 cases and top-3 in 7 of 7, with mean rank 1.29 out of 19 covered slices. The paper emphasizes that the fact that a slice reacts to its own fault is partly by construction; the substantive result is that the damage largely stays off unrelated slices, while foundational layers such as ontology produce a wider blast radius (Zhang et al., 10 Jun 2026).

A further technical detail is that each full-suite pass uses a freshly constructed runtime. Reusing runtime state had produced a phantom off-diagonal effect in the heatmap, and the fresh-runtime design removed that artifact. This detail matters because it shows that even a deterministic harness can create false coupling if state leakage is not controlled.

Replication on a second, structurally different tenant, Starbucks SG, strengthens the claim. The study builds a parallel pure-slice set with 16 cases across the same seven injected layers; all seven matching slices crater, with drops from $\rightarrow \varnothing$7 to $\rightarrow \varnothing$8. For six of seven injections the off-diagonal is clean, and ontology again behaves as the foundational exception because it also affects decomposition (Zhang et al., 10 Jun 2026).

5. Position within LLM-agent evaluation methodology

The paper positions Layer-Isolated Evaluation as a concrete deterministic instantiation of the component-level evaluation that EDDOps prescribes but leaves unimplemented, with CheckList as an ancestor and as the deterministic mirror image of whole-workflow stochastic mutation testing. The comparison with AgentAssay is especially sharp: mutation testing operates on whole workflows, uses stochastic verdicts, keeps the LLM in the loop, and incurs many trials and token costs, whereas Layer-Isolated Evaluation isolates one architectural layer at a time, uses an exact oracle, removes the LLM from the loop, and is suitable for per-PR CI gating (Zhang et al., 10 Jun 2026).

The cost differential is large enough to define different operational niches. The pure suite runs 225 cases in 2.39 seconds, whereas a live agent run has median 73 seconds and $\rightarrow \varnothing$9 192 seconds. One live episode therefore costs more wall-clock time than the full deterministic suite. This suggests that the method is primarily a development-time instrument for continuous integration, while end-to-end and stochastic evaluations remain necessary for deployment-facing assessment (Zhang et al., 10 Jun 2026).

The methodology also introduces an architectural notion of adequacy. A system with uncovered slices cannot claim complete health merely from an aggregate pass rate. This differs from conventional benchmark reporting, where missing coverage is often implicit. In Layer-Isolated Evaluation, uncovered layers are first-class findings.

A plausible implication is that the approach is particularly well suited to production agents whose reliability depends on a large deterministic shell around an LLM. It is less obviously applicable to systems in which the dominant failure modes reside inside an uninspectable generative core.

6. Broader layered-evaluation pattern across domains

Although “Layer-Isolated Evaluation” is introduced for a production ordering agent, related work shows a broader methodological pattern: evaluation is decomposed into separable layers when aggregate metrics obscure attribution. In LLM-driven quantum circuit generation, a layered framework separates gatekeeper screening across seven physical and framework criteria, circuit fidelity analysis against analytical and reference-implementation values for $\rightarrow$0, and design entropy as a run-to-run behavioral consistency metric. That work also reports that two apparent model failures were caused by silent fallback-template substitution in the harness, leading to the explicit claim that evaluation infrastructure belongs inside the same trust boundary as the models it tests (Coleman et al., 16 Jun 2026).

An interpretive analogue appears in Evaluation Cards, where reporting is decomposed into benchmark metadata, evaluation run data, and model metadata, then recomposed into a unified record with four signals: reproducibility, documentation completeness, provenance and risk, and score comparability. The purpose is not regression localization inside an agent, but prevention of interpretive collapse in public reporting. The common structure is nonetheless clear: different questions are attached to different layers of evidence rather than to a flattened score triple (Ghosh et al., 8 Jun 2026).

A physically motivated variant appears in wireless PHY evaluation of generative models. There, the framework isolates channel norm and power through spectral-efficiency analysis, channel direction through codebook fingerprinting, and downstream utility through an application cross-check. The argument is that generic metrics such as MMD or FID lack physical motivation and can mis-rank models relative to actual PHY-layer utility. Here again, layer isolation means factorizing evaluation by failure mode and by the semantics of the underlying system (Baur et al., 2024).

A certification-oriented precursor exists in the assessment of partitioning hypervisors. That framework decomposes isolation into spatial, temporal, and fault dimensions; maps lower-level mechanisms such as M1–M4 and T1–T4 to functional requirements; and maps testing techniques to assurance requirements. Its purpose is evidence generation rather than CI regression gating, but it shares the same core principle: meaningful isolation assessment requires explicit lower-level decomposition, not only observation of externally visible behavior (Cesarano et al., 2022).

Taken together, these works suggest that layer-isolated evaluation is not a single fixed protocol but a methodological family. Its common commitment is to replace undifferentiated outcome scoring with structured attribution: each layer, factor, or evidence tier is evaluated according to the contract it actually implements.