Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent

Abstract: Multi-step enterprise agent tasks fail in a characteristic way: single-pass inference has no checkpoint between deciding an answer and committing to it. We study one production system (Leni) whose architecture installs such checkpoints: verification loops (execute, observe, compare, correct) staffed by lightweight task-specialized post-trained models. We evaluate the unmodified production configuration on three public benchmarks stressing distinct failure modes: SpreadsheetBench Verified (silent computation error), BullshitBench v2 (premise confabulation), and the GAIA validation split (cascade error over long tool chains). The full system improves over its frontier base model by +11.0 percentage points on SpreadsheetBench (91.25% vs 80.25%, n=400, p<0.001), +7 to +10 percentage points on BullshitBench (98% vs 91%, n=100), and roughly +15 points on GAIA validation (75.2% pass@1, n=165; 83.0% best-of-k). Our central contribution is a decomposition of that uplift: most of it comes from scaffolding, routing, and specialist models rather than from the verification step itself, whose isolated contribution is small (+1.5 points) but concentrated at the top of the score distribution, where it converts otherwise-failing tasks. We instrument the loop end-to-end, yielding an empirical verifier confusion matrix (catch rate about 0.20, fix rate 0.75, no false-alarm regressions) that grounds a compounding-reliability model. Specialist-swap ablations suggest that the loop's value depends on who observes it: replacing the small trained verifier with the generating frontier model eliminates most rescues. A valid-premise control shows zero over-rejections in 100 expert-level questions.

• The paper decomposes a production enterprise agent’s performance across SpreadsheetBench, BullshitBench, and GAIA, finding that scaffolding, routing, and specialist models provide most of the uplift over the base model.
• The paper finds that a deterministic verification loop added 1.5 percentage points on SpreadsheetBench through six rescues, with an estimated 20% error-catch rate, 75% fix rate, and no observed false alarms.
• The paper shows that independent specialist observers outperform the generating frontier model in preliminary ablations, while correcting GAIA from 77.6% to a pre-specified 75.2% pass@1 and highlighting vendor-evaluation limitations.

Overview and motivation

This paper examines the sources of reliability in Leni, a production AI business analyst, by evaluating its unmodified production configuration on three public benchmarks that stress distinct failure modes: SpreadsheetBench Verified (silent computation error), BullshitBench v2 (premise confabulation), and the GAIA validation split (cascade error over long tool chains). The central question is not whether the system performs well — it does — but where the performance comes from. The answer is deliberately counterintuitive relative to the paper's own framing: most of the uplift over the frontier base model comes from scaffolding, routing, and specialist models rather than from the verification loop itself, whose isolated contribution is small but concentrated at the top of the score distribution.

The paper's positioning is candid: it is a vendor evaluation of the vendor's own system, with limitations stated explicitly and independent replication identified as the real remedy. The authors also correct an earlier company-reported GAIA figure (77.6%) that mixed selection rules across tiers, replacing it with a pre-specified pass@1 result.

The verification loop framework

The paper formalizes a verification loop as four stages: execute (base-model action), observe (independent read-back through a different code path), compare (against intent), and correct (targeted fix re-entering observation). The load-bearing stage is observation: many defects are structurally invisible to the process that created them and become observable only when re-read independently.

A compounding-reliability model characterizes each loop by catch rate $c$, fix rate $r$, false-alarm rate $f$, and breakage rate $b$. Effective per-step reliability becomes:

$p' = p(1 - fb) + (1-p)\,c\,r$

The loop helps if and only if $(1-p)\,cr > p\,fb$; an imperfect verifier can reduce reliability if it frequently second-guesses correct work destructively. Because beneficial corrections act before errors propagate, the loop's marginal value grows with chain length $n$. The model makes two predictions: loops matter most on long dependency chains (stated as falsifiable, not yet tested), and the verifier's error profile is a first-class design quantity.

Three oracle classes are distinguished: deterministic (external engine re-executes the artifact; highest ceiling), self-reflective (structured re-evaluation; moderate ceiling), and planner-mediated (typed artifacts checked against a plan; high on long chains).

Specialist models inside the loop

Four lightweight specialists (0.5–4B parameters), post-trained from open-weight Qwen3 bases via distillation SFT from Claude Opus 4.6 followed by RLVR/GRPO or DPO, staff the loop stages:

Specialist	Size	Role
Leni-Cell-S	~4B	Cell-diff verification (observe/compare)
Leni-Triage-S	~4B	Premise decomposition + epistemic triage
Leni-Parse-S	~1.5B	Typed-artifact extraction
Leni-Route-XS	~0.5B	Per-step routing

Serving costs are estimated at 0.02–0.1× frontier-model cost per call, which is what makes per-step verification economically viable as a default rather than a luxury.

Headline results

Total uplift over the bare base model is large and, on two benchmarks, statistically unambiguous:

Benchmark	Base	Leni	Uplift
SpreadsheetBench ($n{=}400$)	80.25%	91.25%	+11.0pp ($p<0.001$)
BullshitBench / Sonnet ($n{=}100$)	91%	98%	+7pp (nominal)
BullshitBench / Opus ($r$0)	87%	97%	+10pp ($r$1)
GAIA validation ($r$2)	~60% (est.)	75.2% pass@1	~+15pp

On SpreadsheetBench, 91.25% placed second among public entries behind a neurosymbolic system with a custom spreadsheet runtime (94.25%). On GAIA, at 75.2% pass@1, the system is statistically indistinguishable from reconstructed figures for Genspark (~75.4%) and Manus (~73.4%); no superiority claim is made, and the hidden test set has not been submitted to.

The GAIA correction

An earlier company report claimed 77.6% on GAIA validation. Re-grading all 803 stored trajectories with the official scorer (validated at 218/218 agreement against previously labeled runs) revealed that this figure mixed best-of-$r$3 selection (Level 1) and multi-campaign pooling (Level 3) with legitimate single attempts (Level 2). Under a pre-specified pass@1 rule, the corrected headline is 75.2% [68.0, 81.1], with best-of-all-runs reaching 83.0%. A scripted retrieval audit found GAIA-derived URLs in 25 of 803 trajectories spanning 12 tasks; treating every flagged correct run as a failure gives a conservative lower bound of 70.9%. This correction is notable for its transparency and materially weakens the earlier claim.

Decomposing the uplift

The decomposition is the paper's central contribution. On SpreadsheetBench, prompt-plus-scaffold accounts for +9.5pp of the +11.0pp total; the deterministic recalculation loop adds only +1.5pp (6 rescues out of 400 tasks). On GAIA, after correcting to pass@1, the loop's marginal contribution over estimated structure tiers is ~+1pp and cannot be cleanly isolated. On BullshitBench, the firewall is the scaffold, so layer decomposition does not apply.

The loop's contribution is nonetheless positionally decisive: at the top of a leaderboard, where all systems have exhausted scaffolding gains, +1.5pp separates mid-pack from near-top. Full instrumentation yields what appears to be the first published empirical confusion matrix for a production deterministic loop: catch rate $r$4, fix rate $r$5, zero observed false alarms in 357 confirmations (bounded above by ~1%). With $r$6, the model guarantees the loop cannot hurt, and the measured rescue matches $r$7 within rounding. The telemetry also localizes investment: raising $r$8 is worth up to +8pp; raising $r$9 at most +0.5pp. The 32 missed errors quantify the cost of LLM-mediated versus symbolic comparison, plausibly explaining most of the 3-point gap to the neurosymbolic leader.

Who observes matters

Preliminary specialist-swap ablations support the paper's most consequential claim: replacing the small trained verifier with the generating frontier model reduces SpreadsheetBench rescues from 6 to 2 and reduces BullshitBench correct rejection by 4–5pp. This is consistent with documented self-preference biases in LLM evaluators. However, the swaps cover only two of four specialists, come from single internal runs, and lack the critical third condition — an independent generalist verifier from a different provider — which would separate independence from specialization. The "independent observer" hypothesis is motivated, not established.

A further finding survives qualification: extended-thinking configurations of strong reasoners perform worse than their standard counterparts on premise validation, consistent with inference-time compute being spent constructing answers inside false frames rather than questioning the frames themselves.

Calibration of the epistemic firewall

Because a firewall trained to reject premises could inflate scores by over-rejecting, the authors run DRACO, an internal control of 100 legitimate expert-level questions through the same firewall-active harness: zero over-rejections, bounding the false-positive rate at ≲3.6% (95% upper limit). The control is observational rather than adversarially matched to the benchmark's deception techniques, so calibration against fabrication-styled valid premises remains open.

Limitations

The paper is unusually explicit about threats to validity. Key ones: it is a vendor self-evaluation; headline runs are single scored runs with measured run-to-run sensitivity of roughly ±3pp; GAIA results are validation-only with public answers and a contamination-adjusted bound of [70.9%, 75.2%]; competitor figures are reconstructions from per-level reports; judge-based BullshitBench scoring includes one intra-family judge; the contamination audit covers the question corpus but not workbook-trace or tool-trajectory components; no controlled same-model comparison against ReAct-style, CRITIC-style, or best-of-$f$0 scaffolds at matched compute has been run; and specialist weights and training data are proprietary, permitting method-level but not checkpoint-level replication.

Conclusion

Across three benchmarks stressing unrelated failure modes, the architecture produces large total uplift over its base model, with the two largest comparisons statistically unambiguous. The durable contribution is the decomposition: structure supplies most of the gain, the verification checkpoint supplies a small positionally decisive remainder, and preliminary evidence indicates the checkpoint pays only when the observer is independent of the generator. The empirical verifier confusion matrix converts loop design from folklore into measurement, making catch rate, fix rate, and false-alarm rate estimable quantities that indicate where the next point of reliability will come from.