QuoteBench: How Matched Scores Can Hide Command-Path Failures

Abstract: LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

• The paper introduces a crossed benchmark of 56 Bash tasks that separates command-generation quality from transport damage using fixed-reply replays and exact final-state validators.
• Nested shell transport reduces success by 55.4–73.2 points across eight configurations, with only 25.0–35.4% of direct-path successes surviving reparsing and 23.4–47.0% of failures still returning exit code zero.
• QuoteBench shows that boundary disclosure can trigger substantial model compensation, but hidden transport effects can reverse model rankings, making generation contracts, execution paths, operating points, and final-state validation essential reporting requirements.

Motivation and scope

LLM coding agents issue Bash commands through interfaces that serialize, wrap, and reparse model output. A matched execution score—the pass rate of a reply on the path it was generated for—cannot distinguish a command-generation error from damage introduced after generation. QuoteBench (2608.13547) isolates this boundary with 56 one-shot Bash tasks drawn from 14 operation families (each family has one benign control and three hazardous payload variants), selected from a survey of 86 de-identified internal incidents and 412 screened public reports. Validators inspect exact final state—file bytes, argv vectors, parsed JSON, directory state, or Git history—so any semantically correct implementation receives credit. Exit codes are shown to be an inadequate proxy: across failing executions, 23.4–47.0% exit zero while leaving the wrong final state.

The benchmark's scope is deliberately narrow: POSIX/Bash command construction under quoting and interpolation hazards. PowerShell, Windows CMD, interactive terminal state, authentication, and multi-turn recovery are excluded, and the incident surveys support mechanism coverage rather than prevalence estimation.

The crossed design

The paper's central methodological move is to cross the generation contract (how the model is told to express an action) against the execution transport (how that action reaches a shell), yielding four cells: $RR$ (raw reply, raw transport), $RN$ (raw reply through an added double-quoted parser, i.e., bash -c "R"), $NR$ (a boundary-disclosed reply on raw transport), and $NN$ (disclosed reply, nested transport). Because the same stored reply is replayed through both transports, the decomposition

$Y_{NN}-Y_{RR}=(Y_{RN}-Y_{RR})+(Y_{NN}-Y_{RN})$

is an identity verified per task, splitting the conventional matched gap into fixed-reply transport damage and contract-conditioned compensation.

The nested transport is not arbitrary: replaying each stored raw reply through a real ssh localhost "R" loopback reproduces the synthetic nested damage exactly for seven of eight configurations and within one task for the eighth. A parallel JSON-boundary experiment shows the mechanism generalizes beyond shell: a naive JSON string embedding costs 51.8–66.1 points while a correct round-trip serializer costs zero, confirming that the fault lies in unescaped transforms, not in transformation per se.

Transport damage after correct generation

The headline result is stark. Moving a fixed raw-generated reply from raw to nested transport costs every one of eight same-window configurations between 55.4 and 73.2 points, with all leave-one-family-out estimates remaining negative. Only 123 of 415 direct-path successes survive reparse (25.0–35.4% retention). The loss is not confined to adversarial payloads: benign control tasks alone lose 28.6–57.1 points, because models emit double-quote-active characters even in ordinary commands. ShellCheck flags only 34.6% of nested-only failures because each command is individually well-formed; the fault is in downstream interpolation.

Two trivial repairs eliminate the effect entirely: escaping at the interpolation point reproduces the raw-path outcome for all 448 public pairs, as does executing the reply via a temporary script (which also recovers all 292 nested-only failures on private-v1 payloads). The authors are explicit that precisely because these fixes are obvious, the contribution is the measurement, not the repair—and that both require the caller to control the boundary, which their harness survey shows is often applied downstream of the stated contract.

Masking and rank reversal

Because compensation can offset damage, matched scores hide large opposing components. GPT-5.6-sol's matched gap of −3.6 points decomposes into −64.3 points of transport damage plus +60.7 points of compensation. At the deployment configuration, model orderings partially reverse: the Kendall correlation between raw and nested rankings is 0.57 (bootstrap 95% CI [0.32, 0.82]), with one unambiguous reversal—GPT-5.6-sol versus Gemini-3.5-Flash, behind by one task under raw transport but ahead by eighteen under nested—and four more reversals resting on single-task margins. Selecting by raw success picks GPT-5.5 (56/56 raw, 50/56 nested) over the path-aware pick at 51/56; the regret is small but the top rank flips.

Compensation itself is genuine behavioral adaptation, not generic robustness: disclosed-boundary replies that recover the nested path lose 28.6–64.3 points when replayed on the raw path. A grammar-crossover arm confirms the adaptation is conditioned on the declared wrapper—GPT-5.6-sol passes 53/56 single-disclosed replies on a single-quote wrapper but only 10/56 on a double-quote one, a +80.4-point diagonal advantage. A single disclosure sentence stating the parsing environment, with no quoting advice, elicits 30.4–60.7 points of compensation in six of eight configurations; Qwen3.5-27B (0.0) and Gemini-3.1-Flash-Lite (−5.4) do not adapt. Escaping advice added to the prompt barely moves frontier models, helps middle-tier models most (Sonnet-4.6 +25.0), and does not reach the bottom.

Effort ladders and operating points

Provider effort labels are not comparable compute budgets, and ladders are non-monotonic. Across trial-0 rungs, the unconditional nested-replay pass rate of raw replies varies by at most 5.4 points within any ladder, so where matched success climbs with effort—as for Opus-4.8, whose matched gap narrows from −48.2 to −3.6 points—it does so almost entirely through increased compensation, leaving cross-path portability essentially unchanged. An unset effort field maps to different parts of each provider's ladder (Opus-4.8's unset arm resembles xhigh; Opus-5's resembles medium), reinforcing the paper's argument that evaluators must report the operating point alongside the score.

Robustness

The mechanism transfers to 42 unpublished hostile payloads: both tested models lose 73.8 and 76.2 points under fixed-reply nesting, with GPT-5.6-sol recovering 78.6 points under disclosure versus Opus-4.8's 26.2. Repeated draws preserve negative damage and positive compensation even when reply text differs across draws. Userland effects (GNU vs. BSD/macOS) are smaller than the parser effect but non-negligible—up to 12.5 points for two configurations—and an earlier BSD-live campaign shows dialect affinity that partially reorders models by up to 8.9 points.

Limitations

The paper concedes several constraints plainly. Causal claims rest on fixed replies in eight same-window configurations; effort-ladder rungs rely on a single stored generation per task. The native-tool campaign is observational, and its paired effects do not survive Holm-adjusted sensitivity analysis (all twelve adjusted values exceed .05), so it is explicitly descriptive. Held-out payloads test transfer without difficulty matching, the typed-operation pilot is exploratory (11/36 typed failures, mostly copied instruction delimiters), and other shells and multi-turn recovery remain open. Validator mutation testing covers enumerated corruption classes only; unenumerated false positives remain possible. Hosted model snapshots may change under the same identifier, making query dates part of the result.

Conclusion

QuoteBench demonstrates that matched execution scores can conceal large post-generation failures along the command path: one deliberately added parser costs 55.4–73.2 points uniformly, disclosure recovers much of it for capable models, and the resulting cancellation produces near-zero matched gaps and leaderboard reversals. The paper's prescription is procedural rather than algorithmic—evaluations of command-issuing agents should report the generation contract, execution path, operating point, and final-state validator, treating a matched score as a property of a configured system rather than of a model.