ContractBench Compliance Benchmark

• ContractBench is a benchmark for observation contract compliance that rigorously tests temporal validity (TTL) and byte-level integrity in API workflows.
• It organizes 33 deterministic tasks into quadrants with dual failure modes and assigns specific labels to categorize compliance errors.
• Empirical results show no evaluated model exceeds 80% success, underscoring the need for robust architectural solutions in agent design.

ContractBench is a benchmark for observation contract compliance in tool-augmented LLM agents: it studies whether an agent can preserve externally constrained intermediate artifacts—such as presigned URLs, session tokens, OAuth state parameters, signed request fields, HMAC-protected webhook payloads, ETags, idempotency keys, rate-limit windows, and multi-step token chains—across the observation-to-action pipeline. Its central claim is that this capability is distinct from generic tool-use competence: an agent may call the correct APIs in the correct broad order and still fail because it uses an artifact after expiry or corrupts its bytes in transit. The benchmark introduces 33 deterministic, programmatically graded tasks organized around two orthogonal failure modes, validity and integrity, and reports that no evaluated model exceeds 80% success on the full suite (Wang et al., 17 May 2026).

1. Conceptual foundation

ContractBench defines observation contracts as artifacts whose later use is constrained by the external system that produced them. The benchmark focuses on two independent constraints. The first is temporal validity: an artifact may be usable only within a bounded time window, as with TTL-limited presigned URLs, refreshable tokens, scheduled maintenance windows, or Retry-After backoff intervals. The second is byte-level integrity: an artifact may have to round-trip exactly, as with OAuth state parameters, signed query strings, HMAC inputs, conditional request tokens, or canonicalized request payloads (Wang et al., 17 May 2026).

This framing separates ContractBench from ordinary tool-use benchmarks that treat intermediate outputs as advisory. In the ContractBench view, many real APIs do not tolerate paraphrase, normalization, truncation, or deferred reuse. A presigned URL may fail if a query parameter is reordered; an OAuth state parameter may fail if even a single byte changes; a token refresh may fail if the refreshed token is then mutated; a well-preserved token may still fail if used after expiry. The benchmark therefore treats restraint—waiting when required, avoiding premature retries, preserving opaque strings exactly, and refusing shortcut flows—as a first-class capability rather than a side effect of general reasoning (Wang et al., 17 May 2026).

The paper argues that this capability is both emergent and regression-prone. It is emergent because base checkpoints score 0% on the full suite; it is regression-prone because post-training can improve or erode compliance depending on how it affects byte preservation, temporal planning, and shortcut avoidance. This suggests that observation contract compliance is not reducible to scale alone, nor guaranteed by stronger general agent performance (Wang et al., 17 May 2026).

2. Formal model of observation contracts

The benchmark formalizes an observation contract as

$C = (o,\; t_{\text{issue}},\; \tau,\; \pi),$

where $o$ is the issued artifact, $t_{\text{issue}}$ is its issue time, $\tau > 0$ is its time-to-live, and $\pi : \mathcal{O} \to \{0,1\}$ is an integrity predicate. The corresponding validity window is

$$W(C) := [\,t_{\text{issue}},\; t_{\text{issue}} + \tau\,).$$

In the implementation described by the paper, integrity is usually checked by SHA-256 equality:

$\pi(o') = \mathbbm{1}[\,SHA(o') = h(o)\,].$

An artifact later submitted by the agent satisfies the contract only if it is used within the validity window and still satisfies the integrity predicate. The paper writes the corresponding indicator as $\operatorname{Sat}(C,o',t') \in \{0,1\}$. For a multi-step trace with contracts $\{C_i\}_{i=1}^n$, overall compliance is defined multiplicatively:

$$\operatorname{Compliant}(\{C_i\}, \{(o'_i, t'_i)\}) := \prod_{i=1}^{n} \operatorname{Sat}(C_i, o'_i, t'_i) = 1.$$

Under this formulation, a single violated contract anywhere in the trace makes the episode non-compliant (Wang et al., 17 May 2026).

This formalization supports a clean distinction between the two failure axes. A validity failure occurs when the agent uses an artifact outside $o$0. An integrity failure occurs when the later artifact $o$1 does not satisfy $o$2. The paper emphasizes that these are logically independent: byte-perfect relay does not rescue a stale token, and prompt refresh does not rescue a token that is then normalized, truncated, or re-encoded incorrectly (Wang et al., 17 May 2026).

3. Benchmark construction and task suite

ContractBench contains 33 dual-axis tasks. The suite is organized into four quadrants: Q1 for low validity pressure and low integrity pressure; Q2 for temporal-validity pressure isolated; Q3 for byte-integrity pressure isolated; and Q4 for joint pressure on both axes. The suite is heavily weighted toward Q4: 24 of the 33 tasks are dual-pressure tasks, with 4 Q2 tasks and 5 Q3 tasks (Wang et al., 17 May 2026).

The task catalog spans patterns such as Timing/Backoff, Byte-exact, OAuth/Auth, Signed requests, State chains, Resource management, and Multi-service. Named tasks include scheduled-maintenance, api-rate-limit-patience, token-refresh-workflow, presigned-url-download, csrf-form-submit, url-trap-ellipsis, long-token-handling, extreme-url-length, webhook-hmac-verify, basic-oauth-token, oauth-authorization-code, oauth-pkce-with-rotation, signed-request-canonicalization, session-cookie-chain, etag-conditional-get, idempotency-key-retry, multi-service-saga, and cascading-token-revocation (Wang et al., 17 May 2026).

Each task is packaged as four files: a hidden TOML metadata header, an agent-visible Markdown instruction, a FastAPI server implementing task logic and logging violation events, and a hidden pytest validator that consumes the log and computes the canonical reward. The agent sees only the instruction and HTTP responses from the task server. This design excludes LLM-as-judge ambiguity and makes the benchmark fully programmatic (Wang et al., 17 May 2026).

The paper also describes five deterministic production path mutations that model common integrity failures in deployed agent stacks: truncation, line-wrap insertion, URL re-encoding, query reorder, and UI trap. These mutations reproduce corruption modes such as clipping long artifacts, inserting newlines into long tokens, normalizing %2F into /, re-sorting signed parameters, or exposing link text that differs from the underlying href. Their role is methodological: they turn otherwise anecdotal “agent flakiness” into reproducible integrity stressors (Wang et al., 17 May 2026).

4. Evaluation protocol and failure taxonomy

ContractBench is evaluated under a deterministic and programmatic protocol. Time is controlled by a virtual clock, all task-instance stochasticity is seed-fixed, temperature is set to 0, model IDs are pinned, API parameters are logged, and byte integrity is checked with SHA-256. Each episode produces an HTTP request log, and the hidden validator writes a structured reward.json containing a binary success flag, a primary failure label, detailed failure metadata, and trace metadata including seed, step count, tool-call count, virtual time elapsed, and a SHA-256 trace hash (Wang et al., 17 May 2026).

The main metric is success rate (SR%). Each model is run on all 33 tasks with $o$3 runs per task, yielding $o$4 episodes per model. The paper also reports per-task pass rate and the distribution of primary failure labels across failed episodes (Wang et al., 17 May 2026).

The failure taxonomy contains 15 labels: 4 validity failures, 9 integrity failures, and 2 meta labels. The validity labels are EXPIRED_BEFORE_USE, RATE_LIMITED, SCHEDULED_UNAVAILABLE, and VERSION_CONFLICT. The integrity labels are WRONG_VALUE, MISSING_CONSTRAINT, MUTATED_TOKEN, SIGNATURE_MISMATCH, COMPENSATION_FAILURE, WRONG_HASH, SHORTCUT_TAKEN, MISSING_TOKEN, and BACKOFF_VIOLATION. The meta labels are SUCCESS and OTHER (Wang et al., 17 May 2026).

When multiple violations occur in one episode, the validator assigns one primary label using a most-severe-label rule. The appendix also supplies a severity-weighted mapping reused in the intervention experiment, including $o$5 for SUCCESS, $o$6 for EXPIRED_BEFORE_USE, $o$7 for MUTATED_TOKEN, $o$8 for SHORTCUT_TAKEN, $o$9 for WRONG_VALUE, $t_{\text{issue}}$0 for MISSING_CONSTRAINT, $t_{\text{issue}}$1 for RATE_LIMITED, and $t_{\text{issue}}$2 for OTHER (Wang et al., 17 May 2026).

This taxonomy is not merely descriptive. The paper ties the labels to recognizable API and protocol failures, citing OAuth 2.0, RFC 6585, RFC 9110, RFC 7232, RFC 3986, RFC 9421, AWS SigV4, RFC 9530, FIPS 180-4, and saga-style compensation failures. The benchmark therefore evaluates agents against failure surfaces that correspond to deployed systems rather than to synthetic benchmark-only categories (Wang et al., 17 May 2026).

5. Empirical findings

ContractBench evaluates 38 model variants from 15 families and reports that no evaluated model clears 80% on the full suite. The top model is Claude Opus 4.6 at 77.8%, or 77/99 successful episodes. Other frontier results reported in the main text include GPT-5.2 at 74.8%, GPT-5 at 70.7%, Qwen3.5-397B-A17B at 70.7%, Claude Sonnet 4.5 at 69.7%, MiniMax-M2.5 at 62.6%, MiniMax-M2.1 at 60.6%, Qwen3.5-9B at 56.6%, Gemini 2.5 Pro at 51.5%, GPT-5.1 at 48.5%, Mistral-Small-4 at 42.4%, Gemma-4-26B-A4B at 38.4%, Gemma-4-31B at 37.4%, Ministral-3-14B at 28.3%, GPT-4o at 23.2%, Qwen2.5-72B-Instruct at 23.2%, Gemini 2.5 Flash at 19.2%, and Llama-3.3-70B-Instruct at 7.1% (Wang et al., 17 May 2026).

One of the paper’s most prominent findings is a Qwen 3.5 capability cliff. Within that family, performance jumps from 0% at 4B Instruct to 56.6% at 9B Instruct, then rises to 64.6% at 27B and 70.7% at 397B-A17B. The authors argue that what emerges across this cliff is not tool-call emission but mid-trajectory restraint: waiting through maintenance windows, respecting backoff, changing strategy after failure, and persisting through multi-step workflows without corrupting or prematurely consuming artifacts (Wang et al., 17 May 2026).

A second headline finding is non-monotonic scaling across the GPT-5 family. The paper reports approximately 23.2% for GPT-4o, 70.7% for GPT-5, 48.5% for GPT-5.1, and 74.7/74.8% for GPT-5.2. The regression from GPT-5 to GPT-5.1 is concentrated on the integrity axis, especially WRONG_VALUE and MUTATED_TOKEN, and the paper interprets this as evidence that more agentic post-training can erode observation contract compliance through sycophancy-driven regression (Wang et al., 17 May 2026).

The task-level analyses sharpen this picture. The benchmark reports a universal floor on multi-turn-recall, which requires preserving an 8,192-byte URL across conversation history: every evaluated model scores 0. It also identifies inverse scaling on api-rate-limit-patience, where more capable models sometimes do worse because they retry more aggressively rather than waiting. The appendix reports GPT-4o at 0.80 on that task, while GPT-5.1 and GPT-5.2 score 0.00, with only Claude Opus 4.6 consistently clearing it (Wang et al., 17 May 2026).

ContractBench also evaluates whether its taxonomy is usable as a feedback signal. On the 42 failures of GPT-5.1, the paper reports retry success rates of 6/42 = 14.3% with no hint, 5/42 = 11.9% with a wrong-label hint, and 8/42 = 19.0% with a correct-label hint, yielding the paper’s headline +7.1 percentage point gain on paired failures. This benefit is concentrated on integrity-like failures such as WRONG_VALUE and MISSING_CONSTRAINT; temporal failures are harder to repair once time has already been lost (Wang et al., 17 May 2026).

6. Position in the benchmarking landscape, limitations, and significance

ContractBench situates itself against prior agent benchmarks such as WebArena, AgentBench, SWE-bench, ToolBench, TravelPlanner, $t_{\text{issue}}$3-bench, and Terminal-Bench, as well as temporality-oriented work such as TicToc. Its central comparative claim is narrow but consequential: these benchmarks do not jointly evaluate temporal validity and byte-level integrity of intermediate artifacts. A plausible implication is that ContractBench adds a missing deployment-critical axis to executable-agent evaluation rather than replacing broader environment benchmarks (Wang et al., 17 May 2026).

The paper is explicit about limitations. First, the virtual clock improves reproducibility but abstracts away real network jitter and live API variability. Second, there is no human baseline; instead, solvability is established by deterministic oracle solutions. Third, proprietary frontier models cannot be placed cleanly on a parameter-scaling axis because their parameter counts are undisclosed. Fourth, determinism is high but not perfect: 86.6% of $t_{\text{issue}}$4 pairs are bit-identical across the three runs, while the remaining 13.4% are concentrated in eight timing-sensitive tasks such as oauth-pkce-with-rotation, webhook-hmac-verify, and presigned-url-download (Wang et al., 17 May 2026).

The broader significance of ContractBench lies in how it reframes agent evaluation. It treats observation contract compliance as a fragile, post-training-driven inhibition capability rather than as a by-product of generic competence. The benchmark’s recommended mitigations follow directly from that diagnosis: expose TTLs in machine-readable form, use handle-based storage to avoid byte corruption in-context, add backoff middleware, and surface structured failure labels for retry policies or reinforcement learning. This suggests that the path to robust compliance is likely architectural and systems-oriented, not purely a matter of scaling model size (Wang et al., 17 May 2026).

In this sense, ContractBench defines a specialized but consequential evaluation problem: whether an agent can preserve the temporal validity and byte-level integrity of externally constrained artifacts across multi-step tool-use traces. Its empirical result is not that current agents are incapable, but that they remain systematically unreliable on this dimension. That conclusion places observation-contract compliance alongside planning, grounding, and tool selection as an independent benchmark target for agentic systems (Wang et al., 17 May 2026).