---
title: ContractBench Compliance Benchmark
url: https://www.emergentmind.com/topics/contractbench
type: topic
---

# ContractBench Compliance Benchmark

ContractBench is a benchmark for **observation contract compliance** in tool-augmented LLM agents: it studies whether an agent can preserve externally constrained intermediate artifacts—such as presigned URLs, session tokens, OAuth state parameters, signed request fields, HMAC-protected webhook payloads, ETags, idempotency keys, rate-limit windows, and multi-step token chains—across the observation-to-action pipeline. Its central claim is that this capability is distinct from generic tool-use competence: an agent may call the correct APIs in the correct broad order and still fail because it uses an artifact after expiry or corrupts its bytes in transit. The benchmark introduces 33 deterministic, programmatically graded tasks organized around two orthogonal failure modes, **validity** and **integrity**, and reports that no evaluated model exceeds 80% success on the full suite [2605.17281].

## 1. Conceptual foundation

ContractBench defines **observation contracts** as artifacts whose later use is constrained by the external system that produced them. The benchmark focuses on two independent constraints. The first is **temporal validity**: an artifact may be usable only within a bounded time window, as with TTL-limited presigned URLs, refreshable tokens, scheduled maintenance windows, or `Retry-After` backoff intervals. The second is **byte-level integrity**: an artifact may have to round-trip exactly, as with OAuth state parameters, signed query strings, HMAC inputs, conditional request tokens, or canonicalized request payloads [2605.17281].

This framing separates ContractBench from ordinary tool-use benchmarks that treat intermediate outputs as advisory. In the ContractBench view, many real APIs do not tolerate paraphrase, normalization, truncation, or deferred reuse. A presigned URL may fail if a query parameter is reordered; an OAuth state parameter may fail if even a single byte changes; a token refresh may fail if the refreshed token is then mutated; a well-preserved token may still fail if used after expiry. The benchmark therefore treats **restraint**—waiting when required, avoiding premature retries, preserving opaque strings exactly, and refusing shortcut flows—as a first-class capability rather than a side effect of general reasoning [2605.17281].

The paper argues that this capability is both **emergent** and **regression-prone**. It is emergent because base checkpoints score 0% on the full suite; it is regression-prone because post-training can improve or erode compliance depending on how it affects byte preservation, temporal planning, and shortcut avoidance. This suggests that observation contract compliance is not reducible to scale alone, nor guaranteed by stronger general agent performance [2605.17281].

## 2. Formal model of observation contracts

The benchmark formalizes an observation contract as

$$
C = (o,\; t_{\text{issue}},\; \tau,\; \pi),
$$

where \(o\) is the issued artifact, \(t_{\text{issue}}\) is its issue time, \(\tau > 0\) is its time-to-live, and \(\pi : \mathcal{O} \to \{0,1\}\) is an integrity predicate. The corresponding validity window is

$$
W(C) := [\,t_{\text{issue}},\; t_{\text{issue}} + \tau\,).
$$

In the implementation described by the paper, integrity is usually checked by SHA-256 equality:

$$
\pi(o') = \mathbbm{1}[\,SHA(o') = h(o)\,].
$$

An artifact later submitted by the agent satisfies the contract only if it is used within the validity window and still satisfies the integrity predicate. The paper writes the corresponding indicator as \(\operatorname{Sat}(C,o',t') \in \{0,1\}\). For a multi-step trace with contracts \(\{C_i\}_{i=1}^n\), overall compliance is defined multiplicatively:

$$
\operatorname{Compliant}(\{C_i\}, \{(o'_i, t'_i)\})
:=
\prod_{i=1}^{n} \operatorname{Sat}(C_i, o'_i, t'_i) = 1.
$$

Under this formulation, a single violated contract anywhere in the trace makes the episode non-compliant [2605.17281].

This formalization supports a clean distinction between the two failure axes. A **validity failure** occurs when the agent uses an artifact outside \(W(C)\). An **integrity failure** occurs when the later artifact \(o'\) does not satisfy \(\pi\). The paper emphasizes that these are logically independent: byte-perfect relay does not rescue a stale token, and prompt refresh does not rescue a token that is then normalized, truncated, or re-encoded incorrectly [2605.17281].

## 3. Benchmark construction and task suite

ContractBench contains **33 dual-axis tasks**. The suite is organized into four quadrants: Q1 for low validity pressure and low integrity pressure; Q2 for temporal-validity pressure isolated; Q3 for byte-integrity pressure isolated; and Q4 for joint pressure on both axes. The suite is heavily weighted toward Q4: **24 of the 33 tasks** are dual-pressure tasks, with **4 Q2** tasks and **5 Q3** tasks [2605.17281].

The task catalog spans patterns such as Timing/Backoff, Byte-exact, OAuth/Auth, Signed requests, State chains, Resource management, and Multi-service. Named tasks include `scheduled-maintenance`, `api-rate-limit-patience`, `token-refresh-workflow`, `presigned-url-download`, `csrf-form-submit`, `url-trap-ellipsis`, `long-token-handling`, `extreme-url-length`, `webhook-hmac-verify`, `basic-oauth-token`, `oauth-authorization-code`, `oauth-pkce-with-rotation`, `signed-request-canonicalization`, `session-cookie-chain`, `etag-conditional-get`, `idempotency-key-retry`, `multi-service-saga`, and `cascading-token-revocation` [2605.17281].

Each task is packaged as four files: a hidden TOML metadata header, an agent-visible Markdown instruction, a FastAPI server implementing task logic and logging violation events, and a hidden pytest validator that consumes the log and computes the canonical reward. The agent sees only the instruction and HTTP responses from the task server. This design excludes LLM-as-judge ambiguity and makes the benchmark fully programmatic [2605.17281].

The paper also describes five deterministic **production path mutations** that model common integrity failures in deployed agent stacks: truncation, line-wrap insertion, URL re-encoding, query reorder, and UI trap. These mutations reproduce corruption modes such as clipping long artifacts, inserting newlines into long tokens, normalizing `%2F` into `/`, re-sorting signed parameters, or exposing link text that differs from the underlying href. Their role is methodological: they turn otherwise anecdotal “agent flakiness” into reproducible integrity stressors [2605.17281].

## 4. Evaluation protocol and failure taxonomy

ContractBench is evaluated under a **deterministic and programmatic** protocol. Time is controlled by a **virtual clock**, all task-instance stochasticity is seed-fixed, temperature is set to 0, model IDs are pinned, API parameters are logged, and byte integrity is checked with SHA-256. Each episode produces an HTTP request log, and the hidden validator writes a structured `reward.json` containing a binary success flag, a primary failure label, detailed failure metadata, and trace metadata including seed, step count, tool-call count, virtual time elapsed, and a SHA-256 trace hash [2605.17281].

The main metric is **success rate (SR%)**. Each model is run on all 33 tasks with **\(k=3\)** runs per task, yielding **\(n=99\)** episodes per model. The paper also reports per-task pass rate and the distribution of primary failure labels across failed episodes [2605.17281].

The failure taxonomy contains **15 labels**: **4 validity failures**, **9 integrity failures**, and **2 meta labels**. The validity labels are `EXPIRED_BEFORE_USE`, `RATE_LIMITED`, `SCHEDULED_UNAVAILABLE`, and `VERSION_CONFLICT`. The integrity labels are `WRONG_VALUE`, `MISSING_CONSTRAINT`, `MUTATED_TOKEN`, `SIGNATURE_MISMATCH`, `COMPENSATION_FAILURE`, `WRONG_HASH`, `SHORTCUT_TAKEN`, `MISSING_TOKEN`, and `BACKOFF_VIOLATION`. The meta labels are `SUCCESS` and `OTHER` [2605.17281].

When multiple violations occur in one episode, the validator assigns one **primary label** using a **most-severe-label rule**. The appendix also supplies a severity-weighted mapping reused in the intervention experiment, including \(+1.0\) for `SUCCESS`, \(-1.0\) for `EXPIRED_BEFORE_USE`, \(-1.0\) for `MUTATED_TOKEN`, \(-0.9\) for `SHORTCUT_TAKEN`, \(-0.8\) for `WRONG_VALUE`, \(-0.7\) for `MISSING_CONSTRAINT`, \(-0.5\) for `RATE_LIMITED`, and \(-0.3\) for `OTHER` [2605.17281].

This taxonomy is not merely descriptive. The paper ties the labels to recognizable API and protocol failures, citing OAuth 2.0, RFC 6585, RFC 9110, RFC 7232, RFC 3986, RFC 9421, AWS SigV4, RFC 9530, FIPS 180-4, and saga-style compensation failures. The benchmark therefore evaluates agents against failure surfaces that correspond to deployed systems rather than to synthetic benchmark-only categories [2605.17281].

## 5. Empirical findings

ContractBench evaluates **38 model variants from 15 families** and reports that **no evaluated model clears 80%** on the full suite. The top model is **Claude Opus 4.6** at **77.8%**, or **77/99** successful episodes. Other frontier results reported in the main text include **GPT-5.2** at **74.8%**, **GPT-5** at **70.7%**, **Qwen3.5-397B-A17B** at **70.7%**, **Claude Sonnet 4.5** at **69.7%**, **MiniMax-M2.5** at **62.6%**, **MiniMax-M2.1** at **60.6%**, **Qwen3.5-9B** at **56.6%**, **Gemini 2.5 Pro** at **51.5%**, **GPT-5.1** at **48.5%**, **Mistral-Small-4** at **42.4%**, **Gemma-4-26B-A4B** at **38.4%**, **Gemma-4-31B** at **37.4%**, **Ministral-3-14B** at **28.3%**, **GPT-4o** at **23.2%**, **Qwen2.5-72B-Instruct** at **23.2%**, **Gemini 2.5 Flash** at **19.2%**, and **Llama-3.3-70B-Instruct** at **7.1%** [2605.17281].

One of the paper’s most prominent findings is a **Qwen 3.5 capability cliff**. Within that family, performance jumps from **0%** at **4B Instruct** to **56.6%** at **9B Instruct**, then rises to **64.6%** at **27B** and **70.7%** at **397B-A17B**. The authors argue that what emerges across this cliff is not tool-call emission but **mid-trajectory restraint**: waiting through maintenance windows, respecting backoff, changing strategy after failure, and persisting through multi-step workflows without corrupting or prematurely consuming artifacts [2605.17281].

A second headline finding is **non-monotonic scaling across the GPT-5 family**. The paper reports approximately **23.2%** for **GPT-4o**, **70.7%** for **GPT-5**, **48.5%** for **GPT-5.1**, and **74.7/74.8%** for **GPT-5.2**. The regression from GPT-5 to GPT-5.1 is concentrated on the integrity axis, especially `WRONG_VALUE` and `MUTATED_TOKEN`, and the paper interprets this as evidence that more agentic post-training can erode observation contract compliance through **sycophancy-driven regression** [2605.17281].

The task-level analyses sharpen this picture. The benchmark reports a **universal floor** on `multi-turn-recall`, which requires preserving an **8,192-byte URL across conversation history**: every evaluated model scores **0**. It also identifies **inverse scaling** on `api-rate-limit-patience`, where more capable models sometimes do worse because they retry more aggressively rather than waiting. The appendix reports GPT-4o at **0.80** on that task, while GPT-5.1 and GPT-5.2 score **0.00**, with only Claude Opus 4.6 consistently clearing it [2605.17281].

ContractBench also evaluates whether its taxonomy is usable as a feedback signal. On the **42 failures of GPT-5.1**, the paper reports retry success rates of **6/42 = 14.3%** with no hint, **5/42 = 11.9%** with a wrong-label hint, and **8/42 = 19.0%** with a correct-label hint, yielding the paper’s headline **+7.1 percentage point** gain on paired failures. This benefit is concentrated on integrity-like failures such as `WRONG_VALUE` and `MISSING_CONSTRAINT`; temporal failures are harder to repair once time has already been lost [2605.17281].

## 6. Position in the benchmarking landscape, limitations, and significance

ContractBench situates itself against prior agent benchmarks such as WebArena, AgentBench, SWE-bench, ToolBench, TravelPlanner, \(\tau\)-bench, and Terminal-Bench, as well as temporality-oriented work such as TicToc. Its central comparative claim is narrow but consequential: these benchmarks do not jointly evaluate **temporal validity** and **byte-level integrity** of intermediate artifacts. A plausible implication is that ContractBench adds a missing deployment-critical axis to executable-agent evaluation rather than replacing broader environment benchmarks [2605.17281].

The paper is explicit about limitations. First, the **virtual clock** improves reproducibility but abstracts away real network jitter and live API variability. Second, there is **no human baseline**; instead, solvability is established by deterministic oracle solutions. Third, proprietary frontier models cannot be placed cleanly on a parameter-scaling axis because their parameter counts are undisclosed. Fourth, determinism is high but not perfect: **86.6%** of \((\text{model}, \text{task})\) pairs are bit-identical across the three runs, while the remaining **13.4%** are concentrated in eight timing-sensitive tasks such as `oauth-pkce-with-rotation`, `webhook-hmac-verify`, and `presigned-url-download` [2605.17281].

The broader significance of ContractBench lies in how it reframes agent evaluation. It treats observation contract compliance as a **fragile, post-training-driven inhibition capability** rather than as a by-product of generic competence. The benchmark’s recommended mitigations follow directly from that diagnosis: expose TTLs in machine-readable form, use handle-based storage to avoid byte corruption in-context, add backoff middleware, and surface structured failure labels for retry policies or reinforcement learning. This suggests that the path to robust compliance is likely architectural and systems-oriented, not purely a matter of scaling model size [2605.17281].

In this sense, ContractBench defines a specialized but consequential evaluation problem: whether an agent can preserve the temporal validity and byte-level integrity of externally constrained artifacts across multi-step tool-use traces. Its empirical result is not that current agents are incapable, but that they remain systematically unreliable on this dimension. That conclusion places observation-contract compliance alongside planning, grounding, and tool selection as an independent benchmark target for agentic systems [2605.17281].

Source: https://www.emergentmind.com/topics/contractbench