SpecSuite-Core Benchmark Protocol

• SpecSuite-Core is a benchmark protocol for evaluating the complete agent compilation workflow from behavioral specifications to test-driven prompt compilation.
• It employs detailed YAML TDAD specifications, including policies, tool schemas, decision trees, and mutation intents to stress test process robustness.
• Experimental metrics show high v1 hidden pass rates and reduced v2 compilation success, highlighting challenges in evolving specification-driven development.

Searching arXiv for the specified paper to ground the article in the referenced source. SpecSuite-Core is a benchmark protocol for evaluating agent compilation workflows rather than end-task agent performance. Introduced alongside Test-Driven AI Agent Definition (TDAD), it measures whether a workflow can reliably transform a behavioral specification into a tool-using agent artifact that satisfies the specification, generalizes beyond visible tests, resists blind spots in the test suite, and remains stable under specification change (Rehan, 9 Mar 2026). Its unit of evaluation is a deeply specified “mini-product” rather than an isolated question-answer item or a simple tool-use task, and each benchmark instance comprises a YAML behavioral specification, a generated visible test suite, a generated hidden test suite, a mutation-intent catalog for semantic mutation testing, and a $v1 \rightarrow v2$ evolution scenario (Rehan, 9 Mar 2026).

1. Definition and scope

SpecSuite-Core was introduced because TDAD requires a benchmark that evaluates the full pipeline

$$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$

not merely whether a prebuilt agent completes tasks (Rehan, 9 Mar 2026). The benchmark is therefore “deeply specified”: each task includes explicit policies, tool schemas and sequencing constraints, decision-tree logic, structured response contracts, examples clarifying ambiguous language, and mutation intents corresponding to realistic failure modes.

This design distinguishes SpecSuite-Core from benchmarks such as AgentBench, TAU-Bench, BFCL, ToolLLM, and WebArena, which evaluate agent behavior on task suites but do not evaluate the PRD/spec $\rightarrow$ test generation $\rightarrow$ prompt compilation $\rightarrow$ regression workflow (Rehan, 9 Mar 2026). A common misconception is to treat SpecSuite-Core as another end-task benchmark; the paper’s framing is narrower and more procedural. It asks whether an engineering process can produce an auditable, specification-faithful agent artifact.

The benchmark follows a “depth over breadth” principle. Instead of many shallow tasks, it contains four specifications, each with multi-turn flows, tool contracts, decision trees with 10+ branches, hidden tests comprising 40–60\% of tests, 5–7 mutation intents per spec, and an explicit evolution scenario from $v1$ to $v2$ (Rehan, 9 Mar 2026). This suggests that the benchmark is intended less as a broad leaderboard and more as a stress test for specification-driven development workflows.

2. Benchmark protocol and domains

SpecSuite-Core contains four domains chosen to span distinct classes of agent behavior regarded as central to TDAD: policy compliance, grounded analytics, runbook adherence, and deterministic enforcement (Rehan, 9 Mar 2026).

Spec	Domain type	Key behavioral challenges
SupportOps	policy compliance / customer support operations	auth before action, PII refusal, plan eligibility, escalation triggers
DataInsights	grounded analytics	SQL grounding, numeric precision, ambiguity detection, cost-aware queries
IncidentRunbook	runbook adherence / operational compliance	evidence-first ordering, severity routing, runbook compliance
ExpenseGuard	deterministic enforcement / approval policy	spending caps, FX conversion, receipt requirements, manager approval

These domains are not merely topical labels. Each one anchors a different control problem over tool-using behavior. SupportOps emphasizes policy ordering such as authentication and PII refusal; DataInsights constrains analytic grounding and ambiguity handling; IncidentRunbook formalizes evidence-first operational behavior; ExpenseGuard enforces rule-based approval logic (Rehan, 9 Mar 2026). The selection reflects the claim that production agents often fail through process violations, tool misuse, or policy regressions rather than only through incorrect final answers.

The benchmark’s reported quantitative depth, given as median counts across three stochastic generations, is as follows (Rehan, 9 Mar 2026).

Spec	Nodes	V1 Visible	V1 Hidden
SupportOps	12	47	45
DataInsights	10	34	42
IncidentRunbook	14	39	42
ExpenseGuard	11	47	45

The same report also gives $v2$ visible and hidden counts and mutation-intent counts: SupportOps $53/43/7$, DataInsights $53/43/6$, IncidentRunbook $$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$0, and ExpenseGuard $$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$1, with counts varying by about $$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$2 because tests are regenerated stochastically each trial (Rehan, 9 Mar 2026). This variability is not incidental; it is part of the benchmark design, since fresh generation is used to evaluate workflow robustness under stochastic test synthesis.

3. Behavioral specification format

Each SpecSuite-Core task is represented as a YAML TDAD specification, described as the “single source of truth” (Rehan, 9 Mar 2026). The YAML encodes six components: Tools, Policies, Decision tree, Response contract, Test guidance, and Mutation intents. Concretely, the specification records tool names, schemas, failure modes, and sequencing constraints; behavioral rules with priorities; branch conditions and required actions; structured output requirements via a respond tool called exactly once per turn; examples clarifying ambiguous policy language; and failure modes that the suite should detect.

The specification constrains behavioral process, not only output correctness. Tool order may matter; some tools may be mandatory or forbidden; response fields must follow a schema; policy priorities may dominate helpfulness; and action gating such as identity verification or runbook lookup can be obligatory (Rehan, 9 Mar 2026). Because the benchmark is trace-oriented, these process constraints are first-class evaluation targets.

The paper gives an example of test_guidance used to operationalize ambiguous policy language:

$v2$8

This matters because the benchmark derives tests from the specification. Without explicit disambiguation guidance, generated tests may conflict (Rehan, 9 Mar 2026). A plausible implication is that SpecSuite-Core treats specification quality itself as part of the experimental object: poorly specified behavior propagates into unstable tests and prompt oscillation.

4. From specification to executable benchmark instances

In TDAD, SpecSuite-Core specifications are converted into executable tests by TestSmith, under the rule: “If you cannot cite the specific clause that mandates the behavior, do not write the test.” The conversion procedure is four-stage: traverse the decision tree and generate one MFT (minimum functionality test) per leaf node; generate INV variants by paraphrasing user messages while preserving intent; generate DIR tests from paired inputs differing only in the condition under study; and create deterministic tool fixtures with canary values and unique identifiers (Rehan, 9 Mar 2026).

As a result, the executable tests do not merely check final answers. They can assert required or forbidden tool usage, call ordering, schema-valid structured outputs, response fields such as decision, node_id, evidence, and user_message, grounding properties such as “must call run_sql before answering,” policy properties such as refusing PII, and quantitative properties such as numeric precision (Rehan, 9 Mar 2026). The dedicated respond tool, required exactly once per turn, enables evaluation over tool traces and structured fields rather than heuristic natural-language parsing.

The prompt artifact is then compiled by PromptSmith, which starts from an initial prompt $$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$3 and iteratively edits it based on visible test failures. The resulting compiled artifact includes both the refined system prompt and tool description overrides, with tool descriptions treated as first-class artifacts because they strongly influence tool-use behavior (Rehan, 9 Mar 2026). The loop runs visible tests on the current prompt, returns if all pass, otherwise analyzes failures; if fewer than $$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$4 tests fail, with default $$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$5, it enters a focused inner loop rerunning only failing tests; otherwise it applies broader edits and reruns. Convergence is reported to occur typically in 2–5 iterations, under a budget of 6 outer-loop iterations and 8 inner-loop attempts (Rehan, 9 Mar 2026).

This benchmark pipeline is inseparable from its anti-gaming mechanisms. SpecSuite-Core includes visible/hidden splits, semantic mutation catalogs, and $$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$6 evolutions precisely because the object under test is not only the final agent but the compilation workflow’s ability to generalize, detect regressions, and tolerate requirement change.

5. Evaluation methodology and formal metrics

SpecSuite-Core partitions tests into Visible tests and Hidden tests. Visible tests comprise 40–70\% of the suite and are used during compilation; hidden tests comprise 30–60\% and are withheld until reporting (Rehan, 9 Mar 2026). For each decision branch, the primary MFT is usually visible while many INV and DIR variants are hidden. Hidden tests also include paraphrase variants, boundary conditions, and metamorphic tests. In benchmark mode, tests are regenerated fresh each trial and the hidden set is then frozen within that trial.

Semantic mutation testing is implemented by MutationSmith. After compilation, it takes the prompt $$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$7 and, for each mutation intent $$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$8, produces a semantically faulty prompt $$\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},$$9 such as SKIP_AUTH_GATE, LEAK_PII_ON_DIRECT_REQUEST, HALLUCINATE_NUMBERS, or SKIP_RUNBOOK_LOOKUP (Rehan, 9 Mar 2026). An activation probe checks that the mutation actually changes behavior; if activation fails after $\rightarrow$0 attempts by default, the mutant is marked non-activating and excluded. The visible suite is then run on the valid mutants. This mutation process is behavioral rather than syntactic: activation probes may inspect tool traces, text predicates, or JSON fields.

Each specification also includes a $\rightarrow$1 evolution. Version $\rightarrow$2 introduces 2–3 coordinated changes such as a new tool and branch, a stricter policy, or a schema change. Importantly, $\rightarrow$3 compilation begins from the $\rightarrow$4 prompt artifact and compiles only against $\rightarrow$5 tests; withheld $\rightarrow$6 invariant tests are used afterward to measure backward compatibility (Rehan, 9 Mar 2026). This operationalizes regression safety under changing requirements.

The paper defines the main evaluation metrics as follows (Rehan, 9 Mar 2026):

$\rightarrow$7

$\rightarrow$8

$\rightarrow$9

$\rightarrow$0

$\rightarrow$1

Here $\rightarrow$2 is computed over activating mutants only, and $\rightarrow$3 is recommended for production but not evaluated in this study (Rehan, 9 Mar 2026). The benchmark therefore measures compilation success, hidden-test generalization, visible-suite sensitivity to plausible regressions, and backward compatibility under specification evolution.

6. Experimental configuration and reported outcomes

The experimental setup comprises 4 specs, each with 2 versions and 3 independent trials, for a total of

$\rightarrow$4

All roles—TestSmith, PromptSmith, MutationSmith, and the built agent—use Claude Sonnet 4.5, model ID claude-sonnet-4-5-20250929, with default temperature settings, no explicit temperature override, and no retry policy for individual test assertions (Rehan, 9 Mar 2026). Agents interact through MCP tools and must return final output via respond exactly once per turn. The harness uses pytest, parallel execution with -n auto, deterministic fixtures, and assertion helpers for tool-call traces, structured output validation, PII canary detection, and numeric grounding.

Isolation is a major control. The compilation container mounts only tests_visible/; hidden tests reside in a separate Docker volume; visible tests are read-only during PromptSmith compilation; and hidden tests run in a separate evaluation container afterward (Rehan, 9 Mar 2026). The main procedural controls are thus withheld hidden tests, a separate mutation phase, withheld $\rightarrow$5 invariants during $\rightarrow$6 compilation, and deterministic non-LLM harness components. The paper explicitly states that it does not ablate the anti-gaming mechanisms individually and does not compare against systems such as DSPy, TextGrad, or APE (Rehan, 9 Mar 2026).

Across successful runs, the principal aggregate outcomes are: v1 compilation success $\rightarrow$7, mean v1 hidden pass rate $\rightarrow$8, v2 compilation success $\rightarrow$9, mean v2 hidden pass rate $\rightarrow$0, v1 mutation score $\rightarrow$1, v2 mutation score $\rightarrow$2, and regression safety $\rightarrow$3 (Rehan, 9 Mar 2026). Per specification, SupportOps achieved $\rightarrow$4 $\rightarrow$5 compilations and $\rightarrow$6 $\rightarrow$7; DataInsights $\rightarrow$8 and $\rightarrow$9; IncidentRunbook $v1$0 and $v1$1; ExpenseGuard $v1$2 and $v1$3 (Rehan, 9 Mar 2026).

Failure analysis is also reported. Among $v1$4 failures, 2 runs were due to conflicting tests generated by TestSmith and 3 runs to PromptSmith exhausting the iteration budget (Rehan, 9 Mar 2026). Yet these were near-misses: SupportOps/$v1$5 reached 52/53 visible tests, IncidentRunbook/$v1$6 48/49, ExpenseGuard/$v1$7 44/46, and IncidentRunbook/$v1$8 36/38. The paper describes a recurring failure pattern of oscillation, in which fixing one test breaks another, suggesting conflicting expectations caused by ambiguous specification language.

Mutation testing exposes both strength and blind spots. Selected results include consistent killing of SKIP_AUTH_GATE, LEAK_PII_DIRECT_REQ, SKIP_CONFIRM_CANCEL, ANSWER_WITHOUT_SQL, SKIP_CLARIFY_AMBIG, FAIL_TO_PAGE_SEV1, APPROVE_NO_RECEIPT, and SKIP_FX_CONVERSION; however, HALLUCINATE_NUMBERS in DataInsights survived in 1/3 $v1$9 runs, and SKIP_RUNBOOK_LOOKUP in IncidentRunbook survived in 2/2 successful $v2$0 runs before being fixed in $v2$1 (Rehan, 9 Mar 2026). Across experiments, 87\% of mutation intents successfully activated, with non-activating mutants excluded from mutation score computation.

7. Limitations and interpretation

The paper identifies several limitations specific to SpecSuite-Core. First, coverage is incomplete because TDAD assumes desired behavior can be expressed as executable tests; properties such as empathy are difficult to specify precisely (Rehan, 9 Mar 2026). Mutation testing improves confidence in suite quality but does not prove completeness, and excluding non-activating mutants may overstate test adequacy if some excluded mutants correspond to real blind spots.

Second, benchmark outcomes are stochastic rather than single deterministic scores. Test generation and prompt compilation are both stochastic; reported variance is about $v2$2 for $v2$3 hidden pass rate and $v2$4 for $v2$5 (Rehan, 9 Mar 2026). Third, the benchmark incurs nontrivial overhead: each spec version requires about 30–60 minutes wall-clock and roughly \$2–3 in API cost (Rehan, 9 Mar 2026). Fourth, the paper notes possible LLM self-censorship in test generation, where TestSmith may avoid strongly adversarial or profane inputs, limiting abuse-detection coverage. Fifth, the benchmark scope remains narrow: only four specifications with roughly 10–14 decision nodes, no scaling study to 50+ nodes, no cross-model analysis, and no component-wise ablation (Rehan, 9 Mar 2026).

Within those limits, SpecSuite-Core is best understood as an engineering benchmark for specification-driven agent development. Its central contribution is not simply the presence of four tasks but the fact that each task is embedded in a protocol that evaluates compilation, hidden-test generalization, mutation sensitivity, and regression safety under evolving requirements (Rehan, 9 Mar 2026). The reported $v2$6 results—92\% compilation success and 97.3\% hidden pass rate—indicate that this workflow can often compile deeply specified tool-using agents successfully, while the harder $v2$7 results—58\% compilation success, 77.7\% hidden pass rate, and 97.2\% regression safety—indicate that requirement evolution remains the more demanding regime. This suggests that SpecSuite-Core is less a benchmark of agent capability in isolation than a benchmark of whether specification, testing, and prompt compilation can form a reliable development discipline for tool-using LLM agents.