---
title: SpecSuite-Core Benchmark Protocol
url: https://www.emergentmind.com/topics/specsuite-core
type: topic
---

# SpecSuite-Core Benchmark Protocol

Searching arXiv for the specified paper to ground the article in the referenced source.
SpecSuite-Core is a benchmark protocol for evaluating **agent compilation workflows** rather than end-task agent performance. Introduced alongside Test-Driven AI Agent Definition (TDAD), it measures whether a workflow can reliably transform a behavioral specification into a tool-using agent artifact that satisfies the specification, generalizes beyond visible tests, resists blind spots in the test suite, and remains stable under specification change [2603.08806]. Its unit of evaluation is a deeply specified “mini-product” rather than an isolated question-answer item or a simple tool-use task, and each benchmark instance comprises a YAML behavioral specification, a generated visible test suite, a generated hidden test suite, a mutation-intent catalog for semantic mutation testing, and a \(v1 \rightarrow v2\) evolution scenario [2603.08806].

## 1. Definition and scope

SpecSuite-Core was introduced because TDAD requires a benchmark that evaluates the full pipeline
\[
\text{specification} \rightarrow \text{tests} \rightarrow \text{compiled prompt} \rightarrow \text{generalization / regression evaluation},
\]
not merely whether a prebuilt agent completes tasks [2603.08806]. The benchmark is therefore “deeply specified”: each task includes explicit policies, tool schemas and sequencing constraints, decision-tree logic, structured response contracts, examples clarifying ambiguous language, and mutation intents corresponding to realistic failure modes.

This design distinguishes SpecSuite-Core from benchmarks such as AgentBench, TAU-Bench, BFCL, ToolLLM, and WebArena, which evaluate agent behavior on task suites but do not evaluate the **PRD/spec \(\rightarrow\) test generation \(\rightarrow\) prompt compilation \(\rightarrow\) regression** workflow [2603.08806]. A common misconception is to treat SpecSuite-Core as another end-task benchmark; the paper’s framing is narrower and more procedural. It asks whether an engineering process can produce an auditable, specification-faithful agent artifact.

The benchmark follows a “depth over breadth” principle. Instead of many shallow tasks, it contains four specifications, each with multi-turn flows, tool contracts, decision trees with **10+ branches**, hidden tests comprising **40–60\%** of tests, **5–7 mutation intents per spec**, and an explicit evolution scenario from \(v1\) to \(v2\) [2603.08806]. This suggests that the benchmark is intended less as a broad leaderboard and more as a stress test for specification-driven development workflows.

## 2. Benchmark protocol and domains

SpecSuite-Core contains four domains chosen to span distinct classes of agent behavior regarded as central to TDAD: policy compliance, grounded analytics, runbook adherence, and deterministic enforcement [2603.08806].

| Spec | Domain type | Key behavioral challenges |
|---|---|---|
| SupportOps | policy compliance / customer support operations | auth before action, PII refusal, plan eligibility, escalation triggers |
| DataInsights | grounded analytics | SQL grounding, numeric precision, ambiguity detection, cost-aware queries |
| IncidentRunbook | runbook adherence / operational compliance | evidence-first ordering, severity routing, runbook compliance |
| ExpenseGuard | deterministic enforcement / approval policy | spending caps, FX conversion, receipt requirements, manager approval |

These domains are not merely topical labels. Each one anchors a different control problem over tool-using behavior. SupportOps emphasizes policy ordering such as authentication and PII refusal; DataInsights constrains analytic grounding and ambiguity handling; IncidentRunbook formalizes evidence-first operational behavior; ExpenseGuard enforces rule-based approval logic [2603.08806]. The selection reflects the claim that production agents often fail through process violations, tool misuse, or policy regressions rather than only through incorrect final answers.

The benchmark’s reported quantitative depth, given as median counts across three stochastic generations, is as follows [2603.08806].

| Spec | Nodes | V1 Visible | V1 Hidden |
|---|---:|---:|---:|
| SupportOps | 12 | 47 | 45 |
| DataInsights | 10 | 34 | 42 |
| IncidentRunbook | 14 | 39 | 42 |
| ExpenseGuard | 11 | 47 | 45 |

The same report also gives \(v2\) visible and hidden counts and mutation-intent counts: SupportOps \(53/43/7\), DataInsights \(53/43/6\), IncidentRunbook \(42/45/7\), and ExpenseGuard \(46/42/7\), with counts varying by about \(\pm 10\text{–}30\%\) because tests are regenerated stochastically each trial [2603.08806]. This variability is not incidental; it is part of the benchmark design, since fresh generation is used to evaluate workflow robustness under stochastic test synthesis.

## 3. Behavioral specification format

Each SpecSuite-Core task is represented as a **YAML TDAD specification**, described as the “single source of truth” [2603.08806]. The YAML encodes six components: **Tools**, **Policies**, **Decision tree**, **Response contract**, **Test guidance**, and **Mutation intents**. Concretely, the specification records tool names, schemas, failure modes, and sequencing constraints; behavioral rules with priorities; branch conditions and required actions; structured output requirements via a `respond` tool called exactly once per turn; examples clarifying ambiguous policy language; and failure modes that the suite should detect.

The specification constrains **behavioral process**, not only output correctness. Tool order may matter; some tools may be mandatory or forbidden; response fields must follow a schema; policy priorities may dominate helpfulness; and action gating such as identity verification or runbook lookup can be obligatory [2603.08806]. Because the benchmark is trace-oriented, these process constraints are first-class evaluation targets.

The paper gives an example of `test_guidance` used to operationalize ambiguous policy language:

```yaml
- id: P3_CLARIFY_AMBIGUITY
  text: If request is ambiguous, ask one clarifying question first.
  test_guidance:
    description: |
      Ambiguity is about WHAT data, not HOW MUCH.
      Top-k queries can use reasonable defaults.
    ambiguous_examples:
      - "Show me the data"
      - "What are the numbers?"
    unambiguous_examples:
      - "What is total revenue?"
      - "Show me top 10 customers"
```

This matters because the benchmark derives tests from the specification. Without explicit disambiguation guidance, generated tests may conflict [2603.08806]. A plausible implication is that SpecSuite-Core treats specification quality itself as part of the experimental object: poorly specified behavior propagates into unstable tests and prompt oscillation.

## 4. From specification to executable benchmark instances

In TDAD, SpecSuite-Core specifications are converted into executable tests by **TestSmith**, under the rule: *“If you cannot cite the specific clause that mandates the behavior, do not write the test.”* The conversion procedure is four-stage: traverse the decision tree and generate one **MFT** (minimum functionality test) per leaf node; generate **INV** variants by paraphrasing user messages while preserving intent; generate **DIR** tests from paired inputs differing only in the condition under study; and create deterministic tool fixtures with canary values and unique identifiers [2603.08806].

As a result, the executable tests do not merely check final answers. They can assert required or forbidden tool usage, call ordering, schema-valid structured outputs, response fields such as `decision`, `node_id`, `evidence`, and `user_message`, grounding properties such as “must call `run_sql` before answering,” policy properties such as refusing PII, and quantitative properties such as numeric precision [2603.08806]. The dedicated `respond` tool, required exactly once per turn, enables evaluation over tool traces and structured fields rather than heuristic natural-language parsing.

The prompt artifact is then compiled by **PromptSmith**, which starts from an initial prompt \(P_0\) and iteratively edits it based on visible test failures. The resulting compiled artifact includes both the refined **system prompt** and **tool description overrides**, with tool descriptions treated as first-class artifacts because they strongly influence tool-use behavior [2603.08806]. The loop runs visible tests on the current prompt, returns if all pass, otherwise analyzes failures; if fewer than \(\theta\) tests fail, with default \(\theta = 10\), it enters a focused inner loop rerunning only failing tests; otherwise it applies broader edits and reruns. Convergence is reported to occur typically in **2–5 iterations**, under a budget of **6 outer-loop iterations** and **8 inner-loop attempts** [2603.08806].

This benchmark pipeline is inseparable from its anti-gaming mechanisms. SpecSuite-Core includes visible/hidden splits, semantic mutation catalogs, and \(v1 \rightarrow v2\) evolutions precisely because the object under test is not only the final agent but the compilation workflow’s ability to generalize, detect regressions, and tolerate requirement change.

## 5. Evaluation methodology and formal metrics

SpecSuite-Core partitions tests into **Visible tests** and **Hidden tests**. Visible tests comprise **40–70\%** of the suite and are used during compilation; hidden tests comprise **30–60\%** and are withheld until reporting [2603.08806]. For each decision branch, the primary MFT is usually visible while many INV and DIR variants are hidden. Hidden tests also include paraphrase variants, boundary conditions, and metamorphic tests. In benchmark mode, tests are regenerated fresh each trial and the hidden set is then frozen within that trial.

Semantic mutation testing is implemented by **MutationSmith**. After compilation, it takes the prompt \(P\) and, for each mutation intent \(m_i\), produces a semantically faulty prompt \(P_{m_i}\) such as `SKIP_AUTH_GATE`, `LEAK_PII_ON_DIRECT_REQUEST`, `HALLUCINATE_NUMBERS`, or `SKIP_RUNBOOK_LOOKUP` [2603.08806]. An activation probe checks that the mutation actually changes behavior; if activation fails after \(k=5\) attempts by default, the mutant is marked non-activating and excluded. The visible suite is then run on the valid mutants. This mutation process is behavioral rather than syntactic: activation probes may inspect tool traces, text predicates, or JSON fields.

Each specification also includes a \(v1 \rightarrow v2\) evolution. Version \(v2\) introduces **2–3 coordinated changes** such as a new tool and branch, a stricter policy, or a schema change. Importantly, \(v2\) compilation begins from the \(v1\) prompt artifact and compiles only against \(v2\) tests; withheld \(v1\) invariant tests are used afterward to measure backward compatibility [2603.08806]. This operationalizes regression safety under changing requirements.

The paper defines the main evaluation metrics as follows [2603.08806]:

\[
VPR = \frac{|\{t \in T_{\text{visible}} : t \text{ passes}\}|}{|T_{\text{visible}}|}
\]

\[
HPR = \frac{|\{t \in T_{\text{hidden}} : t \text{ passes}\}|}{|T_{\text{hidden}}|}
\]

\[
MS = \frac{|\{m \in M' : \exists t \in T_{\text{vis}},\ t \text{ fails on } P_m\}|}{|M'|}
\]

\[
SURS = \frac{|\{t \in T^{v1}_{\text{inv}} : t \text{ passes on } A^{v2}\}|}{|T^{v1}_{\text{inv}}|}
\]

\[
RPR_t = \frac{\sum_{i=1}^{N} \mathbf{1}[t \text{ passes on run } i]}{N}
\]

Here \(MS\) is computed over activating mutants only, and \(RPR\) is recommended for production but **not evaluated in this study** [2603.08806]. The benchmark therefore measures compilation success, hidden-test generalization, visible-suite sensitivity to plausible regressions, and backward compatibility under specification evolution.

## 6. Experimental configuration and reported outcomes

The experimental setup comprises **4 specs**, each with **2 versions** and **3 independent trials**, for a total of
\[
4 \times 2 \times 3 = 24 \text{ runs}.
\]
All roles—TestSmith, PromptSmith, MutationSmith, and the built agent—use **Claude Sonnet 4.5**, model ID `claude-sonnet-4-5-20250929`, with default temperature settings, no explicit temperature override, and no retry policy for individual test assertions [2603.08806]. Agents interact through MCP tools and must return final output via `respond` exactly once per turn. The harness uses **pytest**, parallel execution with `-n auto`, deterministic fixtures, and assertion helpers for tool-call traces, structured output validation, PII canary detection, and numeric grounding.

Isolation is a major control. The compilation container mounts only `tests_visible/`; hidden tests reside in a separate Docker volume; visible tests are read-only during PromptSmith compilation; and hidden tests run in a separate evaluation container afterward [2603.08806]. The main procedural controls are thus withheld hidden tests, a separate mutation phase, withheld \(v1\) invariants during \(v2\) compilation, and deterministic non-LLM harness components. The paper explicitly states that it does **not ablate** the anti-gaming mechanisms individually and does **not compare** against systems such as DSPy, TextGrad, or APE [2603.08806].

Across successful runs, the principal aggregate outcomes are: **v1 compilation success** \(11/12 = 92\%\), **mean v1 hidden pass rate** \(97.3\% \pm 2.6\%\), **v2 compilation success** \(7/12 = 58\%\), **mean v2 hidden pass rate** \(77.7\% \pm 13.9\%\), **v1 mutation score** \(95.9\% \pm 7.1\%\), **v2 mutation score** \(100\%\), and **regression safety** \(97.2\% \pm 3.5\%\) [2603.08806]. Per specification, SupportOps achieved \(3/3\) \(v1\) compilations and \(2/3\) \(v2\); DataInsights \(3/3\) and \(2/3\); IncidentRunbook \(2/3\) and \(2/3\); ExpenseGuard \(3/3\) and \(1/3\) [2603.08806].

Failure analysis is also reported. Among \(v2\) failures, **2 runs** were due to conflicting tests generated by TestSmith and **3 runs** to PromptSmith exhausting the iteration budget [2603.08806]. Yet these were near-misses: SupportOps/\(v2\) reached **52/53** visible tests, IncidentRunbook/\(v2\) **48/49**, ExpenseGuard/\(v2\) **44/46**, and IncidentRunbook/\(v1\) **36/38**. The paper describes a recurring failure pattern of **oscillation**, in which fixing one test breaks another, suggesting conflicting expectations caused by ambiguous specification language.

Mutation testing exposes both strength and blind spots. Selected results include consistent killing of `SKIP_AUTH_GATE`, `LEAK_PII_DIRECT_REQ`, `SKIP_CONFIRM_CANCEL`, `ANSWER_WITHOUT_SQL`, `SKIP_CLARIFY_AMBIG`, `FAIL_TO_PAGE_SEV1`, `APPROVE_NO_RECEIPT`, and `SKIP_FX_CONVERSION`; however, `HALLUCINATE_NUMBERS` in DataInsights survived in **1/3** \(v1\) runs, and `SKIP_RUNBOOK_LOOKUP` in IncidentRunbook survived in **2/2** successful \(v1\) runs before being fixed in \(v2\) [2603.08806]. Across experiments, **87\% of mutation intents successfully activated**, with non-activating mutants excluded from mutation score computation.

## 7. Limitations and interpretation

The paper identifies several limitations specific to SpecSuite-Core. First, coverage is incomplete because TDAD assumes desired behavior can be expressed as executable tests; properties such as empathy are difficult to specify precisely [2603.08806]. Mutation testing improves confidence in suite quality but does not prove completeness, and excluding non-activating mutants may overstate test adequacy if some excluded mutants correspond to real blind spots.

Second, benchmark outcomes are stochastic rather than single deterministic scores. Test generation and prompt compilation are both stochastic; reported variance is about \(\pm 2\text{–}4\%\) for \(v1\) hidden pass rate and \(\pm 4\text{–}16\%\) for \(v2\) [2603.08806]. Third, the benchmark incurs nontrivial overhead: each spec version requires about **30–60 minutes** wall-clock and roughly **\$2–3** in API cost [2603.08806]. Fourth, the paper notes possible **LLM self-censorship in test generation**, where TestSmith may avoid strongly adversarial or profane inputs, limiting abuse-detection coverage. Fifth, the benchmark scope remains narrow: only four specifications with roughly **10–14 decision nodes**, no scaling study to **50+ nodes**, no cross-model analysis, and no component-wise ablation [2603.08806].

Within those limits, SpecSuite-Core is best understood as an engineering benchmark for specification-driven agent development. Its central contribution is not simply the presence of four tasks but the fact that each task is embedded in a protocol that evaluates compilation, hidden-test generalization, mutation sensitivity, and regression safety under evolving requirements [2603.08806]. The reported \(v1\) results—**92\%** compilation success and **97.3\%** hidden pass rate—indicate that this workflow can often compile deeply specified tool-using agents successfully, while the harder \(v2\) results—**58\%** compilation success, **77.7\%** hidden pass rate, and **97.2\%** regression safety—indicate that requirement evolution remains the more demanding regime. This suggests that SpecSuite-Core is less a benchmark of agent capability in isolation than a benchmark of whether specification, testing, and prompt compilation can form a reliable development discipline for tool-using LLM agents.

Source: https://www.emergentmind.com/topics/specsuite-core