ScenarioBench: Policy-Grounded Text-to-SQL Evaluation

Updated 14 July 2026

ScenarioBench is a policy-grounded, trace-aware benchmark that evaluates compliance in Text-to-SQL and retrieval-augmented systems by linking decision making with clause-level evidence.
It enforces a strict no-peek and grounding invariant protocol where systems must generate both decisions and ordered, verifiable justifications with canonical SQL queries.
Metrics focus on decision accuracy, trace completeness, retrieval effectiveness, and the scenario difficulty index, balancing explanation quality against latency.

ScenarioBench is a policy-grounded, trace-aware benchmark for evaluating Text-to-SQL and retrieval-augmented generation in compliance contexts. Each YAML scenario includes a no-peek gold-standard package with the expected decision, a minimal witness trace, the governing clause set, and the canonical SQL, enabling end-to-end scoring of both what a system decides and why. Systems must justify outputs using clause IDs from the same policy canon, making explanations falsifiable and audit-ready, while the evaluator reports decision accuracy, trace quality, retrieval effectiveness, SQL correctness via result-set equivalence, policy coverage, latency, and an explanation-hallucination rate (Atf et al., 29 Sep 2025).

1. Compliance-oriented benchmark design

ScenarioBench is designed for regulated settings in which correct outputs are insufficient unless they are supported by explicit, checkable policy evidence. Its central requirement is trace-grounded decision making: a system must produce not only a decision label such as allow, block, safe-rewrite, or escalate, but also a justification trace tied to the governing policy canon. The benchmark therefore couples answer evaluation with evidence evaluation rather than treating explanation as a secondary artifact (Atf et al., 29 Sep 2025).

The benchmark incorporates several structural constraints. A strict grounding invariant requires that all explanations cite only those clause IDs actually retrieved; no external or hallucinated evidence is allowed. A no-peek discipline prevents systems from accessing the gold package during inference. Policy canon material is dual-materialized as Prolog facts for rule execution and as a SQL Policy_DB for retrieval and NLQ-to-SQL. Standard logs capture decisions, traces, retrievals, configurations, and timings for post-hoc analysis. The result is an evaluation setting oriented toward compliance auditing and regulation tests, including jurisdiction-agnostic policy canons and examples such as Canada’s Anti-Spam Legislation.

2. Scenario anatomy and gold-standard package

Each ScenarioBench instance is organized as a YAML-formatted scenario description plus an evaluator-only gold-standard package. The scenario description contains policy-relevant context and candidate content, such as the communication or material being evaluated. The gold-standard package is not available to the system at inference time and provides the reference objects needed for scoring (Atf et al., 29 Sep 2025).

Component	Contents	Role
YAML scenario description	Context; candidate content	System input
Gold decision	Correct classification	Decision reference
Minimal witness trace	Ordered clause IDs and roles	Trace reference
Governing clause set	Necessary and sufficient clause IDs	Coverage and hallucination reference
Canonical SQL query	Query over Policy_DB returning evidence clause IDs	SQL reference

The minimal witness trace is the benchmark’s core explanatory object. Traces are ordered lists of $(\text{clause\_id}, \text{role})$ pairs, with roles such as applies, exception, and precedence, together with brief rationales. The governing clause set closes the evidence set under exceptions and precedence, so that coverage and hallucination can be evaluated against a complete reference rather than only against a minimal proof. Canonical SQL is compared by result-set equivalence on clause IDs rather than by string identity, which makes the SQL task evidence-centric rather than surface-form-centric.

This structure separates evaluator knowledge from model knowledge. A plausible implication is that ScenarioBench treats explanation quality as a first-class supervised object while still preserving realistic inference constraints.

3. Grounding invariant and trace-aware evaluation

ScenarioBench’s distinctive mechanism is the coupling of retrieval, decision, explanation, and SQL generation under a single grounding protocol. All justifications must reference only those policy clauses actually retrieved by the system for the scenario. Because every cited clause ID must be both retrievable and checkable, explanations become falsifiable: unsupported references can be detected directly, and clause-level provenance can be audited after the fact (Atf et al., 29 Sep 2025).

This evaluation protocol applies to both Text-to-SQL and RAG systems. For Text-to-SQL, the benchmark measures whether a system can generate a query whose returned clause_id multiset matches the canonical SQL result. For RAG, it measures whether retrieval produces the necessary evidence and whether the downstream answer and trace use that evidence correctly. In both cases, the benchmark “closes the loop” between answer accuracy and explanation defensibility.

Explanation hallucination is measured in two variants. In the strict variant, any trace citation outside the gold closure is a hallucination. In the liberal variant, extra citations that are in retrieved@k and consistent with the decision are neutral. Policy coverage is defined as the fraction of gold support, violation, and exception clauses appearing in the predicted trace or retrieval set. Because all metrics are calculated using the system’s own retrieved evidence, no external peeking or knowledge leakage is allowed.

A common misunderstanding is to view the trace as a free-form rationale. In ScenarioBench, it is instead a constrained, clause-indexed object with ordering, role structure, and retrieval provenance.

4. Metrics and difficulty indices

ScenarioBench adopts a multi-dimensional evaluation scheme. Decision quality is measured with accuracy and macro-F1 over the operational label set. Trace quality is decomposed into completeness $T_c$ , correctness $T_k$ , and order $T_o$ , with mean trace quality defined as

$T = \frac{1}{3}(T_c + T_k + T_o).$

Order is evaluated by rank-order agreement with the gold trace, for example using Kendall- $\tau$ (Atf et al., 29 Sep 2025).

Retrieval effectiveness is measured by Recall@k, MRR, and nDCG@k. SQL correctness is defined by result-set equivalence on clause IDs; degenerate queries are penalized when the gold result is non-empty. Policy coverage measures whether the benchmark’s support, violation, and exception clauses have been surfaced by retrieval or included in the produced trace. Latency is reported as wall-clock time per scenario, with support for retrieval, NLQ, and reasoning breakdowns.

ScenarioBench also introduces two aggregate indices. The Scenario Difficulty Index is

$\mathrm{SDI} = w_D (1 - \mathrm{Acc}) + w_T \left(1 - \tfrac{1}{3}(T_c + T_k + T_o) \right) + w_R (1 - \mathrm{nDCG}@k),$

with default weights $(w_D, w_T, w_R) = (0.5, 0.3, 0.2)$ . Lower SDI corresponds to an easier, better-performing scenario, and the index is normalized between $0$ and $1$.

The budgeted variant,

$T_c$ 0

introduces a latency term with $T_c$ 1 and $T_c$ 2, where $T_c$ 3 is the latency of a no-reflection baseline. This makes the cost-benefit trade-off between better justification and additional delay explicit.

5. Empirical behavior and observed trade-offs

On a seed suite of $T_c$ 4 scenarios, ScenarioBench reports a characteristic result: decision metrics are saturated, with accuracy and macro-F1 both equal to $T_c$ 5, while the discriminative signal shifts to explanation quality. In the reported baseline, trace-completeness and policy-coverage are both $T_c$ 6, strict hallucination is $T_c$ 7, and latency is $T_c$ 8 ms. Adding a single reflection step increases trace-completeness and policy-coverage to $T_c$ 9 while keeping strict hallucination at $T_k$ 0 and increasing latency only to $T_k$ 1 ms (Atf et al., 29 Sep 2025).

The benchmark also reports that stronger retrieval alone does not guarantee better explanations. A hybrid BM25+vector retriever increases latency from $T_k$ 2 ms to $T_k$ 3 ms in the demonstration without improving trace completeness or coverage unless the trace builder actually uses the richer retrieved set. This isolates an important systems issue: retrieval quality, trace construction, and decision generation are separable components, and improvements in one stage need not propagate automatically to the others.

These findings reposition what counts as progress. When label accuracy is already saturated, the benchmark’s pressure moves toward completeness, ordering, coverage, and grounded explanation quality under explicit time budgets. This suggests that compliance evaluation cannot be reduced to classification accuracy, even when classification appears solved on a small suite.

6. Position within scenario-based benchmarking

ScenarioBench differs from standard Text-to-SQL benchmarks such as Spider and BIRD because those benchmarks focus on SQL correctness and cross-domain generalization but do not require clause-level provenance. It also differs from KILT/RAG-style benchmarks, which evaluate answer correctness and sometimes evidence overlap or attribution but lack a strict, policy-grounded trace protocol. ScenarioBench’s contribution is therefore not merely another scenario collection; it is a change in what is being benchmarked, from answer production alone to answer production plus auditable clause-level justification under no-peek rules (Atf et al., 29 Sep 2025).

Within the broader scenario-benchmark literature, related work shows how “scenario” has been used to organize evaluation in other technical domains. ScenEval constructs scenario-based test datasets for code generation by attaching metadata to each task and using test morphisms that filter cases into scenario-specific subsets (Paul et al., 2024). SemBench uses scenarios as the organizing principle for benchmarking semantic query processing engines across modalities and semantic operators (Lao et al., 3 Nov 2025). Scenario-Wise Rec standardizes dataset processing, model implementation, and evaluation for multi-scenario recommendation, addressing fair comparison and reproducibility in recommender systems (Li et al., 2024).

Taken together, these benchmarks indicate that scenario-based evaluation is increasingly used to capture structure that conventional aggregate metrics miss. ScenarioBench specializes that broader trend for compliance settings by making clause-level provenance, trace order, and retrieval-grounded justification the benchmarked objects rather than optional annotations.