---
title: 'ScaffoldSafety: Deployment Evaluation Framework'
url: https://www.emergentmind.com/topics/scaffoldsafety
type: topic
---

# ScaffoldSafety: Deployment Evaluation Framework

ScaffoldSafety is the entire released resource accompanying “Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety,” comprising an open-source evaluation framework/codebase, the data and scored outputs from a large controlled study, all prompts, scaffold templates, and scoring instructions, and scripts for the full statistical analysis [2603.10044]. It was built to study a specific evaluation–deployment mismatch in language-model safety: most safety benchmarks measure models in isolation and typically in multiple-choice format, whereas production systems often wrap models in agentic scaffolds such as reasoning loops, critic pipelines, and delegation workflows, and query them in open-ended natural language. In this sense, ScaffoldSafety is both an empirical study and a reproducible artifact for auditing how deployment architecture and response format shape measured safety.

## 1. Scope and conceptual basis

ScaffoldSafety targets two mismatches that the underlying study treats as central. The first is an **architectural gap**: benchmarks usually test one prompt, one response, one score, whereas deployed systems frequently introduce agentic scaffolds such as ReAct-style reasoning loops, critic pipelines, map-reduce task decomposition, and multi-agent debate. The second is a **format gap**: widely used safety benchmarks such as BBQ, TruthfulQA, and Anthropic model-written evaluations are predominantly multiple-choice, whereas deployed systems usually interact through open-ended language [2603.10044].

The project’s motivating concern is that **measured safety need not equal deployed safety** once either the deployment architecture or the response format changes. The study therefore treats safety scores as properties of a joint evaluation condition rather than as invariant attributes of a base model. A direct implication is that benchmark scores obtained from direct-API multiple-choice settings may not transfer to realistic scaffolded deployments.

The release itself is designed for reproduction, extension, and audit. In the paper’s terminology, “ScaffoldSafety” refers not merely to a benchmark but to the full implementation and artifact layer associated with the empirical study, including code, prompts, scored outputs, and statistical analysis scripts [2603.10044].

## 2. Experimental design and scaffold configurations

The main experiment uses a full factorial design with **6 frontier models**, **4 scaffold configurations**, and **2,617 unique prompts**, yielding the headline **\(N = 62{,}808\)**. The model set is Claude Opus 4.6, GPT‑5.2, Gemini 3 Pro, Llama 4 Maverick, DeepSeek V3.2, and Mistral Large 2. The four evaluated configurations are Direct API, ReAct agent, Multi-agent with critic, and Map-reduce delegation [2603.10044].

The benchmark suite contains four safety benchmarks and one non-safety control. The safety benchmarks are sycophancy resistance from Anthropic model-written evals, BBQ for bias, XSTest + OR-Bench for over-refusal, and TruthfulQA MC1 for truthfulness. The non-safety control is AI factual recall about model capabilities. Additional experiments include a **mechanistic study** with **7,200 calls**, a **format-dependence study** with **4,400 calls**, and a **dedicated sycophancy run** with **12,000 calls**. All hypotheses and analysis plans were **pre-registered** on OSF, with deviations documented.

The four scaffold configurations are deliberately minimalized to isolate architecture. **Direct API** is a single call per item with a standard system prompt and the benchmark question as user message, with no tools, no chain-of-thought prompts, and temperature 0 where supported. **ReAct agent** uses a single-agent Thought–Action–Observation loop of up to 5 iterations, but with no external tools, so that structured reasoning is varied without varying information sources. **Multi-agent with critic** is a 2-agent pipeline in which a primary agent answers, and a critic agent with a different system prompt evaluates safety and correctness and can request up to 2 revisions. **Map-reduce delegation** decomposes a task into sub-questions in a map phase, answers them in worker calls, and then synthesizes them in a reduce phase.

A central mechanistic finding of the design is that **standard map-reduce drops the original MC answer options when generating sub-questions**. The study therefore also introduces an **option-preserving map-reduce** variant for a subset of items, in which the original MC choices are propagated into all worker calls so that response format is structurally preserved.

The format-dependence experiment constructs a 2×2 design with **Format: MC vs OE** and **Scaffold: Direct vs Map-reduce**. It uses 220 matched item pairs per benchmark. MC items are scored by deterministic extraction of the final letter and comparison to the answer key. OE items are scored either by LLM-as-judge with a rubric—using Gemini 3 Flash as the main judge and Opus 4.6 for cross-check—or by simple string semantics for the capability control [2603.10044].

## 3. Statistical framework and measurement theory

For the main scaffold effects, the study uses **logistic regression with cluster-robust standard errors at the case level**, with treatment-coded indicators for ReAct, Multi-agent, and Map-reduce, and with model fixed effects and benchmark fixed effects [2603.10044]. The omnibus effect of configuration is tested with a Wald test, and pairwise differences use Holm–Bonferroni correction.

For comparisons that are not statistically significant, the analysis uses **Two One-Sided Tests (TOST)** with a pre-registered margin of **\(\Delta = 2\)** percentage points in risk difference. Equivalence is concluded when the 90% confidence interval for the scaffold-minus-direct risk difference lies strictly inside \([-\Delta, +\Delta]\). This is a practical-equivalence criterion rather than a null-effect test.

For deployment-facing interpretability, the paper reports **Number Needed to Harm**:
$$
\text{NNH} = \frac{1}{|\text{RD}|}
$$
where \(\text{RD}\) is the absolute risk difference in safety rate expressed as a proportion. The study also uses **specification curve analysis**, enumerating **384** plausible specifications across **9 degrees of freedom**, with an **18-spec main curve**, to test robustness to analytic choices such as benchmark subsets, model subsets, scoring methods, and partial-compliance handling.

The study’s most consequential measurement-theoretic result comes from **generalizability analysis**. Treating models as objects of measurement and benchmarks and scaffolds as measurement facets, it estimates:
$$
G = 0.000 \quad \text{(bootstrap 95\% CI } [0.000, 0.752])
$$
for an overall safety score aggregated across four benchmarks. In parallel, factorial variance decomposition on the \(6 \times 4 \times 4\) cell means yields \(\eta^2 \approx 19.3\%\) for the benchmark main effect, **3.0%** for Model×Benchmark, **1.2%** for Scaffold×Benchmark, **1.0%** for the model main effect, and only **0.4%** each for the scaffold main effect and Model×Scaffold. The paper summarizes this as benchmark choice explaining **45×** more variance than scaffold architecture [2603.10044].

## 4. Main empirical results

Pooling across the four safety benchmarks, the observed safety rates are **72.8%** for Direct API, **72.1%** for ReAct, **72.2%** for Multi-agent, and **65.5%** for Map-reduce [2603.10044]. ReAct versus Direct yields **OR = 0.95 [0.92, 0.99]**, **RD = −0.7pp [−1.2, −0.2]**, and **NNH ≈ 135**; it is statistically significant but still inside the pre-registered \(\pm 2\)pp TOST margin, so it is treated as practically equivalent. Multi-agent versus Direct yields **OR = 0.96 [0.92, 1.00]**, **RD = −0.6pp [−1.3, 0.0]**, **NNH ≈ 165**, and TOST likewise confirms equivalence within \(\pm 2\)pp. Map-reduce versus Direct yields **OR = 0.65 [0.62, 0.68]**, **RD = −7.3pp [−8.1, −6.4]**, and **NNH = 14**, corresponding to roughly one additional unsafe response per fourteen queries.

The format-dependence experiment shows that the largest shifts arise not from scaffold architecture but from changing identical items from multiple-choice to open-ended format.

| Benchmark | MC safe | OE safe |
|---|---:|---:|
| Sycophancy | 33.7% | 53.3% |
| BBQ | 83.0% | 99.2% |
| TruthfulQA | 79.3% | 85.0% |
| AI Factual Recall | 77.0% | 76.0% |
| MMLU | 85.4% | 76.2% |

On three safety benchmarks—sycophancy, BBQ, and TruthfulQA—switching from MC to OE raises measured safety by **+19.6pp**, **+16.2pp**, and **+5.7pp**, respectively. On the non-safety capability benchmark MMLU, the direction reverses: MC exceeds OE by **9.2pp**. AI factual recall is nearly format-invariant at **−1.0pp**. The paper’s central interpretation is that the main map-reduce degradation in the aggregate study is largely a **format artifact**: when MC answer options are stripped from worker calls, map-reduce silently changes what the benchmark is measuring.

Within each format, the dedicated 2×2 experiment finds **map-reduce ≈ direct**, with effects under **2pp** and TOST-equivalent. The option-preserving map-reduce intervention recovers **40–89%** of the original map-reduce degradation, depending on model. The paper reports recovery rates of **89%** for Opus, **64%** for DeepSeek, **73%** for Mistral, and **40%** for GPT‑5.2, with residual harms ranging from approximately **−1.5pp** for Opus to residual **NNH** values between **10** and **67** in exploratory estimates [2603.10044]. This indicates that most of the harmful map-reduce effect is attributable to loss of MC options rather than to an intrinsic failure of multi-step reasoning itself.

## 5. Model heterogeneity and the collapse of single-number safety rankings

The interaction between configuration and model is reported as **highly significant**, with **Wald \(\chi^2 = 511.3\), \(df = 15\), \(p < 10^{-99}\)** [2603.10044]. Relative to Direct, map-reduce changes pooled safety by **−15.1pp** for DeepSeek, **−8.6pp** for GPT‑5.2, **−6.1pp** for Mistral, **−15.6pp** for Opus, **+0.9pp** for Llama 4, and **+0.7pp** for Gemini. The same scaffold therefore harms some models and mildly improves others.

The most extreme divergence appears on sycophancy. Opus shifts from **49.0%** sycophancy resistance under Direct to **32.2%** under Map-reduce, a change of **−16.8pp**. Llama 4 shifts from **11.0%** to **29.8%**, a change of **+18.8pp**. Gemini shifts from **6.0%** to **19.0%**, a change of **+13pp** in the intention-to-treat analysis, with noted parse caveats. The paper summarizes these opposing effects as a **35pp** interaction span on the same safety property under the same scaffold. This rules out universal claims of the form “map-reduce is safe,” “map-reduce is unsafe,” or “multi-agent improves safety” in any model-independent sense.

The generalizability result sharpens this claim. Because **\(G = 0.000\)** for four-benchmark composite safety, model rankings can reverse depending on which benchmark is included. In the variance decomposition, benchmark choice dominates scaffold architecture, and model-by-benchmark interaction exceeds both scaffold main effects and model-by-scaffold interaction. A plausible implication is that any “overall safety” score collapsing models across benchmarks and configurations is unstable unless measurement facets are expanded substantially. The paper therefore argues that **per-model, per-configuration testing is a necessary minimum standard** [2603.10044].

## 6. Resource contents, limitations, and broader scaffold-aware safety research

The release includes a Python framework, **`ScaffoldSafetyEval`**, with scaffold classes for Direct, ReAct, Multi-agent, and Map-reduce; benchmark loaders for TruthfulQA, BBQ, AI Factual Recall, XSTest/OR-Bench, and control capability benchmarks such as MMLU; deterministic MC extraction and LLM-as-judge scoring modules; a blinding and sanitization pipeline that strips chain-of-thought, metadata, and self-identification; UUID mapping and hashing; and statistical analysis scripts for logistic regression with cluster-robust standard errors, TOST, specification-curve enumeration, and generalizability and ANOVA decomposition [2603.10044]. The release also contains raw prompts and responses for all **62,808** main-study calls plus follow-up experiments, decomposition traces for map-reduce sub-calls, scaffold templates and scoring rubrics, and scorecard tools for a benchmark-by-configuration safety matrix, NNH reporting, and a methodology stamp.

The framework is intended both to reproduce the paper and to extend it. The paper explicitly describes adding a new model through the model specification interface, adding a new scaffold as a subclass, adding new benchmarks, and running the MC/OE × Direct/Map-reduce 2×2 design on new tasks. Its limitations are also explicit: only four proxy safety properties are covered; three of the four main safety benchmarks are multiple-choice; only six chat-style frontier models are included; human auditing is limited; the scaffolds exclude external tools, retrieval-augmented generation, and code execution; and results are a temporal snapshot tied to early 2026 model versions.

Several future directions are named. These include extending format-sensitivity analysis to toxicity, deception, and harmful-instruction compliance; scaling the invocation framework; moving from controlled minimal scaffolds to production frameworks such as LangChain, OpenAI Agents SDK, and CrewAI; extending the analysis to consequential safety properties such as CBRN and cyber risk; and adopting wider use of pre-registration, equivalence testing, specification curves, and generalizability analysis [2603.10044].

In the broader scaffold-aware literature, related work has generalized the safety question beyond benchmarked chat models. “OctoBench” studies scaffold-aware instruction following in repository-grounded agentic coding with **34 environments**, **217 tasks**, and **7,098 objective checklist items**, separating rule-following from task completion [2601.10343]. “Capability Conditioned Scaffolding” defines a typed framework that partitions expertise into **strong**, **mixed**, and **weak** domains and conditions intervention behavior on structured capability profiles [2605.15404]. “SCAFFOLD-CEGIS” addresses latent security degradation in iterative code refinement through **semantic anchoring**, **four-layer gated verification**, and explicit safety monotonicity criteria [2603.08520]. Taken together, these works suggest that scaffold safety is not a unitary property of either a model or a prompt, but a property of a coupled system composed of model, scaffold, task format, and verification regime.

Source: https://www.emergentmind.com/topics/scaffoldsafety