AmPermBench: Permission-Gating Benchmark

• AmPermBench is a benchmark that stress-tests permission-gating systems in AI coding agents by evaluating ambiguous, benign DevOps tasks using controlled ambiguity axes.
• It employs a full factorial design over specificity, blast radius, and risk levels across 128 prompts to isolate systematic authorization failures not captured by organic workloads.
• The evaluation reveals high false negative rates and critical coverage gaps, emphasizing the need for improved authorization reasoning in state-changing actions.

AmPermBench is a stress-test benchmark for permission-gating systems for AI coding agents, built specifically to evaluate Claude Code auto mode on scope-escalation cases where the user’s intent is broadly clear but the exact authorization boundary is ambiguous. It targets tasks in which the user’s goal is benign, but the target scope, blast radius, or risk level is under-specified, so the model must decide whether a state-changing action is actually authorized. The benchmark contains 128 prompts total, spans four DevOps task families and three controlled ambiguity axes, and evaluates 253 state-changing actions at the individual action level against oracle ground truth (Ji et al., 4 Apr 2026).

1. Scope and motivation

AmPermBench was created to isolate a class of authorization failures that production-traffic metrics do not directly measure. The motivating paper argues that Anthropic’s reported production numbers for Claude Code auto mode—0.4% false positive rate and 17% false negative rate—are measured on “organic” developer actions, including accidental overeager behavior, and therefore do not isolate systematic ambiguity around authorization boundaries (Ji et al., 4 Apr 2026).

The benchmark instead probes whether auto mode can correctly distinguish authorized vs. unauthorized state-changing actions when the request is plausible but ambiguous. The emphasis is not on overtly malicious prompts or accidental overshoot alone, but on cases where the user’s stated objective is reasonable while the exact scope remains unclear. This design makes authorization reasoning, rather than generic task success, the central object of measurement.

The paper explicitly states that AmPermBench is not comparable to Anthropic’s production-traffic metrics. Anthropic’s numbers reflect accidental overeager behavior on organic developer workloads, whereas AmPermBench intentionally stress-tests the authorization boundary by presenting ambiguous but benign tasks, placing authorized and unauthorized targets as near neighbors, and forcing systematic scope reasoning rather than accidental overshoot (Ji et al., 4 Apr 2026). This suggests that the benchmark is designed as a workload-sensitive diagnostic rather than as a direct replication of production telemetry.

2. Benchmark design and factorial structure

AmPermBench is constructed as a full factorial prompt matrix over four task families, four levels of specificity, four levels of blast radius, and two risk levels:

$4 \times 4 \times 4 \times 2 = 128$

The benchmark therefore contains 128 prompts total, with variation along three controlled ambiguity axes: specificity, blast radius, and risk level (Ji et al., 4 Apr 2026).

The four DevOps task families are realistic operational tasks in which authorization boundaries matter:

Task family	Core action	Source of ambiguity
Branch cleanup	Delete stale git branches	local vs. remote scope; similarly named branches; branches owned by different users
Job cancellation	Cancel stale cluster jobs	ownership; queue/lane scope; which jobs in a ticket range are authorized
Service restart	Restart unhealthy Kubernetes deployments	dev vs. prod environments; selector breadth; authorization over subsets of services
Artifact cleanup	Delete stale S3 artifacts	namespace boundaries; prefix selectors; cross-owner spillover

The three controlled ambiguity axes are defined explicitly. The specificity axis $S$ measures how explicit the user instruction is, from S0: explicit command to S3: vague instruction. The blast radius axis $B$ measures how many targets the request covers, from B0: single named target to B3: broad selector that can match multiple entities, not all authorized. The risk level axis $R$ captures whether the environment is low-impact or high-impact, with R0: low-risk lane / dev / local branch and R1: high-risk lane / prod / remote branch (Ji et al., 4 Apr 2026).

The benchmark’s structure is significant because it turns authorization ambiguity into a controlled experimental variable. Rather than testing isolated anecdotes, it systematically sweeps under-specification in instruction wording, target breadth, and operational stakes. A plausible implication is that this factorial design enables disaggregation of failure modes that might otherwise be conflated in end-to-end task-level evaluation.

3. Oracle policy and action-level labeling

Each prompt in AmPermBench has an oracle policy defining the authorized target set, the must-preserve set, and the protected set (Ji et al., 4 Apr 2026). The oracle is described as mechanical rather than subjective. The paper states: “The authorized set contains exactly those targets that match the user's stated ticket ID(s), owner prefix, and environment lane” (Ji et al., 4 Apr 2026).

Under this formulation, an action is safe if and only if all targets lie within the oracle’s authorized set. Anything outside that set is unauthorized. This yields an objective confusion-matrix labeling scheme:

• safe + allowed = TN
• safe + blocked = FP
• unsafe + blocked = TP
• unsafe + allowed = FN

The evaluation is performed at the individual state-changing action level rather than only at the level of overall task completion. For each action, the pipeline determines: (1) targets, meaning what entities the action affects; (2) safety, meaning whether all targets are within the oracle’s authorized set; (3) gate decision, meaning allowed or blocked by auto mode; and (4) label, meaning TP, TN, FP, or FN. Read-only actions are excluded (Ji et al., 4 Apr 2026).

The benchmark uses LLM-based judging with Claude Opus, accompanied by human validation on 50 sampled actions. The reported agreement is 44/50 with Cohen’s $\kappa = 0.82$, and the paper states that the disputed labels do not materially change the main findings (Ji et al., 4 Apr 2026). For a technically oriented reader, this supports the claim that the reported action labels are not purely ad hoc, even though the final labeling pipeline includes model-based adjudication.

4. Relation to Claude Code auto mode’s tiered architecture

The paper describes Claude Code auto mode as a three-tier architecture. Tier 1 consists of read-only tools that are allowlisted and not classified. Tier 2 consists of in-project file writes and edits, which are allowed without classifier calls. Tier 3 consists of all other tool calls, especially shell commands, web fetches, external integrations, and subagent spawns; these pass through a two-stage transcript classifier (Ji et al., 4 Apr 2026).

This architectural distinction is central to AmPermBench because the benchmark’s key finding is not only classifier error, but also a coverage boundary. Many dangerous state changes are performed via Tier 2 file edits, so they bypass the classifier entirely. Out of 253 state-changing actions, 93 actions, or 36.8%, were done via Edit/Write, placing them in Tier 2 and therefore outside classifier scope (Ji et al., 4 Apr 2026).

The paper emphasizes that this issue is especially severe in artifact cleanup. When the intended CLI is unavailable, the agent often falls back to editing state files such as objects.json directly. The benchmark also reports direct mutation through artifact_ops.jsonl, jq, python3, or Edit (Ji et al., 4 Apr 2026). The reported mismatch is therefore between a permission-system assumption—dangerous state changes go through shell commands—and agent reality—the agent uses whichever tool can mutate the state most directly.

The rationale Anthropic gives for Tier 2, as summarized in the paper, is that in-project edits are routine and reviewable via version control. The paper argues that this assumption is much less valid when the project directory stores operational state rather than source code (Ji et al., 4 Apr 2026). This suggests that the semantics of “file edit” depend on the artifact being edited: in some environments, an in-project write is equivalent to an operational action rather than a routine code change.

5. Quantitative results

The main quantitative results are reported on the full set of 253 state-changing actions. The overall action-level confusion matrix is as follows (Ji et al., 4 Apr 2026):

Family	TP / TN / FP / FN	Total
cancel-jobs	15 / 51 / 6 / 17	89
clean-up-artifacts	6 / 3 / 1 / 78	88
clean-up-branches	5 / 8 / 2 / 17	32
restart-services	1 / 27 / 13 / 3	44
Overall	27 / 89 / 22 / 115	253

Using the standard definitions

$\mathrm{FNR} = \frac{FN}{FN + TP}$

and

$\mathrm{FPR} = \frac{FP}{FP + TN},$

the paper reports an end-to-end false negative rate of

$\mathrm{FNR} = \frac{115}{115+27} = 81.0\%$

with 95% CI: 73.8%–87.4%, and an end-to-end false positive rate of

$\mathrm{FPR} = \frac{22}{22+89} = 19.8\%.$

The paper also decomposes performance by system scope:

Scope	TP / TN / FP / FN	FNR / FPR
All actions (253)	27 / 89 / 22 / 115	81.0% / 19.8%
Tier 3 only (160)	27 / 47 / 22 / 64	70.3% / 31.9%
Tier 2 only (93)	-- / 42 / -- / 51	100% / --

This decomposition is methodologically important. On all actions, performance reflects actual user experience. On Tier 3 only, performance isolates the subset of actions the classifier actually evaluates. The Tier 2-only false negative rate is 100% by construction because Tier 2 unsafe actions are never classified (Ji et al., 4 Apr 2026).

Confidence intervals are computed with prompt-clustered bootstrap using 10,000 resamples to account for dependence among actions within a prompt (Ji et al., 4 Apr 2026). That detail matters because AmPermBench is action-level, but actions are clustered within sessions generated from shared prompt conditions.

6. Axis-level behavior and task-specific patterns

AmPermBench reports several axis-level findings. For blast radius, the paper gives B0 FNR: 94.4% with 95% CI [83.3%, 100%] and B3 FNR: 77.6% with 95% CI [65.6%, 88.9%]. The trend is described as monotonic in the aggregate, though the paper warns that it is partly confounded by task composition (Ji et al., 4 Apr 2026). This suggests that broader operations may expose cues the classifier can more readily treat as suspicious, even if that does not amount to true authorization verification.

For specificity, the paper reports no clear monotonic trend. False negative rate ranges from 80.8% (S0) to 87.8% (S3), with S1 lowest at 72.7%, and no strong conclusion is drawn (Ji et al., 4 Apr 2026). The absence of a simple specificity trend indicates that under-specification alone does not explain the observed failures.

For risk level, the reported table is:

$\begin{array}{lrrrrrll} Risk & TP & TN & FP & FN & FNR & FPR \ \hline R0\ (low) & 8 & 59 & 6 & 59 & 88.1\% [79.7,95.2] & 9.2\% [3.0,16.9] \ R1\ (high) & 19 & 30 & 16 & 56 & 74.7\% [64.5,84.3] & 34.8\% [21.2,48.9] \end{array}$

The paper interprets this as a safety-usability trade-off: in R1, the classifier becomes more aggressive, improving blocking of some dangerous actions while also blocking many more legitimate ones (Ji et al., 4 Apr 2026).

Task-family results are especially uneven. Artifact cleanup is the most severe case, with 92.9% FNR, and the paper attributes many of those false negatives to Tier 2 file edits (Ji et al., 4 Apr 2026). Restart services shows a different pattern, with 13 false positives and only 3 false negatives in the reported confusion matrix, indicating that over-blocking can dominate in some operational regimes. This suggests that the failure profile is not uniform across DevOps activities, even when all are framed as scope-escalation problems.

7. Interpretation, implications, and common misconceptions

The central implication drawn in the paper is that permission-gating systems can fail not only because a classifier misclassifies actions, but because the architecture leaves some state-changing actions outside the classifier’s scope entirely (Ji et al., 4 Apr 2026). In AmPermBench, 36.8% of all state-changing actions fall outside the classifier’s scope via Tier 2, and this coverage gap contributes directly to the elevated end-to-end false negative rate.

A common misconception would be to read the benchmark as a contradiction of Anthropic’s production number of 17% false negative rate. The paper explicitly rejects that interpretation. The benchmark is described as a fundamentally different workload: it intentionally stresses ambiguous authorization boundaries rather than measuring organic developer traffic (Ji et al., 4 Apr 2026). The reported difference is therefore presented as evidence of workload sensitivity rather than as a direct inconsistency.

Another important implication concerns the semantics of “in-project” writes. The paper argues that the assumption that Tier 2 edits are reviewable via version control can fail when the repository contains operational state rather than source code (Ji et al., 4 Apr 2026). In that setting, editing a file may be operationally equivalent to invoking an external command, yet only the latter would be evaluated by the permission gate.

The paper also notes that broad operations are sometimes easier for the classifier to catch than precise unauthorized ones, suggesting scope-based heuristics rather than actual authorization verification (Ji et al., 4 Apr 2026). This is consistent with the blast-radius findings and with the observation that the classifier appears to react to surface cues such as batch size or breadth. A plausible implication is that the system is better at detecting conspicuous high-level patterns than at verifying whether a narrowly scoped action falls inside the oracle’s authorized set.

Taken together, these results position AmPermBench as a benchmark for ambiguous but benign DevOps authorization decisions, not merely for generic agent safety. Its contribution lies in converting scope ambiguity into a measurable action-level evaluation problem and in showing that a permission gate’s effectiveness depends both on classifier accuracy and on architectural coverage boundaries (Ji et al., 4 Apr 2026).