Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments

Abstract: Coding agents increasingly run inside organizations whose security controls (scoped credentials, restricted egress, read-only filesystems, non-root execution) constrain them like any other software. Existing benchmarks, however, evaluate agents almost exclusively in permissive sandboxes, so it is unknown how performance changes when policy is enforced. In this work, we evaluate 12 coding agents on Terminal-Bench 2.1 across nested security policy levels derived from common real-world enterprise restrictions. Hardening is never free but far from uniform: under the strictest policy, success losses reach 18.3 points and cost inflation 167.3\%, and the two axes disagree; the model that best preserves success is also the one that loses the most efficiency, so model choice is policy-dependent. Beyond aggregate scores, we characterize how agents behave when policy blocks their actions and decompose the failures hardening induces: runs grind into timeouts or wrong solutions rather than stopping early, in a mix that differs by model. To ground comparisons, we verify task solvability under the strictest policy, separating model failures from tasks the policy forecloses. We release Boundary-Bench, an open-source hardening plugin enabling policy-constrained evaluation of coding agents on Terminal-Bench and compatible benchmarks.

• The paper introduces Boundary-Bench, a framework that layers graded security policies (network, filesystem, privileges) onto coding agent benchmarks.
• It finds that policy hardening causes non-uniform degradation, with some models facing significant success loss and drastic cost inflation.
• The study underscores the need for policy-adaptive agents and cost-aware scoring to ensure reliable deployment in secure, enterprise environments.

Policy-Enforced Security as a First-Class Axis in Coding Agent Evaluation

Motivation and Context

With the increasing real-world deployment of coding agents in enterprise environments, these agents encounter operational security controls—such as scoped credentials, restricted egress, read-only filesystems, and enforced least privilege—that are standard in modern IT infrastructure. However, most contemporary benchmarks, like Terminal-Bench and FeatureBench, historically assess agent performance in permissive, unconstrained environments, ignoring the operational impacts of security policy enforcement. This gap leaves unknown how agent success and associated efficiency are affected by practical organizational constraints.

The paper "Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments" (2608.02670) systematically addresses this shortcoming via policy-graded benchmarking of coding agents. The authors introduce Boundary-Bench, a framework for layering natively enforced execution policies onto standard agent benchmarks, enabling rigorous, reproducible measurement of how performance and resource cost shift across incremental policy hardening corresponding to well-established control baselines (e.g., NIST SP 800-53).

Boundary-Bench and Policy Enforcement Framework

Boundary-Bench provides an extensible evaluation harness, enforcing security policies directly at the system level—without agent cooperation—using three orthogonal control axes: network egress, filesystem writability/scope, and process privilege. Each axis features a three-level granularity: open (unrestricted), restricted, and locked (maximally constrained). The composition of these axes yields a cumulative "severity ladder":

• Control: No restrictions; agents run as root, with open network and full writable filesystem.
• Non-root: Only privilege dropped; agent executes as non-root, but retains open network and writable filesystems.
• NIST-derived high: Simultaneous enforcement of a strict egress allowlist (developer domains), read-only global filesystem (except a dedicated writable workspace), and non-root execution with all privilege escalation blocked (no sudo, capabilities removed, setuid bits stripped).

The intention is to model the real-world trajectory as policy hardening increases, tracking the drift in agent performance, efficiency, and behavior as a function of enforced policy, not simply as a one-off state.

Empirical Evaluation on Terminal-Bench 2.1

Experimental Design

Twelve model–harness bundles were evaluated, spanning major model families (OpenAI GPT-5.6 Sol/Terra/Luna, Anthropic Claude Sonnet/Opus/Fable, xAI Grok, and open weights via Terminus-2). Each bundle was assessed over 89 coding tasks from Terminal-Bench 2.1 under every policy configuration, with three valid replicate trials per task/policy pair. The environment for each run was a clean cloud sandbox, fully enforcing the specified policy with no agent-side knowledge or adaptation.

Benchmarked Metrics

Primary outcomes were per-bundle success rate (fraction of tasks solved) and mean cost (USD spent on model inference/tokens per-trial), constituting a two-dimensional success–cost Pareto frontier. Task solvability under the strictest policy ("solvability witness") was established by replaying shipped reference solutions and constructing minimal policy-compliant alternatives when necessary.

Figure 1: Hardening moves the Pareto frontier down and to the right, by amounts that differ across model-harness bundles.

Core Findings: Non-Uniform Degradation and Currency of Cost

Pareto Frontier Deformation under Hardening

Every evaluated bundle is forced down and to the right (decreased success, increased cost) as policy severity intensifies, but not in a uniform or predictable fashion. The magnitude and "currency" of degradation—whether an agent loses more success or suffers more from inflated cost—are model/harness-specific, implying that model selection must be policy-aware.

• (Figure 1) shows that, under NIST-derived high, the success–cost Pareto frontier is significantly shifted, with certain agent bundles accruing drastically more inefficiency for relatively preserved success, and others losing success in favor of retaining more efficiency.

  Figure 2: Every bundle loses success and gains cost non-uniformly under strict policy; some models incur high cost inflation for minimal success loss.

Quantitatively:

• Grok 4.5 has the smallest loss in success rate (${−7.1}$pp) but with extreme cost inflation ($+167.3\%$ over baseline).
• Claude Sonnet 5 loses the most success (${−18.3}$pp) with only moderate cost increase ($+21.4\%$).
• GPT-5.6 Luna accepts a large drop in success (${−18.0}$pp) for the lowest additional cost ($+16.0\%$).

Failure Mode Decomposition

Policy-induced performance degradation does not trigger simple early surrender. Instead, it leads to protracted trajectories; failures transition predominantly into timeouts (e.g., agent runs until the wall-clock limit without solution) and wrong solutions (completed trajectories that fail verification), with little increase in early quits.

Mechanism of Cost Inflation

Policy hardening causes the median cost to increase nearly threefold (Figure 3), regardless of whether success is preserved or lost. The main source of added cost is repeated workarounds—agents detect a blocked or denied operation (e.g., network blocks or denied privilege), retry alternate approaches, or reconstruct toolchains from allowable sources, expending budget often with no gain in outcome.

Figure 3: Enforcing security policy (NIST-derived high) triggers a near tripling of per-run median cost, affecting both successful and unsuccessful runs.

• Notably, both passing and failing runs incur this inflation—failed trajectories typically consume the full inflated budget, not less.

Task-Level and Solution Compatibility Analysis

To separate true agent failures from those that stem from task–policy incompatibility, the authors establish a "solvability witness" for each task: either the original reference trajectory or a minimally adapted, policy-compliant one. Out of 89 tasks, 82 remain solvable under strict policy; the remaining 7 are design-blocked (e.g., task requirements inherently demand forbidden resources or actions). This clarity enables blame assignment: failure on a solvable task indicates agent–enforcement interaction, while failure on inherently blocked tasks reflects inevitable incompatibility.

Policy-induced cost and success penalties are concentrated almost exclusively on the subset of tasks whose canonical reference solution fails under policy (Figure 4), while unaffected tasks preserve baseline efficiency and success.

Figure 4: Cost inflation under policy is concentrated on tasks whose reference solution is incompatible with the enforced policy; unaffected tasks remain near parity.

• On affected tasks, mean cost multipliers rise to $2.59\times$ baseline (NIST-derived high), while unaffected tasks remain close to $1.14\times$.
• The same stratification reliably predicts success penalty.

Blocked Action and Exposure Statistics

A granular analysis of blocked operations reveals that policy enforcement has concentrated impact:

• Under strict policy, about a third of all trials encounter at least one blocked action, primarily egress denials.
• A small number of tasks ("head tasks") account for a majority of blocked actions, while most tasks are only minimally impacted.
• For tasks that are solvable, encountering a blocked action is associated with lower pass rates and increased cost, but many passing runs incur at least incidental denials as part of alternate solution strategies.

Limitations

• Results are obtained on Terminal-Bench 2.1 exclusively; generalization to other benchmarks would require analogous solvability audits and enforcement integrations.
• Attribution is at the model–harness bundle level due to architectural dependencies.
• Only one operating point (high effort, three trials per cell) was evaluated; results may vary at different trial depths or agent effort configurations.
• Policy implementations are modeled on standard controls but do not constitute a deployment-ready security baseline.

Implications and Future Directions

Practical Implications

Benchmarks must report agent performance under realistic policy constraints to inform meaningful model selection for enterprise deployment; unconstrained leaderboards are fundamentally misleading when security controls are non-negotiable in operational contexts.

• Cost-aware scoring (Pareto analysis) reveals discriminative headroom masked by unconstrained benchmarks, demonstrating that high aggregate performance can mask deeply divergent failure and cost profiles under realistic constraints.
• Reference solution compatibility is a powerful predictor of which tasks (and thus agent trajectories) will be impacted by security controls; this analysis can be performed prior to agent benchmarking.

Theoretical Implications

Non-uniform, model-dependent responses to policy enforcement suggest that reasoning, retry, and error-recovery strategies in current coding agents are highly brittle, lacking explicit adaptivity to environmental restrictions. Policy-compliant solution synthesis is neither algorithmically trivial nor emergent.

• There is a clear need for policy-adaptive agents: architectures that proactively model, detect, and dynamically work around (or even anticipate) execution-time denials, rather than reacting naively.

Speculation on Future AI Directions

We anticipate a new generation of evaluation standards that integrate security policy axes alongside task complexity and stochastic perturbations. The development of agents that are robust not only to adversarial input (see e.g., (Zhu et al., 2023, Wang et al., 2021)) or environmental fault injection (Kara et al., 28 Sep 2025, Gupta, 3 Jan 2026) but to systemic privilege and access constraints, will be essential for production deployment across organizational boundaries.

• Policy-sensitive meta-planning, runtime privilege detection, and pro-active capability negotiation are likely to become core features in agent system design.
• Benchmark construction methodologies will need to expand solvability audits and artifact-corrected scoring, as over-specified verifiers and reference solution dependencies can otherwise confound interpretation of baseline performance.

Conclusion

This paper establishes, via robust empirical evidence, that coding agent performance under realistic security policies is substantially and non-uniformly degraded, with strong dependence on both agent architecture and specific task-policy interactions. Reporting agent performance without a policy dimension obscures critical deployment risks and misguides model selection. Embedding graded, natively enforced security controls into agent benchmarks, accompanied by task solvability audits and cost-success Pareto analyses, should be normalized in agent evaluation practice. This enables accurate operational forecasting and accelerates the development of agents capable of effective, compliant performance in real organizational contexts.

Natural future work includes the extension of this methodology to additional benchmarks and environments, and the engineering of agents that actively adapt not only to input and tool perturbations, but to security posture itself—from inference to planning to execution.