Permission Denied: How Security Policies Break Coding Agents
This lightning talk reveals a critical blind spot in coding agent evaluation: virtually all benchmarks test agents in unrestricted environments that bear no resemblance to real enterprise security controls. The authors introduce Boundary-Bench, a framework that enforces realistic security policies during agent evaluation, and show that every tested agent suffers significant, non-uniform degradation in both success rate and cost efficiency when faced with common restrictions like network controls, read-only filesystems, and privilege limits. The findings demonstrate that unconstrained leaderboards are fundamentally misleading for production deployment decisions.Script
Most coding agent benchmarks test models in wide-open playgrounds with root access, open networks, and writable filesystems. But real enterprise environments enforce strict security policies. What happens when agents face the locked-down infrastructure they'll actually encounter in production?
The authors introduce Boundary-Bench, which enforces security controls across three orthogonal axes: network egress, filesystem writability, and process privilege. Each axis operates at three levels of restriction, composing into a severity ladder that mirrors real organizational hardening, from wide-open control baselines up through NIST-derived strict policies with allowlist-only egress, read-only global filesystems, and zero privilege escalation.
Testing 12 model and harness bundles on 89 coding tasks, they found that every single agent is forced down and to the right on the success versus cost frontier as policies tighten. But the degradation is strikingly non-uniform: Grok 4.5 lost only 7 percentage points of success yet saw cost inflate by 167 percent, while Claude Sonnet 5 dropped 18 points in success with just 21 percent cost increase.
The cost inflation isn't from agents giving up early. Instead, policy enforcement triggers protracted workaround attempts: agents detect a blocked operation, retry alternate approaches, reconstruct toolchains from allowable sources, and burn through budget whether or not they ultimately succeed. Median cost nearly triples under strict policy, affecting both passing and failing runs equally.
By establishing solvability witnesses for each task, the authors discovered that cost penalties concentrate almost entirely on the subset of tasks whose reference solutions are incompatible with the enforced policy. Affected tasks see cost multiply by 2.6 times baseline, while unaffected tasks remain near parity, providing a powerful predictor of which workloads will struggle under real security controls.
Unconstrained benchmarks are fundamentally misleading when security controls are non-negotiable in production. The future of agent evaluation must integrate policy enforcement as a first-class dimension, enabling accurate operational forecasting and guiding the development of agents that can adapt to the real constraints they'll face. To dive deeper into policy-graded benchmarking and build your own videos on cutting-edge research, visit EmergentMind.com.
