- The paper demonstrates that enforcing substrate constraints boosts oversight recall from 54.5% to 90.9% by using code-level invariant checks.
- It employs a minimal CLI tool and isolated execution environments to enable efficient, targeted invariant verification.
- The study critiques prompt-based controls and advocates robust, machine-enforceable oversight mechanisms to curb adversarial exploits.
Steerability via Substrate Constraints for Scalable Oversight of Coding Agents
As autonomous coding agents achieve high competence in generating and modifying large-scale codebases, the primary bottleneck transitions from generation capability to scalable, trustworthy oversight. The failure of context to scale—on both the model and human side—leads to security vulnerabilities and an erosion of effective control as codebases enlarge. Existing approaches that optimize retrieval and prompt context for the agent, such as Model Context Protocol (MCP) and repo-mapping tools, are fundamentally limited by the context window and tend to exacerbate cognitive overload. Moreover, defense measures relying solely on prompts remain advisory and are demonstrably ineffective against adversarial or indirect exploits, as shown in recent large-scale prompt injection studies.
The work departs from this paradigm by proposing substrate-level constraints, drawing directly from proven methods in software engineering (e.g., access control, strict coding conventions, linter enforcement). These substrate constraints provide non-advisory, machine-enforcible boundaries, promising a pathway toward scalable oversight that is robust to agent drift and adversarial behavior in codebases with minimal guarantees, such as Python.
System Design and Principles
The paper presents a system for agent oversight comprised of the following major principles:
- Substrate Constraints: All enforceable rules are actuated at the code or system level using linters (ruff, bugbear, bandit), runtime contract validators (beartype, pydantic, pyrefly), strict type annotations (jaxtyping), and immutable dataclass patterns. Unlike prompt-based controls, these constraints actively block execution of non-conforming code.
- Environment Isolation: Agents are executed in a tightly controlled environment: virtual machines or security-constrained containers, with egress filtering and an inference proxy that enables semantic inspection and dynamic rewriting of prompts and responses. This yields strong isolation without the complexity of orchestrator-level multi-tenancy.
- Minimal, Inspectable Tooling: Instead of protocol-bound orchestration, a composable 200-LoC CLI tool built on griffe projects repository information (public APIs, docstrings, type contracts, examples) in response to backend queries. This CLI enables efficient, tractable review sessions by surfacing the most relevant invariants and contracts per function or module, embodying the projection principle: agents and reviewers operate on structured, contract-exposing slices rather than the raw, entire codebase.
- Source as Specification: The codebase is self-describing, with module-level invariants colocated in docstrings. The system avoids distributed external architectural specs, which inevitably diverge during iterative development.
Empirical Evaluation
Experimental Protocol
The empirical analysis uses a synthetic Python codebase (~500 LoC), modified into unconstrained and constrained variants (differing only in substrate enforcement). Eleven backdoors—stratified across detection difficulty (syntactic, semantic, multi-file, in-module, and global invariant violations)—were injected. Review is performed by a fixed-size LLM (Gemma 4 e4b, ~8B params, quantized), with per-function multi-pass review forced by context limits (≤4k tokens per pass), representative of production PR-review mechanics.
Quantitative Results
| Setup |
Recall |
Precision |
| Unconstrained, no tools |
54.5% |
100% |
| Unconstrained, docs CLI |
81.8% |
100% |
| Constrained, no tools |
81.8% |
88.9% |
| Constrained, docs CLI |
90.9% |
95.7% |
- Combined substrate and tooling (constrained codebase + docs CLI) drives reviewer recall from 54.5% to 90.9%, demonstrating the decisive value of substrate constraints and contract projection.
- Independent contributions: Both substrate constraints and tooling provide orthogonal, additive gains. The most challenging backdoors (global invariants, multi-file) are only reliably flagged when invariants are locally visible and dependency contracts are surfaced by the CLI.
- Precision Effects: Substrate constraints alone can lead to slightly reduced precision (i.e., spurious flagging of correct functions), but the CLI restores this by allowing more targeted invariant verification.
- Protocol Sensitivity: Multi-pass, function-local review extracts more findings compared to single-pass, due to reduced context dilution and fixational biases in small models.
Theoretical and Practical Implications
The substrate-constrained approach narrows the oversight asymmetry between a strong (untrusted) coding agent and a weaker (trusted) reviewer. By restricting code generation to be textually self-exposing—via strongly typed signatures, dataclass shape exposure, and local invariants—oversight reduces to invariant checking, a problem tractable even for restricted models. This has clear implications for scalable delegation of oversight: verified constraints compress the adversarial surface and transform code review from holistic comprehension to targeted contract verification.
Critically, the work highlights fundamental flaws in prompt-based steering for agentic code generation, substantiated by adversarial prompt injection literature. Substrate constraints offer a path to enforcement that is orthogonal to (and robust against) the limitations of context size and prompt adherence, offering composability with stronger verification stacks (Rust, Lean), and the potential for neuro-symbolic integration (e.g., FormalJudge-style oversight).
The approach demonstrates compatibility with modern agent workflows; using composable, human-interpretable shells (CLI tools) rather than multi-tenant, protocol-bound orchestration, the system remains lightweight and directly inspectable. The design aligns with modern security best practices for VM isolation and fine-grained egress control, and, by abstracting content-level message inspection out of the VM/agent boundary, is readily extensible (e.g., by layering on NVIDIA OpenShell).
Limitations and Future Directions
The evaluation is limited in scope—synthetic codebase, hand-crafted backdoors, single reviewer model—and further empirical rigor is needed on naturalistic codebases and attack corpora. Open research directions include:
- Scaling up to larger reviewer models to close context comprehension gaps.
- Extending to real-world codebases and adversarially inserted backdoors (e.g., from CVE corpora).
- Rigorous heads-up comparison with mature agent sandboxes (e.g., NVIDIA OpenShell), focusing on integration of semantic content inspection in addition to protocol-level policies.
- Composing substrate constraints with neurosymbolic verifiers and property-based testing to further formalize agent oversight.
The generalization to languages with richer static guarantees (Rust, Lean) and the systematization of overlaying semantic probes across agent/model traffic are prominent directions for further research.
Conclusion
Steerability via substrate constraints, enforceable by the runtime and code structure rather than advisory prompts, provides a robust substrate for scalable, agentic codebase oversight. Early-stage experimentation supports strong gains in oversight recall—up to 90.9% reviewer recall in the presence of both structural constraints and targeted, contract-exposing CLI tools—demonstrating that human-team engineering discipline can be adapted and enforced at lower linguistic and runtime layers, yielding concrete benefits for scalable AI safety and verification. The system's composability, language-agnostic design, and minimal, inspectable footprint set the agenda for further research in robust agent oversight architectures.