Reproduce PolicyGuard and NOMOS in a matched evaluation

Conduct a same-harness comparison of PolicyGuard and the NOMOS deterministic gate on -bench airline to determine their relative performance under identical evaluation conditions.

Background

The paper compares NOMOS with several systems for compiling written policies into deterministic enforcement mechanisms. PolicyGuard is discussed as a system that compiles policy documents into tool-level checklists but uses an LLM verifier at runtime, whereas NOMOS compiles policies into deterministic gates that make no runtime LLM calls.

The authors explain that PolicyGuard has subsequently evaluated ToolGuard on -bench airline, but that the available materials do not provide a directly matched comparison with NOMOS. Establishing such a comparison would clarify the relative task success, safety, latency, and runtime-cost trade-offs of the two approaches under the same benchmark harness.

References

PolicyGuard has since run ToolGuard as a baseline on $\tau2$-bench airline; a same-harness comparison with the gate remains open.

— NOMOS: Compiling Written Policies into Statically Verified Tool-Call Gates for LLM Agents  (2610.11030 - Yu et al., 8 Oct 2026) in Related Work, subsection “Policy compliance for tool-using agents”