Policy-to-Tests: Traceable Test Generation
- Policy-to-Tests (P2T) is a specification-based method that converts natural language policy into executable tests using structured intermediate representations.
- The approach employs canonical representations such as DSARCP and Abstract Violation Templates to ensure each test is directly traceable to its originating policy clause.
- P2T pipelines leverage both rule-based and LLM-driven strategies to generate tests, achieving high coverage, auditability, and validated behavioral outcomes.
Searching arXiv for papers on “Policy-to-Tests” and closely related formulations. Policy-to-Tests (P2T) denotes a family of methods that derive executable tests from an explicit policy artifact, rather than treating testing as an isolated downstream activity. In the literature, the policy artifact varies by domain: natural-language access control statements are translated into Rego and OPA unit tests in Prose2Policy (Gupta et al., 16 Mar 2026); natural-language safety policies are compiled into First-Order Logic and then instantiated into adversarial prompts in POLARIS (Lu et al., 24 May 2026); AI governance documents are normalized into a domain-specific language of executable rules and testable evidence channels in the Policy→Tests framework for executable governance (Datla et al., 4 Dec 2025). Related usages of the same acronym appear in other domains—most notably “Prompt to Transfer” for tabular transfer learning (Nam et al., 2024)—so the term is context-sensitive. In the policy-centric sense, however, P2T refers to pipelines that preserve traceability from policy text or policy abstractions to concrete tests, coverage criteria, or enforcement checks.
1. Definition and scope
Policy-to-Tests is best understood as specification-based test synthesis in which the policy is the primary source of truth. In Prose2Policy, the source policy is a natural-language access control policy describing who can or cannot perform an action on a resource, optionally under conditions or for a purpose; the system converts this prose into DSARCP JSON, Rego, and OPA unit tests (Gupta et al., 16 Mar 2026). In POLARIS, the source policy is a natural-language safety document from companies or regulators, which is compiled into First-Order Logic and then into a Semantic Policy Graph whose traversals yield executable safety probes for LLMs (Lu et al., 24 May 2026). In executable governance for AI, P2T is a framework that converts natural-language policy documents into normalized, machine-readable rules that can then be compiled into executable tests and guardrails (Datla et al., 4 Dec 2025).
Across these systems, the test artifact is domain-specific. In access control, a test is an OPA unit test with an input JSON and an expected boolean decision (Gupta et al., 16 Mar 2026). In LLM safety evaluation, a test is a natural-language query linked to a violating policy clause through an Abstract Violation Template and a graph path (Lu et al., 24 May 2026). In governance-oriented rule extraction, a testable rule is one whose evidence signals include channels such as io_check, log_check, config_check, ci_gate, data_check, repo_check, access_check, or attest_check, and example prompts may be synthesized for rules marked testable (Datla et al., 4 Dec 2025).
This suggests that P2T is not a single algorithm but an architectural pattern: policy formalization, intermediate representation, executable test derivation, and behavioral validation. A plausible implication is that the unifying property of P2T is not the test format itself, but the preservation of a traceable semantics from policy statement to test outcome.
2. Core architectural pattern
The most explicit pipeline decomposition appears in Prose2Policy. Its end-to-end workflow is: pre-processing, component extraction, schema validation, Rego generation and linting, and automatic test generation and execution (Gupta et al., 16 Mar 2026). The pre-processing stage performs policy detection, co-reference resolution, and text segmentation. The extraction stage produces DSARCP, a structured representation consisting of Decision, Subject, Action, Resource, Condition, and Purpose. This DSARCP JSON is the central internal representation from which both Rego and tests are generated (Gupta et al., 16 Mar 2026).
POLARIS instantiates the same pattern with a different formal core. Raw policy text is decomposed into atomic rules; each rule is compiled into an Abstract Violation Template in First-Order Logic; AVTs are mapped into a Semantic Policy Graph; graph traversal identifies compositional violation scenarios; and those scenarios are instantiated into executable natural-language test queries (Lu et al., 24 May 2026). The resulting traceability chain is explicit: policy text to segmented clause, clause to AVT, AVT to graph nodes and edges, graph traversal to path, and path to query (Lu et al., 24 May 2026).
The governance-oriented P2T framework follows a similar structure but emphasizes normalization and downstream enforceability. Its pipeline includes ingestion and chunking, optional clause mining, structured extraction into a DSL, LLM-based judge and repair, de-duplication, testability tagging, and optional example generation (Datla et al., 4 Dec 2025). The output is a set of executable atomic rules with provenance fields such as document identifier, citation, and span_id, plus explicit testability metadata (Datla et al., 4 Dec 2025).
A recurrent design principle is the use of a canonical intermediate representation. DSARCP in Prose2Policy, AVTs in POLARIS, and the JSON DSL in executable governance each serve as a source of truth from which both enforcement artifacts and tests are derived (Gupta et al., 16 Mar 2026, Lu et al., 24 May 2026, Datla et al., 4 Dec 2025). This suggests that P2T systems gain auditability and controllability when policy parsing is decoupled from test materialization by an explicit semantic layer.
3. Policy representations and traceability mechanisms
The policy representation determines both what can be tested and how faithfully tests align with the underlying normative content. Prose2Policy structures a natural-language access control policy as a six-component DSARCP object: decision, subject, action, resource, condition, and purpose (Gupta et al., 16 Mar 2026). The paper states that this representation is derived from natural language via an LLM, constrained by an organization-specific schema, and consumed by both Rego synthesis prompts and test generation (Gupta et al., 16 Mar 2026). Rego rules are annotated with their originating DSARCP for auditability, preserving the link from NLACP to policy code and then to tests (Gupta et al., 16 Mar 2026).
POLARIS formalizes each atomic rule as an AVT of the form
where is constructed from entities, actions, and deontic modality (Lu et al., 24 May 2026). This formalization supports both semantic graph construction and coverage claims. The paper reports a fine-grained score average of $9.10/10$, binary accuracy of , and entity extraction Exact Match , Semantic Match across 13 policy sources (Lu et al., 24 May 2026). Those metrics pertain specifically to the policy-to-logic translation stage.
The governance DSL is intentionally compact and provenance-first. It requires fields such as rule_id, source, [scope](https://www.emergentmind.com/topics/scope), requirement, is_testable, and testability, and includes optional or additional fields such as hazard and severity (Datla et al., 4 Dec 2025). Its purpose is not to prescribe a single executable backend, but to normalize obligations, scope, hazard, and evidence in a form that can later be compiled into guardrails, policy engines, or evaluation prompts (Datla et al., 4 Dec 2025). De-duplication operates both structurally, using a canonical signature over scope, hazard, conditions, exceptions, requirements, and severity, and semantically, via embedding-based merging (Datla et al., 4 Dec 2025).
Traceability is therefore intrinsic rather than auxiliary. In Prose2Policy, auditors can trace NLACP to DSARCP to Rego rule to OPA test (Gupta et al., 16 Mar 2026). In POLARIS, each query retains a machine-readable mapping back to the originating path and policy rule (Lu et al., 24 May 2026). In executable governance, each canonical rule carries source document, citation, and span provenance (Datla et al., 4 Dec 2025). A common misconception is that LLM-generated tests are necessarily opaque; these systems show that opacity is reduced when extraction artifacts are schema-constrained and retained as first-class objects.
4. Test generation paradigms
Two broad P2T paradigms dominate the cited literature: derivation of tests from structured policy components, and derivation of tests from formalized policy semantics.
In Prose2Policy, tests are generated from the DSARCP intermediate representation rather than directly from raw prose or purely from Rego analysis (Gupta et al., 16 Mar 2026). The system supports two modes. Rule-based generation uses deterministic logic over DSARCP, such as providing the DSARCP subject, action, and resource for a positive test and changing one attribute at a time for negative tests. LLM-based generation instead prompts the model to create test inputs and expected decisions from DSARCP and-or NLACP (Gupta et al., 16 Mar 2026). The resulting tests are emitted as OPA unit tests and executed via opa test. Conceptually, for a policy with , a positive test selects an input matching the policy constraints and expects 0, while a negative test 1 violates at least one constraint and expects denial (Gupta et al., 16 Mar 2026).
POLARIS adopts a graph-based generation paradigm. Random walks of length 8 are performed on the Semantic Policy Graph, with 2 to 4 action edges per path and 2 paths per node, producing Abstract Violation Scenarios (Lu et al., 24 May 2026). Those scenarios are then instantiated into natural-language queries using grounding, contextual framing, and intent masking (Lu et al., 24 May 2026). Coverage is a first-class objective: the paper defines Policy Clause Coverage as the percentage of unique policy rules for which at least one violating query was successfully instantiated, and reports 2 clause coverage across 13 policy sources (Lu et al., 24 May 2026). It also defines density-weighted semantic coverage and novelty scores using all-mpnet-base-v2 embeddings and 3 nearest neighbors (Lu et al., 24 May 2026).
Executable governance uses rules rather than policy graphs as the immediate precursor to testing. Only rules marked is_testable = true and with appropriate evidence channels are candidates for direct execution (Datla et al., 4 Dec 2025). When io_check applies, the system can synthesize benign prompts that should pass and adversarial prompts that should violate the rule (Datla et al., 4 Dec 2025). This suggests a distinction between rules that are operationally testable and rules that remain governance metadata or audit artifacts.
The contrast is important. Prose2Policy emphasizes conformance of policy code to intended authorization behavior. POLARIS emphasizes coverage-driven discovery of policy-violating model responses. Executable governance emphasizes normalization of obligations so they can feed multiple enforcement or test backends. These are not competing definitions so much as domain-specific realizations of the same policy-to-test translation principle.
5. Evaluation criteria and empirical results
Because P2T systems target different artifacts, their evaluations emphasize different notions of success: syntactic validity, behavioral consistency, coverage, novelty, or downstream safety impact.
Prose2Policy was evaluated on the ACRE dataset using 485 access control statements from RAGent’s preprocessed output. Of these, 389 were accepted as valid NLACPs, and 371 compiled successfully to Rego, giving a 4 compile rate (Gupta et al., 16 Mar 2026). For the 371 compiled policies, LLM-based test generation achieved an 5 positive-test pass rate 6 and a 7 negative-test pass rate 8; rule-based tests yielded 9 positive-test pass rate and $9.10/10$0 negative-test pass rate (Gupta et al., 16 Mar 2026). The authors explicitly attribute many positive-test failures to limitations in automated test generation for complex policies rather than to defects in the generated Rego (Gupta et al., 16 Mar 2026).
POLARIS evaluates both policy representation quality and test-suite effectiveness. In addition to the AVT translation metrics noted earlier, it reports $9.10/10$1 policy clause coverage and high semantic coverage relative to existing safety benchmarks; at $9.10/10$2, Coverage Scores exceed $9.10/10$3 for most baselines, with examples such as AdvBench at $9.10/10$4, and Novelty Scores remain substantial, including $9.10/10$5 for SOS-Bench (Lu et al., 24 May 2026). On attack discovery, POLARIS consistently achieves higher Attack Success Counts than baselines across six target models. For example, against Gemma, SOS-Bench yields 956 or 1,015 depending on evaluator, whereas POLARIS yields 4,344 or 5,200; against Mistral-7B, POLARIS yields 13,722 or 11,045 (Lu et al., 24 May 2026). The paper emphasizes absolute violation counts rather than rates, explicitly aligning its objective with failure discovery.
Executable governance evaluates extraction quality and downstream behavioral effect. Its corpus summary reports 427 unique rules after de-duplication across the EU AI Act, NIST AI RMF, HIPAA Privacy Rule, and Microsoft Responsible AI Standard v2, with an overall $9.10/10$6 of unique rules marked testable and 50 rules associated with io_check evidence (Datla et al., 4 Dec 2025). Document-level metrics include Span F1 values of 0.79 for EU AI Act, 0.83 for NIST AI RMF, 0.75 for HIPAA, and 0.84 for Microsoft RAI, with structured extraction slot similarities ranging from 0.24 to 0.30 (Datla et al., 4 Dec 2025). In the HIPAA case study, adding three HIPAA-derived guardrails reduced overall violation rate from 0.34 to 0.05, obfuscated-prompt violations from 0.58 to 0.08, and compositional-prompt violations from 0.42 to 0.06 (Datla et al., 4 Dec 2025).
These results show that “test quality” in P2T is multi-dimensional. A compiled policy with high negative-test pass rate does not guarantee exhaustive semantic correctness (Gupta et al., 16 Mar 2026). A safety suite with full clause coverage still depends on the fidelity of the policy-to-logic translation and graph exploration heuristics (Lu et al., 24 May 2026). A governance rule corpus with good span-level agreement still requires subsequent compilation or runtime enforcement to affect behavior (Datla et al., 4 Dec 2025).
6. Limitations, ambiguities, and related meanings of “P2T”
The central limitations of P2T recur across domains. In Prose2Policy, the hardest cases involve external constants, nested or compound conditions, contextual variables, type conversions, or time-based evaluations; these complicate both policy synthesis and test construction (Gupta et al., 16 Mar 2026). The paper also notes non-determinism of LLM outputs and proposes future decoupling of policy and test generation to reduce correlated errors (Gupta et al., 16 Mar 2026). POLARIS identifies dependence on policy quality, residual FOL translation errors, heuristic graph exploration, and restriction to single-turn prompts as major limitations (Lu et al., 24 May 2026). Executable governance highlights ambiguity, nonlocal qualifiers, nested exceptions, and DSL compression of complex temporal or probabilistic requirements as failure modes (Datla et al., 4 Dec 2025).
Another source of ambiguity is terminological. “P2T” is not unique to policy-derived testing. In tabular machine learning, P2T denotes “Prompt to Transfer,” a prompting-based framework that uses source data to create pseudo-demonstrations for target prediction tasks (Nam et al., 2024). That usage is unrelated to policy-driven test generation, despite sharing the acronym. A further complication is that adjacent literature sometimes studies policy–test relationships in a different sense: RPOSST composes small sets of test cases for selecting reinforcement-learning policies rather than generating tests from policy specifications (Morrill et al., 2023). This suggests that the phrase “policy-to-tests” should be disambiguated by domain whenever precision matters.
A broader historical perspective also exists. Earlier work such as Armstrong converted context-dependent network policies into tests over stateful data planes using Dynamic Policy Graphs and finite-state abstractions (Fayaz et al., 2015). That work predates the recent LLM-centered P2T pipelines and shows that the policy-to-tests idea is not confined to natural-language policy extraction. The common thread is still specification-led test synthesis under explicit coverage or correctness criteria.
The contemporary literature therefore supports a narrow but technically robust meaning of Policy-to-Tests: a traceable transformation from policy specification to executable tests or guardrails, mediated by an explicit semantic representation and validated by behavioral evidence. Where the term is used differently, the underlying methodology rather than the acronym determines whether a system belongs to this lineage.