Open-Domain Safety Policy Construction

Abstract: Moderation layers are increasingly a core component of many products built on user- or model-generated content. However, drafting and maintaining domain-specific safety policies remains costly. We present Deep Policy Research (DPR), a minimal agentic system that drafts a full content moderation policy based on only human-written seed domain information. DPR uses a single web search tool and lightweight scaffolding to iteratively propose search queries, distill diverse web sources into policy rules, and organize rules into an indexed document. We evaluate DPR on (1) the OpenAI undesired content benchmark across five domains with two compact reader LLMs and (2) an in-house multimodal advertisement moderation benchmark. DPR consistently outperforms definition-only and in-context learning baselines, and in our end-to-end setting it is competitive with expert-written policy sections in several domains. Moreover, under the same seed specification and evaluation protocol, DPR outperforms a general-purpose deep research system, suggesting that a task-specific, structured research loop can be more effective than generic web research for policy drafting. We release our experiment code at https://github.com/xiaowu0162/deep-policy-research.

• The paper introduces DPR, which utilizes iterative query generation, rule extraction, and keyphrase clustering to autonomously draft safety policies.
• It demonstrates improved moderation performance, with F1 scores rising from 0.810 to 0.831 for Qwen2.5 7B and 0.752 to 0.792 for Llama 3.1 8B.
• The framework reduces human labor in policy drafting and paves the way for advancements in alignment, multimodal policy drafting, and rule-based training.

Open-Domain Safety Policy Construction: An Expert Analysis

Problem Framing and System Overview

"Open-Domain Safety Policy Construction" (2604.01354) presents Deep Policy Research (DPR), a minimal agentic pipeline for autonomously synthesizing actionable content moderation policies from a one-sentence domain specification and web search. The motivation is the persistent bottleneck in drafting and maintaining high-quality, domain-specific safety policies for user- and model-generated content, which historically demands significant human labor, domain expertise, and iterative updates. DPR reframes this as an open-domain construction task—taking only concise seed guidance and leveraging web search to expand, distill, and structure policy rules iteratively for downstream use.

DPR operates via three core stages per iteration: targeted query generation meant to explore policy gaps, extraction and consolidation of rule predicates from retrieved web sources, and organization of rules through keyphrase-based clustering to produce an indexed and readable policy document. This cyclic agentic process is designed to promote comprehensive coverage, actionable precision, and systematic policy structure without extensive human feedback or agent orchestration.

Figure 1: Illustration of Deep Policy Research workflow: iterative web search, rule extraction, and keyphrase-based clustering from a seed specification.

Methodological Design

Query Generation and Web Integration

Each DPR iteration begins with analysis of the current policy document and index to identify coverage gaps or ambiguity. An LLM generates highly targeted web search queries (prompted to seek definitional boundaries, edge cases, enforcement signals, and high-risk subtypes) and extracts the top $m$ relevant snippets from web search.

Rule Extraction and Consolidation

Using the evidence corpus, the LLM formulates candidate rules in a consistent predicate schema, each clearly separated by decision boundary and qualifiers. A subsequent self-critique phase filters out irrelevant, ambiguous, or redundant rules, leveraging both cross-source corroboration and source quality as consolidation signals. This tight feedback loop ensures both coverage and precision.

Keyphrase-Based Indexing

To enhance navigation and contextualization—critical for long-context reader LLMs—DPR employs keyphrase extraction per rule, clusters rules using k-means, and synthesizes section titles and summaries per cluster, resulting in a hierarchical and indexed policy. This structure directly supports in-context reasoning and reduces inference brittleness in downstream moderation.

Evaluation and Results

DPR is evaluated extensively across two settings:

• The OpenAI undesired content benchmark, spanning five canonical moderation domains (Sexual, Hate, Violence, Harassment, Self-Harm), with compact LLMs as readers.
• An in-house multimodal advertisement moderation dataset, measuring DPR’s generalizability to vision-language policy domains.

Convergence and Policy Diversity

Empirical analysis in the Harassment domain demonstrates that the number of unique rules and keyphrase clusters grows rapidly in initial iterations and plateaus by $k=3$, supporting the efficiency and coverage sufficiency of the iterative approach.

Figure 2: Rule count and cluster count per iteration for Harassment, confirming rapid convergence and diminishing returns.

Performance on Content Moderation Benchmarks

On the OpenAI benchmark, DPR-attached policies universally enhance downstream moderation F$_1$ compared to definition-only and in-context-learning baselines, as well as outperforming a generic deep research agent under identical seeding and protocols. Notably, for Qwen2.5 7B, average F$_1$ increases from 0.810 to 0.831 with DPR; for Llama 3.1 8B, from 0.752 to 0.792. Performance gains are most pronounced in subjective domains (Violence, Harassment, Self-Harm), with negligible or no regression in well-specified domains. DPR’s task structure and hierarchical output confer advantages beyond what simple search-and-summarization baselines achieve.

Figure 3: Iterative performance gains in the Harassment domain with Qwen2.5 7B as reader, illustrating the effect of each DPR iteration versus competing approaches.

Figure 4: Iterative performance profile for Llama 3.1 8B reader, highlighting the policy utility improvement trajectory with DPR.

In the multimodal ad setting, substituting a domain section with a DPR-generated policy recovers substantial moderation accuracy relative to removing the section or relying on a one-sentence spec. For example, in the Misrepresentative domain, DPR achieves 0.727 F$_1$ (single sample), nearly closing the gap to the expert-written 0.740. For domains where public conventions are present (e.g., Exploitative, Offensive), DPR achieves near parity with human policy; however, for organization-specific or compliance-heavy domains (Finance Claims), gaps persist, underscoring the limits of web-based synthesis for proprietary standards.

Structural and Source Analyses

An analysis of rule and cluster dynamics shows DPR’s coverage saturates quickly, and indexing strategies (clustered vs. flat policies) affect reader LLMs differently, depending on their long-context capabilities. Additionally, DPR sources are shown to be diverse, drawing from a mix of Wikipedia, governmental content, academic articles, and relevant web posts.

Figure 5: Domain distribution of sources accessed by DPR, evidencing its broad web coverage across both OpenAI and in-house settings.

Practical and Theoretical Implications

The work offers a systematized approach for LLM-driven, open-domain policy drafting that demonstrably improves moderation without requiring intensive manual curation or continual policy writing. The indexing and distillation techniques show that minimal agentic setups—when coupled with structured prompts and modest human scaffolding—are capable of producing actionable, high-utility domain policies.

Theoretically, the efficacy of the task-specific loop over general-purpose research agents suggests that structure, schema, and policy format bias have outsized impact on LLM-driven policy construction utility. The framework further supports developments in alignment by facilitating rapid, reproducible policy drafting to be used in deliberative alignment, chain-of-thought safety reasoning, and rule-based reward training.

Future Directions

Promising future research avenues include extension to richer multimodal or longitudinal policy drafting, integration with interactive human feedback for contextually sensitive or organizational domains, and optimization of policy presentation for specific reader architectures. Exploring LLM-generated illustrative examples alongside rule predicates, developing adaptive section indexing in response to downstream model behavior, and evaluating for policy drift over real product update cycles are immediate next steps.

Conclusion

DPR demonstrates that LLM-powered open-domain safety policy construction is both feasible and effective. The iterative, agentic research process, grounded in web search and lightweight scaffolding, yields policies that consistently improve automated moderation accuracy and approach the fidelity of expert-drafted guidelines in many domains. For practitioners, DPR provides a practical foundation for semi-automated policy development and maintenance, while for researchers, it exposes new questions around agent design, rule generalization, and the limits of open-domain synthesis for regulated and proprietary contexts.