HaloGuard 1.0: An Open Weights Constitutional Classifier for Multilingual AI Safety
Abstract: We present HaloGuard 1.0, an open-weights implementation of the constitutional-classifier paradigm for input safety. It achieves state-of-the-art performance on English and multilingual prompt-safety benchmarks at roughly one-tenth the model size of current leading open guard models. The safety constitution is the organising structure of the corpus: a natural-language constitution of 46 policies and 2,940 subcategories drives synthetic data generation, with exhaustive one-to-one paired counterfactuals that hold topic and vocabulary fixed while flipping intent, a two-tier harmless design that separately targets boundary and baseline false positives (FPs), and balanced multilingual materialisation across 46 languages that treats language as a surface form appearing on both sides of the boundary rather than as an adversarial signal. Across seven prompt-safety benchmarks, HaloGuard 1.0-0.8B attains the best average F1 (90.9) of any open guard we evaluate, outperforming baselines up to 27B parameters (over 30 times larger) while holding false-positive rate (FPR) to 4.3 and false-negative rate (FNR) to 9.5. The HaloGuard 1.0-4B variant reaches average F1 of 92.1 and FPR of 3.5, spending its extra capacity on precision rather than recall. A structured adjudication of the remaining failures indicates that most apparent missed-harm cases are benchmark mislabels rather than genuine model misses. An always-on adversarial red-teaming protocol continuously hardens the guard against both content-level and agentic attacks. We release the models as open weights.
Summary
- The paper introduces a constitutional-driven classifier that generates 46 policies and 2,940 subcategories using paired counterfactual data to effectively reduce false positives and negatives.
- It employs a fine-grained synthetic data pipeline and sliding-window inference to achieve robust multilingual prompt safety with models as compact as 0.8B parameters.
- The approach supports continuous adversarial red-teaming and auditability, providing a reproducible framework for safe deployment in diverse, agentic, and multilingual scenarios.
HaloGuard 1.0: Open-Weights Constitutional Classification for Robust Multilingual AI Safety
Motivation and Problem Setting
The expansion of LLMs from chat-based messaging to agentic use cases introduces substantially larger attack surfaces and more severe downstream risks. Input guards, serving as the first layer of defense, must operate on the prompt side to block unsafe requests before generation. However, the core challenge is not simply detecting overtly harmful prompts; it is the identification and correct adjudication of boundary cases—prompts that share safety-sensitive tokens with legitimate requests but differ in user intent. Over-reliance on lexicon results in frequent false positives (FPs) with over-refusal, while excessive permissiveness increases false negatives (FNs), allowing policy violations to pass.
HaloGuard 1.0 directly targets this FP/FN frontier by employing a fine-grained, constitution-driven methodology. Rather than using a static, flat taxonomy post hoc, the constitution itself generates 46 policies and 2,940 subcategories, providing nuanced coverage and systematic counterfactual data synthesis. The models are released as open weights at two scales (0.8B and 4B parameters), demonstrating state-of-the-art performance across English and multilingual prompt-safety benchmarks with significant efficiency gains relative to previous open guard models.
Constitutional Grounding and Taxonomy
HaloGuard 1.0 is architected on a detailed natural language constitution, which serves as both the generative specification for synthetic data and as a mechanism for auditing coverage and failures. The hierarchical taxonomy decomposes policies into clusters and hundreds of categories and subcategories, facilitating exhaustive hard-negative construction and localization of failures. Four risk tiers (critical, high, moderate, sensitive) inform boundary width and data composition strategies, though the runtime interface remains binary for deployment simplicity.
The critical advance is the systematic pairing of harmful and harmless-boundary data for each subcategory. The benign mirror retains topic and harm-adjacent lexicon but alters intent, grounded in legitimate purposes such as education, journalism, clinical, defensive security, or legal review. This approach circumvents the keyword shortcut failure mode, ensuring the classifier distinguishes unsafe intent rather than simply unsafe phrasing.
Figure 1: The HaloGuard 1.0 constitutional pipeline contrasts conventional flat taxonomy collection with constitution-driven synthetic generation across 46 policies and 2,940 subcategories.
Synthetic Data Pipeline and Hard Negatives
HaloGuard 1.0’s data pipeline is constitution-centric, generating exhaustive paired counterfactuals, boundary-focused harmless examples, adversarial overlays, and deterministic surface transformations. Style anchoring—retrieving real register prompts from stratified pools—bridges the gap between synthetic and real deployment distributions. Surface augmentation through overlays and encodings, crossed rigorously over generation grids, ensures robustness against adversarial indirection and obfuscation.
Paired counterfactuals—holding topic and surface cues constant while flipping intent—directly target boundary discrimination. For each harmful prompt, a one-to-one benign mirror is generated, with both grounded in real legitimate anchors. This systematic pairing and surface matching extend coverage and label reliability well beyond prior guard systems.
Figure 2: For each harmful subcategory, a benign mirror is constructed with topic and vocabulary fixed, isolating intent as the sole discriminative variable.
FP/FN Frontier and Data Composition
Distinct strategies are applied for boundary FPs (benign prompts near harmful edges) versus baseline FPs (ordinary benign prompts misclassified due to spurious features). Tier 1 data (harmless-boundary) finely targets constitution-local boundaries, while Tier 2 (shared-harmless) anchors general benign coverage, informed by adversarial confusion link metadata.
The composition is managed via tunable pool fractions, ensuring balance across the harmful/harmless boundary and avoiding shortcut learning. Quality control gates enforce consistency, operational boundary, and refusal filtering, with splits governed by deterministic hash at the conceptual component level to prevent contamination between training, evaluation, and test sets.
Figure 3: HaloGuard 1.0 targets the FP/FN frontier using boundary-focused data, paired counterfactuals, surface matching, and calibrated thresholds.
Multilingual Expansion and Register Preservation
Robust multilingual safety is achieved by translating base and paired counterfactuals into 46 languages, with per-language naturalness contracts, register preservation, and script fidelity. Overlays and deterministic transforms are applied to translated data, further extending robustness to surface-level attacks.
Quality controls include semantic refusal detection, degeneracy filtering, and intent-preservation verification; lower-resource languages adapt verification protocols to maximize accurate coverage. Composition controls maintain per-language balance and mitigate dilution of English signal, with explicit caps and stratified sampling.
Model Architecture, Training, and Inference
The input guard is implemented as generative classifiers on Qwen3.5 decoder architectures (0.8B and 4B scales), conditioned to emit constitution-attributed labels. Training employs autoregressive cross-entropy loss on finely packed prompt sequences, optimizing for both capacity and FP/FN balance via FullFinetune adapters.
Thresholds are calibrated per category based on held-out data, supporting diverse deployment tiers. The smaller model (0.8B) serves as a low-latency hot-path filter, while the larger variant (4B) is deployed for higher-risk or adjudication scenarios. Sliding-window inference enables robust coverage for unbounded inputs, flagging any unsafe window in long sequences and mitigating truncation-based adversarial attacks.
Figure 4: Sliding-window classification partitions long inputs; each window is independently classified, reducing blind spots from naive truncation.
Safety Benchmarking and Numerical Results
HaloGuard 1.0 delivers strong numerical results: HaloGuard 1.0-0.8B achieves macro-F1 of 90.9 with FPR 4.3% and FNR 9.5% across seven prompt-safety benchmarks—outperforming open guards up to 27B parameters. HaloGuard 1.0-4B further improves macro-F1 to 92.0 and reduces FNR to 7.7%. Significantly, these strengths are not restricted to single benchmarks; both models avoid severe performance drops on any column and maintain robustness across adversarial, conversational, and refusal-oriented tasks.
After structured adjudication, much of the remaining missed-harm (FNs) is attributed to benchmark mislabels—demonstrating the boundary-centric corpus achieves high reliability at the decision margin.
Figure 5: Average prompt-harmfulness F1 versus model size—HaloGuard 1.0-0.8B leads the compact frontier, while HaloGuard 1.0-4B achieves the highest reported F1.
Multilingual Generalization and Limitations
Multilingual evaluation on PolyGuardPrompts (17 languages) reveals strong generalization: HaloGuard 1.0-0.8B achieves macro-F1 of 86.1 and HaloGuard 1.0-4B reaches 88.0, level with or above PolyGuard baselines of much larger parameter counts. However, over-refusal remains elevated in Indic and Southeast Asian languages, driven by thin in-language safety coverage; capacity upgrades narrow but do not resolve this boundary, indicating a need for targeted shared-harmless augmentation.
Figure 6: Cross-lingual guard performance—FNR remains low across groups, but FPR spikes for Indic/Southeast Asian languages, highlighting remaining over-refusal pitfalls.
Continuous Red-Teaming and Robustness
HaloGuard 1.0’s robustness is further enhanced by a continuous adversarial red-teaming protocol: cyclic incentivized adversarial submissions are curated and used for retraining, continuously hardening the model against both content-level and agentic attacks. Exact-string matching for agentic failures prevents judge-only evaluation pitfalls.
Figure 7: The continuous adversarial red-teaming loop maintains ongoing guard hardening and attack-distribution diversity.
Implications and Future Directions
Practically, HaloGuard 1.0 provides a deployable, reproducible, and auditable solution for pre-generation prompt moderation, enabling stringent policy enforcement without excessive refusal of benign traffic. The constitutional-driven pipeline is generalizable, supporting policy adaptation, rigorous boundary auditing, and systematic coverage expansion.
Theoretically, this approach demonstrates the utility of natural language constitutions for synthetic data generation, robust boundary learning with counterfactuals, and surface-matched augmentation—methods anticipated to influence future LLM safety and moderation frameworks, especially as deployments expand into mixed-language, agentic, and multimodal domains.
Future research includes extending coverage to response-side and streaming moderation, code-switched and mixed-language inputs, and agentic/tool-use safeguards. The continuous red-teaming protocol is expected to supply empirical hardening curves, transforming robustness into a measurable trend. Expansion into ambient toxic speech detection will require dedicated taxonomy and methodology developments.
Conclusion
HaloGuard 1.0 exemplifies rigorous, data-centric guard design by making the safety constitution the central generative mechanism for synthetic corpus construction. Exhaustive paired counterfactuals, boundary-focused harmless data, tiered taxonomy, and multilingual materialization collectively deliver superior boundary discrimination and transfer across languages, with strong performance at a fraction of competing model sizes. Open release of models, pipelines, and constitutions provides a reproducible path for broader AI safety research, policy mapping, and adversarial robustness.
Reference: "HaloGuard 1.0: An Open Weights Constitutional Classifier for Multilingual AI Safety" (2607.02079)
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper introduces HaloGuard 1.0, a “safety guard” for AI systems that read and answer user prompts. Think of it like a smart bouncer at a concert: it checks what people are asking the AI to do before the AI responds. Its job is to block truly harmful requests (like asking how to commit a crime) while allowing harmless requests that just mention sensitive topics (like a school report about cyberattacks). HaloGuard is special because it’s:
- Open-weights (anyone can use and inspect the model),
- Multilingual (works across many languages),
- Built around a detailed written rulebook (a “constitution”) that teaches it to tell harmful intent apart from harmless discussion.
Key Objectives
The paper aims to:
- Build a reliable first line of defense that checks user prompts before the AI generates any answer.
- Reduce two kinds of mistakes: false positives (blocking safe prompts) and false negatives (letting harmful prompts through).
- Handle tricky “boundary” cases, where the same words can be used for bad or good purposes (for example, “malware” in a hacking guide vs. a cybersecurity class).
- Work well across many languages without treating non-English text as automatically suspicious.
- Share an open, smaller model that still matches or beats much larger safety models.
Methods and Approach
To make the guard both careful and fair, the authors use a few key ideas:
- A detailed rulebook (“constitution”): Instead of a short list of categories like “violence” or “self-harm,” HaloGuard uses 46 policies and 2,940 fine-grained subcategories. This rulebook doesn’t just name bad behaviors; it also lists the legitimate, safe uses nearby (like history, education, journalism, or victim support). That helps the model focus on intent, not just keywords.
- Paired “twin” examples that look similar but flip the intent: For each harmful subcategory, the team generates a matching safe example that uses the same topic and words but is clearly harmless. Imagine two prompts about “arson”: one asks how to start a fire (harmful), the other analyzes how investigators detect arson (harmless). Training on these twins teaches the model to avoid the “keyword shortcut” and pay attention to what the user actually wants.
- Realistic style: Synthetic data can sound too perfect. The authors “anchor” generated prompts to the style of real-world texts (length, tone, phrasing), so the model learns from prompts that feel like what users actually write.
- Coverage “grid”: Each topic is generated across different forms (like direct questions, roleplay, or hypotheticals), different depths (high-level vs. step-by-step), and formats (prose, lists, Q&A). This stops the model from only learning one narrow way a harmful idea might appear.
- Three data buckets:
- Harmful prompts (what to block),
- Harmless boundary prompts (safe but near a danger zone),
- Shared harmless prompts (ordinary safe content like education, coding, health info).
- This helps control different kinds of false positives.
- Multilingual balance: The same safe and unsafe patterns are created across 46 languages, so language itself isn’t treated as a “gotcha.”
- Continuous “red teaming”: The team constantly tries to break the model with new tricks (like roleplay, hidden instructions, or odd formatting) and then retrains to harden it.
Technically, HaloGuard is a “generative classifier” built on Qwen models (0.8B and 4B parameters). Instead of just checking boxes, it outputs a decision plus the relevant policy category in plain language, which helps developers understand and route safety issues.
Main Findings and Why They Matter
HaloGuard 1.0 performs strongly across seven safety benchmarks (covering harmful prompts, toxic chats, adversarial attacks, and over-refusal tests):
- The 0.8B model achieves an average F1 score of 90.9, with a low false-positive rate (~4.3%) and a low false-negative rate (~9.5%).
- The 4B model pushes F1 to 92.0 and lowers false positives further (~3.5%), trading a bit of recall for higher precision (fewer innocent prompts get blocked).
- It beats or matches open baselines up to 27B parameters (over 30 times larger) while staying much smaller and faster.
- Many “missed harm” cases in benchmarks turned out to be mislabeled data, not actual model failures, according to the authors’ reviews.
Why this matters:
- Better balance: It blocks genuinely unsafe requests while preserving legitimate ones, which is crucial for real apps that serve students, professionals, and general users.
- Efficiency: Being small but strong means it can run quickly and cheaply as a first-layer safety check.
- Transparency: Open weights let researchers and builders audit, improve, and adapt the guard to their own policies.
- Global reach: Multilingual design makes safety more inclusive and reduces unfair treatment of non-English users.
Implications and Potential Impact
HaloGuard 1.0 is designed to sit in front of AI assistants, agents, and tools. It helps decide whether to allow, block, log, escalate, or route a request (for example, to a larger checker or to human review). Because it explains which policy was triggered, teams can tailor responses (e.g., auto-block a high-confidence cyber-abuse prompt but send a borderline medical query to a professional review).
In simple terms:
- Safer AI apps: Lower risk of harmful actions or advice slipping through.
- Less over-blocking: Fewer frustrated users who are asking legitimate questions about sensitive topics.
- Easier to audit: The fine-grained rulebook and open model make it clearer why a decision was made.
- Foundation for more: This release focuses on input prompts; the authors plan to add response and streaming output moderation later, creating a more complete safety stack.
Overall, HaloGuard 1.0 shows that a carefully designed, rules-driven, open, and multilingual safety guard can be both strong and fair—helping AI systems be useful without being risky.
Knowledge Gaps
Below is a single, actionable list of the paper’s knowledge gaps, limitations, and open questions that remain unresolved.
- Output-side moderation is out of scope: no response classification, streaming/token-level output guard, or prompt–response pair moderation; integration plans and the impact on the FP/FN frontier are unspecified.
- Real-world deployment validation is missing: no live A/B tests, on-call telemetry, or longitudinal studies of FP/FN and user impact in production across diverse applications.
- Reproducibility and external auditability are limited: open weights are released, but the synthetic corpus, full training scripts, detailed hyperparameters, and evaluation code are not clearly provided for independent replication.
- Benchmark reliance vs. real traffic: claims about benchmark mislabels lack third-party adjudication; need independent re-labeling studies and benchmarks reflecting realistic, messy, multilingual user distributions.
- Constitution governance is unspecified: how policies are authored, reviewed, versioned, and updated; who the stakeholders are; and how cross-jurisdictional/regulatory needs are incorporated.
- Multi-policy support is undeveloped: the approach appears tailored to a single constitution; mechanisms to customize, swap, or blend constitutions per-deployment (and measure drift when switching) are not detailed.
- Runtime calibration is underexplored: methods to set thresholds per application and risk tier, confidence calibration (e.g., ECE/Brier), and cost-sensitive operating points (precision/recall trade-offs) are not reported.
- Multilingual robustness gaps: beyond “46 languages,” there is no quantified per-language FPR/FNR, nor coverage for code-mixing, dialects, transliteration, non-standard orthographies, and low-resource scripts.
- Fairness and disparate impact are unmeasured: no analysis of over-refusal or missed harm across languages, dialects, protected groups, reclaimed/historically sensitive language, or culturally specific contexts.
- Threat model is not formalized: continuous red-teaming is described, but the scope, attack taxonomies, refresh cadence, measurable gains, and coverage of obfuscations (base64/ROT13, homoglyphs, zero-width, leetspeak), payload splitting, and tool-mediated attacks are not quantified.
- Long-context and conversation handling is unclear: sliding-window classification is mentioned, but performance, context-length limits, multi-turn injection handling, and memory of prior turns are not evaluated.
- Agentic/tool-chain scenarios are not evaluated: prompts that trigger tool calls, system prompts, and cross-surface instructions (e.g., retrieved content, logs) remain untested in guard placement and effectiveness.
- Generative vs. discriminative classifier trade-offs are not assessed: latency, throughput, stability of label emission, multi-label consistency, and resource usage compared to classification heads are not benchmarked.
- Explanation quality is limited: the model returns category attribution and scores but not rationales, evidence spans, or structured justifications to aid auditing, appeals, and user feedback.
- Runtime severity granularity is missing: collapsing fine-grained constitution to a binary verdict discards severity/context; graded risk/severity signals and triage pathways are not explored.
- Overfitting risk to paired counterfactuals: generalization to unpaired, naturally occurring benign-hard negatives and messy real prompts is not demonstrated; need contrastive tests beyond mirrored pairs.
- Coverage completeness of the 2,940 subcategories is uncertain: sector-specific harms (e.g., medical devices, finance), regional regulations, and emerging domains may be underrepresented; mapping to external standards (NIST/ISO/regulatory taxonomies) is absent.
- Evaluation metrics are limited: beyond F1/FPR/FNR, calibration, ROC/PR curves, cost-sensitive metrics, utility-aware measures (e.g., refusal recovery rates), and per-constitution breakdowns are not reported.
- Inference resource/latency characteristics are missing: there are no CPU/GPU latency, memory footprint, batching, throughput, or mobile/edge feasibility results for both 0.8B and 4B variants.
- Data contamination safeguards need verification: style-anchoring uses real pools; stronger evidence of decontamination (e.g., exact-match/hash/blocklist checks against test splits) and leakage audits is needed.
- Red-team artifacts are not openly available: lack of a public adversarial set (or synthetic recipes) limits external stress-testing and apples-to-apples robustness comparisons.
- Multi-modal prompts are out of scope: handling images, audio, PDFs, code attachments, URLs, or embedded files, and cross-modal obfuscation are not addressed.
- PII/privacy moderation specifics are thin: thresholds, de-identification strategies, and false-positive handling for clinical/legal contexts (where privacy and utility conflict) require concrete evaluation.
- Confidence scoring reliability is unassessed: how generation probabilities translate to well-calibrated scores per category, monotonicity under adversarial framing, and stability across languages are not analyzed.
- Usability and recovery flows are unstudied: user experience costs of refusals, clarification prompts, escalation flows, and guard–assistant coordination are not measured.
- Guard security is not discussed: risks of prompt injection targeting the guard, side-channel attacks on generative decoding, and sandboxing/isolation strategies are not addressed.
- Component ablations are absent: the contribution of style anchoring, coverage axes, paired counterfactuals, judge gates, adversarial overlays, and multilingual materialization to final performance is not quantified.
Practical Applications
Immediate Applications
Below are concrete, deployable use cases that can be implemented today using the paper’s open‑weights, multilingual, constitution‑attributed input guard (HaloGuard 1.0–0.8B and 4B). Each item notes sectors, potential tools/workflows, and key dependencies or assumptions.
- Sector: Software/AI Platforms (SaaS, DevTools)
- Use case: Pre‑generation safety gateway for chatbots, code assistants, RAG systems, and tool‑using agents. Reduce unsafe prompts while minimizing over‑refusals near policy boundaries.
- Tools/workflows: Sidecar microservice or reverse proxy that intercepts prompts; category‑based routing (block/allow/escalate), confidence‑threshold tuning by risk tier; SDK integrations (LangChain/LlamaIndex “guard node”).
- Dependencies/assumptions: Meet latency budgets (0.8B for edge/CPU, 4B for GPU or quantized CPU); calibrate FP/FN thresholds to application tolerance; ensure Qwen3.5‑based license compatibility.
- Sector: Enterprise IT/Security
- Use case: Guardrails for internal assistants and copilots to prevent data exfiltration prompts, social‑engineering guidance, or operational cyber misuse while allowing defensive/forensic analysis.
- Tools/workflows: Constitution‑attributed logging for SOC review; category‑specific auto‑block (e.g., “cyber misuse” high‑confidence), human escalation for ambiguous “regulated advice.”
- Dependencies/assumptions: Policy mapping from the 46 policies to enterprise policies; on‑prem deployment for privacy; incident workflows for escalations.
- Sector: Customer Support and CX
- Use case: Protect brand and compliance in customer‑facing support bots while permitting educational/legal discussion of sensitive topics (e.g., fraud awareness, safety tips).
- Tools/workflows: Risk‑tiered response strategies (deny + safe alternative, clarify intent, or handoff to human); telemetry dashboards reporting constitution categories and confidence.
- Dependencies/assumptions: Localization and multilingual coverage across 46 languages; set per‑market enforcement thresholds; maintain low‑latency targets.
- Sector: Finance and Fintech Compliance
- Use case: Block prompts requesting actionable investment advice, fraud facilitation, or PII exploitation; preserve benign compliance education or risk disclosure content.
- Tools/workflows: Category‑based gating integrated with CRM/communication platforms; auto‑disclaimer injection and human‑in‑the‑loop for “regulated advice” categories.
- Dependencies/assumptions: Map constitution categories to firm’s policy and jurisdiction; audit logging to satisfy regulatory evidence requirements.
- Sector: Healthcare and Telehealth
- Use case: Triage and guard for self‑harm, eating disorders, and medical advice prompts; allow support‑seeking and psychoeducation while blocking instructions for harm.
- Tools/workflows: “High‑risk” auto‑escalation to crisis resources/human clinicians; safe responses templates for support‑seeking; confidence‑threshold tuning for minimal false negatives.
- Dependencies/assumptions: Clinical escalation pathways and disclaimers; jurisdictional compliance (HIPAA/GDPR); ongoing calibration with clinical SMEs.
- Sector: Education and EdTech
- Use case: Classroom or tutoring assistants that distinguish educational discussion of dangerous topics (history of terrorism, lab safety) from requests for operational misuse.
- Tools/workflows: Course‑context policy profiles (stricter for K‑12, different for university labs); category‑aware constraints and safe alternatives (e.g., “historical analysis”).
- Dependencies/assumptions: Age‑appropriate thresholds; educator oversight; content standards alignment.
- Sector: Public Sector and Policy
- Use case: Multilingual moderation front‑end for citizen‑facing AI services, balancing access with safety; transparent, auditable policy attribution for oversight.
- Tools/workflows: Constitution‑attributed dashboards, per‑policy routing (e.g., “privacy violation” → auto‑block + report); publish policy mapping for transparency.
- Dependencies/assumptions: Policy harmonization with agency rules; public audit trails; on‑prem or sovereign‑cloud deployment.
- Sector: Cybersecurity (Blue Teams/Vendors)
- Use case: Differentiate offensive cyber misuse from defensive analysis, allowing blue‑team content (threat detection, forensics) while blocking malware authoring requests.
- Tools/workflows: Pre‑generation filter in internal SOC tooling; defender‑side “direction‑of‑intent” prompts and templates to clarify safe context.
- Dependencies/assumptions: Regular calibration via red‑teaming; category thresholds adapted to lab/testing environments.
- Sector: Social/Community Platforms Integrating AI
- Use case: Safety gate for creator tools or AI assistants embedded in apps; mitigate toxic or illegal content requests without over‑blocking benign but sensitive conversations.
- Tools/workflows: Inline guard before passing prompts to generative models; telemetry and A/B testing on FP/FN frontier; multilingual support for global user bases.
- Dependencies/assumptions: Trust & safety review processes; scalability under peak loads; compliance with content policies.
- Sector: ML Ops and Governance
- Use case: Auditable policy artifacts and constitution‑attributed logs for model governance; fine‑grained failure analysis by subcategory.
- Tools/workflows: Observability stack (e.g., Prometheus/Grafana) for category/confidence metrics; change‑management workflows tied to constitution updates.
- Dependencies/assumptions: Organizational policy mapping to the 46‑policy schema; data retention/privacy guardrails.
- Sector: SMEs/Edge/On‑Device Assistants
- Use case: Low‑compute, open‑weights guard for privacy‑sensitive or offline deployments (e.g., on kiosks, field devices, or BYOD).
- Tools/workflows: Quantized 0.8B model for CPU/edge; caching of category outputs for common prompts; lightweight proxy integration.
- Dependencies/assumptions: Acceptable runtime on target hardware; periodic updates for new adversarial patterns.
- Sector: Research and Benchmarking (Academia)
- Use case: Study FP/FN frontier, boundary cases, multilingual safety, and counterfactual hard negatives; create new evaluation sets and teaching materials.
- Tools/workflows: Use open weights for reproducible experiments; re‑use taxonomic structure for dataset design; adopt continuous red‑team protocols.
- Dependencies/assumptions: Ethical review for sensitive content; maintain separation of training/eval splits to avoid contamination.
Long‑Term Applications
These opportunities require further research, scaling, customization, or ecosystem development (some are planned future releases in the paper).
- Sector: Software/AI Platforms
- Use case: Streaming output moderation paired with input guard (token‑level classification) for end‑to‑end safety.
- Tools/workflows: Dual‑surface guard (input + streaming output); unified confidence/risk scoring; rollback/stop‑generation controls.
- Dependencies/assumptions: Efficient low‑latency streaming classifiers; integration with generation servers; robustness to context length.
- Sector: Agentic Systems and Tool‑Use
- Use case: Safety‑aware tool permissioning and action gating informed by constitution categories and risk tiers.
- Tools/workflows: Policy‑aware planner/dispatcher that consults HaloGuard outputs before tool invocation; stateful risk budgets per session.
- Dependencies/assumptions: Coordination across agent frameworks; formalized “action policies”; provenance tracking for audit.
- Sector: Domain‑Specific Packs (Healthcare, Legal, Finance)
- Use case: Specialized constitutions and operating points (e.g., medical triage, legal privilege, investment advice) with sector‑specific thresholds and safe alternatives.
- Tools/workflows: Policy “packs” and templates; SME‑curated counterfactuals; validation against sectoral benchmarks.
- Dependencies/assumptions: Regulatory approval and expert input; ongoing maintenance of policy packs; liability frameworks.
- Sector: Global Governance and Standards
- Use case: Reference open standard for constitutional policies and auditability; cross‑org comparability of guard performance and logs.
- Tools/workflows: Standards bodies endorsing schema; shared evaluation suites; interoperable category mappings.
- Dependencies/assumptions: Stakeholder consensus; compatibility with regional laws; governance for updates.
- Sector: Personalization and Context‑Aware Safety
- Use case: User‑ and context‑specific safety thresholds (age, role, locale) without sacrificing fairness or privacy.
- Tools/workflows: Policy profiles and dynamic thresholding; consent‑aware personalization; bias monitoring and mitigation.
- Dependencies/assumptions: Privacy‑preserving profile handling; fairness audits; clear UX for user controls.
- Sector: Multimodal Moderation (Text+Image+Audio+Code)
- Use case: Extend constitutional classification to other modalities and cross‑modal consistency checks (e.g., text prompt + image).
- Tools/workflows: Shared policy ontology across modalities; fused classifiers; cross‑signal adjudication.
- Dependencies/assumptions: Multimodal datasets and training; efficient deployment on target hardware.
- Sector: Continuous Adversarial Resilience
- Use case: Auto‑hardening pipelines that ingest red‑team findings (content‑level and agentic) and update the guard safely.
- Tools/workflows: CI/CD for safety models; canary deployments; automated evaluation on contrast sets and new overlays.
- Dependencies/assumptions: Safe data/labeling pipeline; monitoring for regressions; roll‑back capabilities.
- Sector: Base‑Model Alignment and Co‑Training
- Use case: Use boundary‑focused synthetic data and paired counterfactuals to fine‑tune or preference‑train base models, reducing downstream refusal/over‑refusal.
- Tools/workflows: Data distillation to RLHF/DPO pipelines; joint training with classifier signals; eval on over‑refusal sets (e.g., XSTest).
- Dependencies/assumptions: Access to base model weights; careful avoidance of overfitting to synthetic artifacts; human evaluation.
- Sector: Compliance and Audit Automation
- Use case: Constitution‑attributed evidence generation for audits (who, what, why content was blocked/allowed); impact analysis by policy area.
- Tools/workflows: Automated compliance reports; traceable category/confidence logs; policy change diffing and replay.
- Dependencies/assumptions: Legal acceptance of automated evidence; secure log storage; data minimization.
- Sector: Marketplace of Policies and Multi‑Policy Switching
- Use case: Deploy multiple policy profiles (e.g., consumer vs. regulated; region‑specific rules) and switch at runtime.
- Tools/workflows: Policy registry and selector; per‑tenant configuration in multi‑tenant SaaS; policy A/B testing.
- Dependencies/assumptions: Clear mapping between policies; governance for conflicts and precedence rules.
- Sector: Home and Robotics/IoT Assistants
- Use case: On‑device guards for voice assistants and home robots that enforce safety while respecting multilingual households and offline constraints.
- Tools/workflows: Embedded, quantized models; wake‑word pipelines with pre‑gen gating; parent/guardian profiles.
- Dependencies/assumptions: Hardware acceleration; robust far‑field speech → text integration; update channels for security patches.
- Sector: Benchmarking and Data Quality Ecosystem
- Use case: Community‑curated boundary benchmarks and tools to diagnose mislabels and policy‑adjacent failures across languages.
- Tools/workflows: Open repositories of contrast sets; adjudication tools; shared leaderboards for FP/FN frontier.
- Dependencies/assumptions: Sustained community participation; standardized annotation guidelines; funding for maintenance.
Notes across applications:
- Immediate deployment favors the 0.8B model for latency/edge; the 4B variant can be used where precision is paramount and compute allows.
- Open‑weights enable on‑prem and privacy‑sensitive use; verify license terms for commercial deployment.
- Effectiveness depends on mapping the 46‑policy constitution to organizational policies, selecting operating thresholds, and maintaining a continuous red‑team/update cycle.
- Current release is input‑only; pairing with an output/streaming moderator is recommended for high‑risk deployments once available.
Glossary
- Adjudication: A structured process of reviewing and deciding on ambiguous cases or failures. "A structured adjudication of the remaining failures indicates that most apparent missed-harm cases are benchmark mislabels rather than genuine model misses."
- Adversarial overlays: Attack techniques that add obfuscating layers or patterns to prompts to evade safety detection. "adversarial overlays, deterministic transformations"
- Adversarial red-teaming protocol: A continual process of probing a system with adversarial inputs to expose weaknesses. "An always-on adversarial red-teaming protocol continuously hardens the guard against both content-level and agentic attacks."
- Adversarially augmented prompts: Inputs modified with adversarial tactics (e.g., roleplay, encodings) to conceal harmful intent. "Adversarially augmented prompts."
- Agentic: Refers to autonomous, tool-using or goal-directed behavior or attacks in AI systems. "agentic attacks"
- Anchor pools: Curated sets of real prompts used to ground synthetic generation in realistic style and distribution. "Anchor pools. We maintain stratified pools of real and benchmark-derived prompts"
- Anchored factor-crossing: Systematically combining semantic categories with stylistic/form axes to create comprehensive data coverage. "This is the constitutional basis for the anchored factor-crossing."
- Attack surfaces: The exposed areas where a system can be attacked or manipulated. "attack surfaces are exponentially bigger"
- Baseline FPs: False positives arising from ordinary benign inputs, distinct from boundary false positives near policy edges. "boundary FPs versus baseline FPs"
- Benign confusions: Legitimate uses that are frequently mistaken for harmful ones due to shared vocabulary or context. "benign_confusions"
- Boundary FPs: False positives that occur on legitimate prompts near safety-policy boundaries. "boundary FPs versus baseline FPs"
- CBRN: Abbreviation for Chemical, Biological, Radiological, and Nuclear threats. "Chemical, Biological, Radiological, and Nuclear(CBRN)"
- Composite safety category labels: Combined labels that include both a verdict (safe/unsafe) and a policy category. "a set 𝓛 of 𝐾 composite safety category labels"
- Component-disjoint splits: Data partitioning that avoids overlap of structured components between train, validation, and test sets. "compiles a release with component-disjoint splits."
- Constitutional Classifiers (CCs): Classifiers trained using a written set of natural-language safety rules to synthesize robust training data. "Constitutional Classifiers (CCs)"
- Constitution-attributed scores: Outputs that include the predicted policy category and confidence alongside the binary safety verdict. "HaloGuard 1.0 therefore returns constitution-attributed scores"
- Contrast sets: Minimal edits to inputs that flip labels, used to probe and sharpen decision boundaries. "contrast sets"
- Counterfactually-augmented data: Paired examples where intent is flipped while topic and wording are held constant to teach fine-grained distinctions. "counterfactually-augmented data"
- Coverage axes: Orthogonal style/interaction dimensions used to systematically vary prompts (e.g., interaction shape, depth, form). "Coverage axes as generation grid"
- CSAM: Child Sexual Abuse Material; a critical, zero-tolerance content category. "CSAM; non-consensual sexual content; sexual content (general)"
- Decoder backbone: A generative LLM architecture used as the basis for classification by emitting labels. "built on a decoder backbone"
- Defence-in-depth: A layered security strategy employing multiple safeguards to reduce risk. "A practical defence-in-depth solution requires multiple layers"
- Deterministic transformations: Systematic, rule-based alterations of prompts (e.g., format changes) used to test robustness. "deterministic transformations"
- Direction-of-intent conditioning: Prompt-generation control that enforces defender-side framing (e.g., prevention, support) versus perpetrator content. "Direction-of-intent conditioning."
- False-positive/false-negative (FP/FN) frontier: The trade-off curve balancing missed harmful prompts (FNs) against over-blocking benign prompts (FPs). "the boundary cases or the false-positive/false-negative (FP/FN) frontier."
- FNR (False Negative Rate): The proportion of harmful inputs incorrectly labeled as safe. "FN rate (FNR)"
- FPR (False Positive Rate): The proportion of safe inputs incorrectly labeled as unsafe. "FP rate (FPR)"
- Generative classifier: A classifier that generates labels with a LLM rather than using a fixed classification head. "HaloGuard 1.0 is a generative classifier"
- Generative specification: Treating the written safety policy as the source that specifies what data to generate and how to label it. "treats the constitution as a generative specification"
- Guard models: External classifiers that evaluate prompts or responses against safety policies before or during generation. "Guard models have become a common safety layer for LLM deployments."
- Harmless-boundary: Benign examples that are intentionally close to harmful categories in topic and wording but differ in intent. "harmless-boundary"
- Hard negatives: Challenging benign examples that share surface features with harmful prompts, used to reduce over-refusal. "benign hard negatives"
- Intent sensitivity: The capability to distinguish harmful from benign requests based on user intent, not just keywords. "The key requirement across these inputs is intent sensitivity."
- Jailbreaks: Prompting strategies designed to circumvent safety constraints and elicit unsafe model behavior. "improve robustness against jailbreaks"
- Judge gate: A quality-control step using a judge model or process to filter or validate generated data. "Label noise is bounded by binary labels plus a judge gate."
- Length-band targeting: Generating data across controlled length ranges to match real-world prompt distributions. "Length-band targeting."
- Multilingual materialisation: Expanding the dataset across languages in a balanced way so language isn’t a spurious signal. "balanced multilingual materialisation across 46 languages"
- Multi-label: A task setup where multiple labels may apply to a single input. "the model is multi-label at the task level"
- Multi-policy moderation: Moderation that adapts to different safety policies or enforcement thresholds across deployments. "multi-policy moderation"
- Open weights: Model parameters released to the public for inspection and use. "open weights implementation"
- Over-refusal: Incorrectly blocking benign inputs, especially those near policy boundaries. "over-refusal behavior"
- Paired counterfactuals: Matched harmful/benign prompt pairs that hold topic and vocabulary fixed while flipping intent. "exhaustive 1:1 paired counterfactuals"
- Payload splitting: An evasion tactic that spreads harmful intent across multiple parts or messages. "payload splitting"
- Policy boundary: The line between permitted and disallowed content under a written safety constitution. "policy boundary"
- Risk taxonomy: An organized set of risk categories used to label and structure safety moderation tasks. "risk taxonomy"
- Risk tiers: Levels indicating how strict or tolerant a policy area is (e.g., critical, high, moderate, sensitive). "The Master Constitution carries one of four risk tiers"
- Roleplay-jailbreak: A jailbreak style that uses roleplay framing to mask harmful intent. "roleplay-jailbreak anchors"
- Safety constitution: A detailed, natural-language policy that defines restricted and permitted content and drives data generation. "It makes the safety constitution the organising structure of the corpus"
- Severity-aware: Labeling or modeling that accounts for different severity levels of unsafe content. "severity-aware safe, controversial, and unsafe labels"
- Shared-harmless: Broadly benign content categories used to anchor the safe distribution apart from boundary cases. "shared-harmless"
- Spurious correlations: Unintended correlations (e.g., with language or script) that can mislead a classifier. "introduce spurious correlations"
- Streaming output moderation: Real-time monitoring and moderation of generated tokens during model output. "response classification or streaming output moderation"
- Style-anchored generation: Synthesizing prompts by conditioning on real prompt styles to reduce distribution shift. "Style-Anchored Generation"
- Surface cues: Tokens or phrases that appear in both harmful and benign contexts, potentially confusing a classifier. "surface_cues"
- Surface form: Treating language (or script) as presentation rather than as an adversarial signal. "treats language as a surface form"
- Token-level response monitoring: Monitoring generated output token by token to detect emerging unsafe content. "token-level response monitoring"
- Two-tier harmless design: Separating benign data into boundary-closer and broadly safe tiers to address different false-positive modes. "a two-tier harmless design"
- Universal jailbreak overlays: General-purpose jailbreak wrappers or transformations applied to prompts across languages or formats. "universal jailbreaks overlays"
Collections
Sign up for free to add this paper to one or more collections.