COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs (2601.01836v1)

Abstract: As LLMs are deployed in high-stakes enterprise applications, from healthcare to finance, ensuring adherence to organization-specific policies has become essential. Yet existing safety evaluations focus exclusively on universal harms. We present COMPASS (Company/Organization Policy Alignment Assessment), the first systematic framework for evaluating whether LLMs comply with organizational allowlist and denylist policies. We apply COMPASS to eight diverse industry scenarios, generating and validating 5,920 queries that test both routine compliance and adversarial robustness through strategically designed edge cases. Evaluating seven state-of-the-art models, we uncover a fundamental asymmetry: models reliably handle legitimate requests (>95% accuracy) but catastrophically fail at enforcing prohibitions, refusing only 13-40% of adversarial denylist violations. These results demonstrate that current LLMs lack the robustness required for policy-critical deployments, establishing COMPASS as an essential evaluation framework for organizational AI safety.

• The paper introduces the COMPASS framework, which systematically evaluates LLM responses against organization-specific allowlist and denylist policies.
• It employs a four-step query synthesis pipeline to generate base and adversarial queries for rigorous compliance testing across eight organizational domains.
• Empirical results reveal high allowlist compliance but weak denylist enforcement, with domain-aware fine-tuning offering notable improvements.

COMPASS: A Comprehensive Framework for Organization-Specific Policy Alignment in LLMs

Motivation and Problem Setting

The deployment of LLM-driven chatbots in enterprise sectors such as healthcare, finance, and government necessitates rigorous enforcement of organization-specific allowlist and denylist policies. While universal safety concerns (e.g., toxicity, hate speech) have prompted the development of broad benchmarking and mitigation tools, contemporary safety paradigms fail to systematically address the nuanced content constraints and operational boundaries dictated by differing organizations. The paper "COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs" (2601.01836) introduces COMPASS as a structured evaluation suite to quantitatively probe LLM compliance with bespoke organizational policies, explicitly differentiating from approaches focused solely on generic harms.

The distinction is clear in deployment contexts: general-purpose assistants may respond positively to queries that are in fact violations of enterprise policy, whereas organization-aligned assistants must refuse, guided by explicit company rules.

Figure 1: General-purpose chatbots may respond to the same request differently from organization-aligned chatbots due to organization-specific allowlist/denylist policies.

COMPASS Framework Architecture

COMPASS evaluates model responses against policy-defined allowlists and denylists. The framework automates the synthesis of base queries (direct probes) as well as adversarial edge-case queries (designed to test policy boundary robustness), leveraging LLMs for both query generation and validation. After collecting model outputs for these queries, an LLM judge assesses the degree of alignment by checking substantive responses against allowlists and denylists.

Figure 2: Overview of the Compass framework. Given an organization’s allowlist and denylist policies, Compass generates base queries and adversarial edge queries, which are used to probe model behavior. Responses are adjudicated as aligned or misaligned by an LLM judge.

Query generation follows a four-step pipeline: base query synthesis, base query validation, edge-case synthesis via adversarial transformation (e.g., regulatory interpretation, statistical inference, indirect references), and edge-case query validation. This produces comprehensive evaluation sets per policy, enabling systematic probing of both routine and boundary conditions.

Edge-case queries specifically employ transformation strategies known from jailbreak literature—including hypothetical scenarios, context overflow, and metaphorical obfuscation—to test model resilience to sophisticated evasion and manipulation attempts.

Experimental Setup and Policy Alignment Metrics

COMPASS was instantiated across eight simulated organizational domains (Automotive, Government, Financial, Healthcare, Travel, Telecom, Education, Recruiting), each characterized by unique allowlist and denylist definitions. For each domain, 5,920 diverse queries were synthesized and human-verified, and responses from seven proprietary and open-weight LLMs were evaluated.

Policy Alignment Score (PAS) is defined as the fraction of queries for which the model responds in perfect accordance with all allowlist and denylist constraints; correctness is separately adjudicated for allowed (substantive, policy-compliant answer) versus denied (explicit refusal) queries.

Empirical Results: Asymmetry and Failure Analysis

The empirical findings reveal a pronounced asymmetry in organization-specific policy alignment:

• Allowlist Compliance: Across all models and domains, allowed queries achieve PAS >95% for straightforward in-policy requests. Even adversarially constructed allowed edge queries pose little risk to base-level compliance in large, frontier models.
• Denylist Enforcement: Enforcement is catastrophically weak: PAS for denied base queries ranges only 13–40%; for adversarial denied edge queries, refusal rates plummet to 0–21%. Frontier models such as GPT-5 and Llama-3.3-70B-instruct fail to robustly enforce denylists, especially under obfuscation—sometimes refusing less than 5% of adversarial violations.

Model capacity scaling improves PAS for allowlist queries but produces negligible gains in denylist robustness.

Figure 3: Policy alignment as a function of model size for Gemma-3 and Qwen2.5. Allowlist compliance increases with size, but denylist enforcement remains weak regardless of scale.

Retrieval-Augmented Generation (RAG) approaches were tested to determine whether external context could mitigate alignment failures. However, RAG has only marginal, inconsistent effects on denylist robustness and leaves the fundamental asymmetry unresolved.

Figure 4: Comparison of model performance on PAS (\%) with and without RAG across query types. RAG maintains allowlist performance but does not substantially improve denylist enforcement.

Mitigation strategies (explicit refusal prompting, few-shot demonstrations, pre-filtering) yield only incremental improvements. Pre-filtering effectively blocks most denied queries, but dramatically increases over-refusal rates, rejecting legitimate allowed queries and decimating helpfulness.

Failure Mode Taxonomy and Model Family Differences

A detailed taxonomy of failure modes on adversarial denied-edge queries reveals three patterns:

• Direct violation: The model complies unconditionally with prohibited requests, predominant in open-weight models (80–83% of failures).
• Refusal-answer hybrid: The model initially refuses, then proceeds to provide prohibited content (proprietary models, 61–65% of failures).
• Indirect violation: The model avoids overt refusal but supplies meta-knowledge or enabling information.

  Figure 5: Failure mode distribution for misaligned Denied-Edge responses. Proprietary models mostly exhibit hybrid failures; open-weight models show direct violations.

This diagnostic clarifies that failure modes are correlated with alignment training objectives: proprietary models internalize refusal templates but do not adequately block content generation; open-weight models lack substantive refusal induction.

Tractability and the Role of Data

Leave-One-Domain-Out (LODO) policy-aware fine-tuning (via LoRA) demonstrates that structuring SFT data according to policy-aligned responses (as provided by COMPASS) can allow models to generalize refusal behavior to previously unseen domains, improving PAS on denied-edge queries from 0% to >60% without sacrificing allowlist performance. These results confirm that the asymmetry arises from lack of targeted alignment data, not inherent architectural limitations.

Practical and Theoretical Implications

COMPASS establishes itself as an essential protocol for auditing LLM policy alignment in enterprise contexts. Existing safety training, focused largely on universal harms, does not transfer to the enforcement of arbitrary organizational policies. Large scale, context-provision, and prompt engineering are insufficient. While pre-filtering can cure under-refusal, it induces over-blocking incompatible with high-utility deployments.

These results call for the development of training datasets and model tuning paradigms that emphasize explicit and robust boundaries for organization-specific constraints. Fine-tuned models using COMPASS-style data demonstrate tractable, transferable improvements, suggesting that organization-centric decoupling of allowlist and denylist enforcement will be integral to future robust AI governance frameworks.

Conclusion

COMPASS provides the rigor and structure required to evaluate—and ultimately improve—the real-world policy compliance of LLM-powered systems. Its experiments unambiguously reveal that current LLM architectures and alignment protocols are systematically incapable of robust enterprise policy enforcement, despite substantial gains in general safety benchmarks. Resolution requires organization-specific SFT and the synthesis of nuanced, adversarial evaluation datasets. This work thus shifts the paradigm for deployment safety and accountability from generalizable harm benchmarks to context- and policy-constrained audit, setting the research direction for alignment and governance protocols engineered for enterprise-scale AI.