PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models

Abstract: LLMs are increasingly integrated into real-world decision-making, including in the domain of public policy. Yet, their ability to comprehend and reason about policy-related content remains underexplored. To fill this gap, we present \textbf{\textit{PolicyBench}}, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension, comprising 21K cases across a broad spectrum of policy areas, capturing the diversity and complexity of real-world governance. Following Bloom's taxonomy, the benchmark assesses three core capabilities: (1) \textbf{Memorization}: factual recall of policy knowledge, (2) \textbf{Understanding}: conceptual and contextual reasoning, and (3) \textbf{Application}: problem-solving in real-life policy scenarios. Building on this benchmark, we further propose \textbf{\textit{PolicyMoE}}, a domain-specialized Mixture-of-Experts (MoE) model with expert modules aligned to each cognitive level. The proposed models demonstrate stronger performance on application-oriented policy tasks than on memorization or conceptual understanding, and yields the highest accuracy on structured reasoning tasks. Our results reveal key limitations of current LLMs in policy understanding and suggest paths toward more reliable, policy-focused models.

• The paper introduces PolicyLLM, featuring a novel benchmark (PolicyBench) and a Mixture-of-Experts (PolicyMoE) model to enhance LLMs' policy comprehension.
• The evaluation across US and Chinese policy tasks reveals significant performance gaps, particularly in deep conceptual analysis and cross-lingual understanding.
• The PolicyMoE model shows modest improvements through modular expert routing, highlighting the importance of policy-specific adaptation in LLM frameworks.

PolicyLLM: Towards Excellent Comprehension of Public Policy for LLMs

Introduction

This paper introduces PolicyLLM, an extensive evaluation and modeling suite targeting the comprehension of public policy by LLMs (2604.12995). The study identifies critical challenges in assessing and enhancing LLMs’ ability to process, reason about, and apply complex policy texts—tasks that require capabilities far beyond surface-level language understanding. Addressing these gaps, the authors present PolicyBench, a cross-system benchmark (spanning US–China policy contexts) and the PolicyMoE architecture, which leverages Mixture-of-Experts (MoE) specialization to boost model performance across factual, conceptual, and applied policy tasks.

PolicyBench Design and Characteristics

PolicyBench is a large-scale, systematically curated benchmark comprising 21,000 cases covering memorization, understanding, and application, inspired by Bloom’s Taxonomy. It targets two governance contexts: China and the United States.

Figure 1: Three cognitive levels evaluated in PolicyBench: Memorization, Understanding, and Application.

The benchmark is stratified into three levels:

• Level 1 (Memorization): Tests factual recall such as dates, definitions, and institutional details.
• Level 2 (Understanding): Probes conceptual and contextual comprehension, focusing on “Ideas, Interests, Institutions.”
• Level 3 (Application): Evaluates ability to solve policy-relevant scenario problems, execute numerical/statistical reasoning, and justify procedural outcomes.

  Figure 2: Example PolicyBench items at all levels, in both English and Chinese.

Question generation for multiple-choice tasks is intentionally diverse, leveraging distractors synthesized by a heterogeneous model pool to reduce bias and overfitting to any single LLM pattern. Furthermore, expert domain annotation ensures the validity and complexity of understanding and application-level queries.

Empirical Findings: LLMs on Policy Comprehension

A comprehensive empirical assessment is performed across leading foundation models (OpenAI GPT-4o, Gemini-2.5/-2.0, Claude-3.7/-3.5, LLaMA-4, Qwen-QwQ-32B, DeepSeek-V3, DeepSeek-R1, etc). The main results can be summarized as:

• Performance Progression: Accuracy consistently increases from Level 1 to Level 3; for example, open-weight and API models show a 15–20% absolute improvement from factual to application tasks in both language regimes. This contradicts the expectation that application tasks would be the most difficult, and is attributed to instruction-tuning and RLHF phases emphasizing structured reasoning rather than mere factual extraction.

  Figure 3: Fine-grained model performance per sub-task, evidencing differential difficulty and proficiency.

• Structured Reasoning vs. Abstract Comprehension: Models achieve up to 80% accuracy in scenario-based calculations and decision-making, yet exhibit marked underperformance (<60%) on tasks requiring deep conceptual or institutional analysis, especially with Chinese language content. This signals a deficiency in the internalization of policy logic, particularly for abstract relationships and stakeholder mappings.
• Cross-lingual Disparity: All evaluated LLMs, except QwQ-32B, achieve higher scores on US policy than Chinese policy. This is likely a result of pretraining corpus imbalances and higher semantic density in Chinese texts.
• LLM vs Human Baseline: Human subjects (policy non-experts) achieve comparable or better accuracy than state-of-the-art LLMs only at Level-1 (factual recall), with a significant gap at Level-2 and Level-3. Policy experts outperform LLMs by more than 20 points at higher cognitive levels, indicating that current LLMs lack nuanced comprehension, especially for intent and institutional logic.

  Figure 4: Human participant accuracy by level, distinguishing open-book and closed-book conditions.

PolicyMoE: Domain-Specialized Mixture-of-Experts Architecture

The PolicyMoE model decomposes the policy comprehension workload into three modular experts—Memory, Understanding, Application—each parameterized via LoRA adapters atop a base LLM. A lightweight linear router dynamically selects or ensembles experts in response to incoming queries.

Figure 5: Router allocation distribution: clear specialization for factual tasks, with distributed attention on deeper reasoning.

Performance evaluation demonstrates that PolicyMoE fine-tuning yields notable gains over vanilla LoRA adaptation, especially on Level 1 and Level 3. US–Level 1 tasks improve from 23.4% to 35.4%, and US–Level 3 from 46.7% to 57.5%. Average improvement over standard LoRA is +2.68%, with particularly strong gains in memorization and application, indicating successful expert decoupling and reduced gradient interference.

Error Analysis and Observed Pathologies

Qualitative analysis reveals systematic model failures:

• Level 1: Semantically plausible distractors induce factual confusion.
• Level 2: Ideological and stakeholder mapping errors (surface cue overreliance).
• Level 3: Hallucinated policy justifications, procedural missteps, and failures in numerical reasoning.

  Figure 6: Representative error archetypes at each cognitive depth in PolicyBench.

These error patterns are robust to LLM size and architecture and persist even after fine-tuning, highlighting inherent limitations in current model architectures and training data.

Implications and Future Directions

The findings imply that, despite advances in general NLU, LLMs’ capacity for high-fidelity policy understanding is bottlenecked by their lack of policy-specific supervision, insufficient abstraction alignment, and cross-lingual limitations. PolicyMoE’s modularity effectively addresses task interference but achieves only modest improvement in deep interpretive tasks. The negative or negligible correlation between PolicyBench scores and those on MMLU-Pro/LegalBench further emphasizes that policy comprehension is orthogonal to general or legal QA benchmarks.

Potential future directions include:

• More granular expert activation (beyond top-1 routing) and cross-module ensembling for multi-faceted queries.
• Retrieval-augmentation to support context anchoring and hallucination mitigation.
• Task formulation expansion to incorporate open-ended, value-sensitive, and multi-turn dialogic reasoning for policy deliberation.

Conclusion

PolicyLLM establishes a rigorous diagnostic and model-building framework for evaluating and enhancing the public policy comprehension of LLMs. PolicyBench delivers a robust, cross-lingual benchmark that exposes core limitations in existing architectures, which are only partially resolved by MoE adaptation. The work highlights a persistent gap between human and LLM proficiency in nuanced policy understanding and sets a formal agenda for the development of specialized, policy-aware LLMs.