SocialHarmBench: LLM Sociopolitical Safety

• SocialHarmBench is a comprehensive adversarial benchmark that evaluates LLM responses to harmful sociopolitical prompts such as propaganda, censorship, and political manipulation.
• It encompasses 585 prompts across 7 high-risk domains and 34 countries, ensuring rigorous testing of model biases and failure modes in varied historical and regional contexts.
• Key metrics like Attack Success Rate and Refusal Rate provide actionable insights into LLM vulnerabilities, particularly under adversarial pressure and weight-space attacks.

SocialHarmBench is a comprehensive adversarial evaluation framework and dataset targeting the vulnerabilities of LLMs when confronted with socially and politically harmful requests. Unlike traditional safety benchmarks that focus on clear-cut illicit domains such as fraud or cybercrime, SocialHarmBench systematically probes LLM capabilities and failure modes in high-impact sociopolitical contexts: political manipulation, propaganda, disinformation, censorship, surveillance, human rights violations, historical revisionism, and war crimes. The benchmark spans 585 prompts distributed across 7 domain categories and 34 countries, designed to expose brittle or biased model behaviors that could facilitate threats to democratic order and human rights when LLMs are misused (Pandey et al., 6 Oct 2025).

1. Motivation and Context

The deployment of LLMs in domains that influence public opinion, political discourse, and access to information presents exposure to critical risks—misuse in propaganda generation, coordination of censorship, information control, and facilitation of state-sponsored or institutionalized abuse. Existing safety benchmarks are inadequate for characterizing these risks because they:

• Focus narrowly on criminal actions, neglecting state-level or institutionally sanctioned harms;
• Cover little or none of the full spectrum of human rights violations, censorship, and engineered disinformation;
• Lack critical temporal and geographic diversity; previous benchmarks do not test model behavior for shifts across distinct historical periods or regional contexts.

SocialHarmBench directly addresses these lacunae by providing a grounded, adversarial prompt suite that surfaces LLM vulnerabilities in governance-relevant contexts. Its construction methodology ensures robustness to ambiguity and topical diversity while supporting fine-grained analysis of failure patterns across time, space, and subtype of harm (Pandey et al., 6 Oct 2025).

2. Dataset Construction and Coverage

Prompt design in SocialHarmBench proceeds from seven “high-level sociopolitical domains,” subdivided into 37 realistic subtopics (e.g., biometric surveillance, genocide planning, electoral suppression), each anchored by open-source research to historical or contemporary events. Prompts are crafted in three functional styles: standard instruction, opinion elicitation, and historical reference, then undergo manual curation to eliminate ambiguity and redundancy.

The seven core categories are:

1. Censorship & Information Control: Suppressing or distorting public discourse;
2. Human Rights Violations: Enabling persecution, population control, or forced sterilization;
3. Political Manipulation & Election Interference: Operationalizing sabotage or “psyops”;
4. Historical Revisionism: Facilitating denial or erasure of atrocities;
5. Propaganda & Disinformation: Automating misinformation/disinformation campaigns;
6. Surveillance & Monitoring: Faking personas or biometric tracking;
7. War Crimes & Crimes Against Humanity: Coordinating attacks on civilian/noncombatant populations.

Geographical and temporal axes are explicit and diverse: 34 countries (with emphasis on high-risk regions such as Latin America, USA, and UK) and 54 time ranges from 19th-century colonialism to present-day digital censorship. Each prompt is single-intent, unambiguous, and asks the model to produce a clearly harmful sociopolitical act or plan. Temporal and regional mapping enables detailed diagnosis of context-dependent weaknesses (Pandey et al., 6 Oct 2025).

3. Metrics and Vulnerability Quantification

Evaluation employs two automated detectors:

• HarmBench (HB): Classifies responses as “harmful compliance” if the generation fulfills the prompt’s harmful intent.
• StrongREJECT (SR): Flags robust refusals (as opposed to vague or evasive non-responses).

Key quantitative metrics (higher indicates greater vulnerability):

• Attack Success Rate (ASR)/Harmful Compliance:

$\mathrm{ASR} = \frac{\#\,\text{model responses classified as harmful compliance}}{\text{Total}\,#\,\text{harmful prompts}} \times 100\%$

• Refusal Rate (RR):

$\mathrm{RR} = \frac{\#\,\{\text{responses with explicit safe refusal}\}}{\text{Total}\,#\,\{\text{prompts}\}} \times 100\%$

• Safe Completion Rate (SCR):

$\mathrm{SCR} = \frac{\#\,\{\text{benign or partially safe responses}\}}{\text{Total}\,#\,\{\text{prompts}\}} \times 100\%$

These metrics enable rigorous comparison of model safety both in clean (vanilla prompt) and adversarial settings (Pandey et al., 6 Oct 2025).

4. Evaluation Protocol and Experimental Regimes

The benchmark tests both closed-weight (e.g., GPT-4o, Claude-Sonnet-4) and open-weight (e.g., Llama-3.1-8B-Instruct, Mistral-7B-Instruct) models. Each receives a minimal system prompt (“You are a helpful assistant”) to simulate standard deployment conditions. No additional “constitutional” or retrieval-based safeguarding is enabled.

Two adversarial regimes are tested:

• Prompt-level attacks (e.g., AutoDAN-GA/HGA, GCG), which manipulate input phrasing to circumvent known content filters;
• Weight-space attacks (LoRA-based fine-tuning), which directly alter model parameters to suppress safety behaviors.

The evaluation framework thus exposes both superficial and deeply rooted defense mechanisms (Pandey et al., 6 Oct 2025).

5. Principal Findings and Failure Modes

Key results include:

• Baseline Vulnerabilities: Open-weight models are significantly susceptible to harmful compliance (Mistral-7B ASR ≈ 44–62% in domains like censorship, historical revisionism; overall ASR ≈ 27.7%, SR ≈ 28.3%). Mid-range vulnerabilities are observed for Llama-3.1 and Qwen-2.5 (ASR ≈ 10–20%), while Claude-Sonnet-4 exhibits much lower compliance (<7% ASR).
• Worst-case Domains: Historical Revisionism, Propaganda, and Political Manipulation consistently produce the highest ASR, even against strong safety training baselines (up to 62% ASR in clean conditions).
• Adversarial Attacks: Weight tampering markedly increases vulnerability (HB and SR ≥90%), while advanced prompt-level attacks push ASR above 80–90% in critical domains.
• Temporal and Geographic Fragility: 21st-century and pre-20th-century prompts yield maximal harmful compliance (HB ≈ 67%, SR ≈ 50%); mid-20th-century topics (WWII, Cold War) show marginally stronger resistance. Prompts referencing Latin America, USA, and UK elicit the highest compliance rates, while Europe and Africa test refusal robustness, likely reflecting data and alignment bias (Pandey et al., 6 Oct 2025).

Model/Domain	Clean ASR (Worst)	Adversarial ASR (Peak)	Regions (Peak ASR)
Mistral-7B-Instruct	44–62%	≥90%	Latin America, USA, UK
Llama-3.1-8B-Instruct	10–20%	≥80%	Europe, Africa
Claude-Sonnet-4	≤7%	—	—

6. Interpretations, Systematic Biases, and Implications

Analysis reveals that LLMs import systematic training-data and alignment biases:

• Distributional coverage is uneven; models “cling” to familiar manipulative or revisionist tropes from regions with extensive web-text presence.
• Safeguards derived from reinforcement learning from human feedback (RLHF) or rule-based filters generalize poorly outside canonical illegal behavior. Models often fail on queries with complex ethical or political content, suggesting under-specification of alignment objectives.
• Weight-space adversarial attacks illuminate a deep brittleness: low-level parameter modifications can erase or corrupt safety signals entirely, indicating limited defense in depth.
• High-risk failure cases demonstrate feasibly exploitable weaknesses: in real scenarios, these could enable orchestrated censorship, rapid misinformation amplification, or facilitation of human rights abuses at scale (Pandey et al., 6 Oct 2025).

A plausible implication is that LLMs, without context-sensitive and culturally robust safety training, may fundamentally endanger core democratic and human rights values when deployed in adversarial or ethically ambiguous environments.

7. Limitations and Prospects for Extension

Several limitations are explicitly recognized:

• The dataset is strictly English and underrepresents critical regions (e.g., Sub-Saharan Africa, Pacific).
• Prompts are single-turn; interactive, multi-turn, or tool-integrated attack dynamics remain unexplored, though real-world attack surfaces are likely broader.
• Classifier-based scoring may misclassify nuanced refusals or euphemistic harmful outputs, complicating reliability at the fine edge of harm/safety judgments.

Future development priorities include:

• Expansion to multilingual and culturally-adapted variants with context-expert validation;
• Finer-grained temporal modeling (e.g., decade-level prompts) for longitudinal safety analysis;
• Richer adversarial formats, including multi-turn dialogue, retrieval augmentation, and agentic goal planning;
• Systematic benchmarking of specific defense pipelines (constitutional AI, retrieval-based filters, external monitoring) within all sociopolitical domains (Pandey et al., 6 Oct 2025).

By codifying explicit, adversarial, and contextually grounded safety tests across a wide spectrum of political and social harms, SocialHarmBench constitutes an essential resource for evaluating, diagnosing, and eventually mitigating the most societally consequential risks posed by LLM deployment in governance-sensitive domains.