PsychEthicsBench: Ethical AI Benchmark

• PsychEthicsBench is a principle-grounded framework designed to assess ethical reasoning and jurisdictional relevance in AI-powered mental health applications.
• It employs diverse instruments, including MCQs, open-ended tests, and psychometric scales like SPERET, to systematically measure alignment with established ethical standards.
• The framework rigorously maps ethical principles to model behaviors using empirical metrics and expert reviews, identifying biases and guiding targeted improvements.

PsychEthicsBench is a principle-grounded benchmarking framework designed to evaluate the ethical alignment, reasoning, and jurisdictional relevance of LLMs and other intelligent systems in psychological and mental health domains. Building upon foundational insights from benchmark ethics, specialized domain standards, and empirical investigations into AI value alignments, PsychEthicsBench provides both quantitative and qualitative instruments for auditing, comparing, and improving ethical behaviors in high-stakes settings such as clinical support, research, and automated decision-making.

1. Theoretical Underpinnings and Ethical Constructs

PsychEthicsBench emerges from recognition that benchmarks in psychology, mental health, and artificial intelligence are inherently value-laden, reflecting social, cultural, and institutional priorities at every stage of their design and application (Blili-Hamelin et al., 2022). Drawing on the concept of “thick concepts” from feminist philosophy of science, benchmarks such as PsychEthicsBench do not merely measure factual capability—they encode and propagate ethical stances regarding autonomy, justice, data privacy, and well-being. Key sources include national ethics codes (e.g., Australian Psychological Society, Royal Australian and New Zealand College of Psychiatrists) and frameworks for Responsible Research and Innovation (RRI), which inform how ethical reflexivity and empirical evaluation are operationalized (Shen et al., 7 Jan 2026).

The benchmark addresses multiple dimensions of psychological ethics, covering both empirical behaviors (e.g., model responses, human practices) and normative imperatives (e.g., respect for autonomy, avoidance of harm, fairness across diverse populations).

2. Benchmark Design: Scope, Task Formats, and Data Curation

PsychEthicsBench encompasses a diverse set of evaluative instruments and scenario types, each constructed with explicit mapping to core ethical principles and real-world regulatory constraints (Shen et al., 7 Jan 2026). The overarching design involves:

• Task Modalities: Multiple-choice questions (MCQs; single- and multiple-answer), open-ended questions (OEQs) capturing free-form ethical reasoning, and psychometric scales for measuring reflexivity (e.g., SPERET) (Hindennach et al., 24 Nov 2025).
• Jurisdictional Framing: Scenarios and evaluation criteria are parameterized by local legal and professional guidelines, supporting “Aussie” (Australian context) and “Global” (unspecified or cross-national) test modes (Shen et al., 7 Jan 2026).
• Curation Pipeline:

1. Extraction of ethical principles from professional codes and guidelines (e.g., APS Code of Ethics, RANZCP Code of Ethics/Conduct).
2. Expert-in-the-loop scenario generation, including both clinical psychologists and psychiatrists to ensure content validity and principle alignment.
3. Quality gates utilizing LLM-based judges and expert rubrics to enforce threshold plausibility and relevance (e.g., minimum scores on expert rubrics for inclusion).

The resulting corpus includes thousands of MCQs and OEQs, each coded to specific principles, with tasks situated from the perspective of patients, practitioners, and third parties (Shen et al., 7 Jan 2026). Complementary datasets such as EthicsMH (Kasu, 15 Sep 2025) and PapersPlease (Myung et al., 27 Jun 2025) provide rich scenario templates, encompassing confidentiality, bias, autonomy, beneficence, and motivational needs.

3. Instrument Types: Reflexivity Scales and Scenario-Based Tests

A. Reflexivity Scales

• SPERET Scale: The “Scale to measure Privacy and Ethics Reflexivity within Eye Tracking” is a prime example of how psychometric instruments are incorporated into PsychEthicsBench for researcher self-assessment (Hindennach et al., 24 Nov 2025).
  • Constructs:
  • Data Privacy Reflexivity
  • Sampling-Bias Reflexivity (“WEIRD Participant Burden”)
  • Misuse-Fears Reflexivity
  • Structure: 23 items, rated on a 7-point Likert scale, partitioned into three subscales.
  • Psychometrics: Internal consistency (Cronbach’s α) for the final 23 items was 0.772 (N=20). Factor structure is defined a priori, with formulas for composite reliability and AVE provided for future deployments.

B. Scenario-Based Decision Tasks

• Multiple-Choice and Open-Ended Instruments: Scenarios are mapped to explicit ethical dilemmas (e.g., involuntary treatment, disclosure to third parties, bias in diagnostic AI), with answer options and expected reasoning keyed to authoritative guidelines (Shen et al., 7 Jan 2026, Kasu, 15 Sep 2025).
• ERG-Theory Moral Dilemmas: PapersPlease provides 3,700 narratives structured around Existence, Relatedness, and Growth needs, supplemented by social identity cues. Models are evaluated not only for overall decision rates but also for systematic patterns of bias (e.g., acceptance rates by identity, effect sizes via χ² and Cramér’s V) (Myung et al., 27 Jun 2025).

4. Evaluation Metrics and Annotation Schema

PsychEthicsBench employs multi-dimensional, formally specified metrics for evaluating model and human behavior:

• MCQ Scoring:
  • Exact Match (EM): $\mathrm{EM}(\hat{y},y)$ equals 1 if all options match; otherwise 0.
  • Partial Credit (PC, MMCQs): $0.5$ credit for non-empty proper subsets; 1 for exact match (Shen et al., 7 Jan 2026).
• Open-Ended Response Metrics:
  • Refusal Rates: Greedy Refusal Rate (keyword-based) and Judge-based Refusal Rate (LLM-annotated) (Shen et al., 7 Jan 2026).
  • Quality Pass Rate (QPR) and Ethicality Rates: $QPR=\frac{1}{N}\sum_i q(r_i)$, $OER=\frac{1}{N}\sum_i E(r_i)$, $CER|Q=\sum_i E(r_i)/\sum_i q(r_i)$, where $q(r)$ and $E(r)$ indicate passing quality and ethicality gates, respectively.
  • Rule-Break Tracking: Fine-grained annotation schema capturing credential misrepresentation, confidentiality breach, dual relationship, empathy failures, safety planning lapses.
• Fairness & Value Alignment (PapersPlease, EthicsMH):
  • χ² statistics for dependence on motivational class or identity, Cramér’s V for effect sizes, and Δ acceptance rates for social identity effects (Myung et al., 27 Jun 2025).
  • Explanation quality (coherence and completeness) and professional norm alignment, using human or LLM annotators (Kasu, 15 Sep 2025).
• Psychometric Reliability:
  • Cronbach’s α and formulas for composite reliability/AVE for scales such as SPERET, emphasizing internal and construct validity (Hindennach et al., 24 Nov 2025).

5. Key Findings from Benchmark Applications

Empirical deployments of PsychEthicsBench and related frameworks yield several notable results:

• Refusal is Insufficient: High refusal rates do not guarantee ethical appropriateness—LLMs may refuse to answer but still produce content misaligned with mental health best practices (Shen et al., 7 Jan 2026). Clinically inadequate refusals can be counter-therapeutic.
• Fine-Tuning Risks: Domain-specific fine-tuning on counseling data can degrade ethical alignment; several specialized LLMs underperform their foundational backbones in principle-based ethical tasks (Shen et al., 7 Jan 2026).
• Jurisdictional Gaps: Most evaluated models default to US-centric advice (e.g., referencing American hotlines) even when prompted with Australian-specific scenarios, revealing persistent pretraining biases (Shen et al., 7 Jan 2026).
• Identity and Value-Sensitivity: Distinct clusters of models show systematic differences in how they prioritize motivational needs (Existence, Relatedness, Growth) and respond to social-identity cues—some models amplify or attenuate bias in morally charged contexts, modifiable for fairness diagnostics (Myung et al., 27 Jun 2025).
• Normative Benchmarks: For SPERET, aggregate means facilitate the identification of high and low reflexivity; similarly, scenario-based tasks use quantifiable alignment with professional norms to identify ethical shortcomings and areas for model/system improvement (Hindennach et al., 24 Nov 2025).

6. Implications for Benchmark Ethics and Responsible AI

PsychEthicsBench operationalizes practical recommendations for embedding ethics into benchmarking pipelines (Blili-Hamelin et al., 2022):

• Documentation of Value-Laden Choices: Explicit rationales for scenario/task selection, weighting, and metric prioritization are central, facilitating transparency and reflexivity throughout the benchmark lifecycle.
• Participatory and Inclusive Scenario Construction: Involvement of psychologists, ethicists, clinicians, and marginalized communities in scenario design mitigates path dependence and increases relevance and justice (Blili-Hamelin et al., 2022).
• Continuous Update and Governance: The infrastructure supports dynamic inclusion of new scenarios, expert review cycles, audit logs, and stakeholder-driven adaptation of legal/normative shifts (Kasu, 15 Sep 2025).
• Cross-Domain and Contextual Adaptation: Methods and metrics are extensible to other domains (e.g., sensor privacy, biometric use) by adapting instrument text and re-validating constructs through empirical and expert-driven procedures (Hindennach et al., 24 Nov 2025).

The framework incentivizes more nuanced, context-aware, and justice-oriented approaches to AI safety and professional alignment in high-impact, person-centered decision domains.

7. Future Directions and Extensions

Key avenues for continued development and refinement of PsychEthicsBench include:

• Automated Ethicality Classifiers: Development of rule-based or learned classifiers for scalable, reproducible evaluation, supplementing expert and LLM-based judging (Shen et al., 7 Jan 2026).
• Multidimensional Scenario Expansion: Extension beyond binary outcomes and static vignettes to support interactive, multi-turn investigations and broader ethical systems (utilitarianism, virtue ethics, culturally specific frames) (Myung et al., 27 Jun 2025).
• Empirical Studies and Standardization: Inter-annotator agreement studies, expansion of demographic and cultural coverage, and tooling for rubric-based scoring to support generalization and equity (Kasu, 15 Sep 2025).
• Evolving Benchmarks for Justice: Ongoing efforts to identify and reduce barriers faced by marginalized populations in psychological technologies, explicitly measuring and addressing systemic sources of harm and inclusion gaps (Blili-Hamelin et al., 2022).

PsychEthicsBench thus establishes a replicable, context-sensitive, and ethically self-aware foundation for systematic evaluation and improvement of intelligent systems engaged in psychological, clinical, and related domains, catalyzing responsible research and deployment at scale.