PsychoBench: Evaluating the Psychology Intelligence of Large Language Models (2510.01611v2)

Abstract: LLMs have demonstrated remarkable success across a wide range of industries, primarily due to their impressive generative abilities. Yet, their potential in applications requiring cognitive abilities, such as psychological counseling, remains largely untapped. This paper investigates the key question: Can LLMs be effectively applied to psychological counseling? To determine whether an LLM can effectively take on the role of a psychological counselor, the first step is to assess whether it meets the qualifications required for such a role, namely the ability to pass the U.S. National Counselor Certification Exam (NCE). This is because, just as a human counselor must pass a certification exam to practice, an LLM must demonstrate sufficient psychological knowledge to meet the standards required for such a role. To address this, we introduce PsychoBench, a benchmark grounded in U.S.national counselor examinations, a licensure test for professional counselors that requires about 70% accuracy to pass. PsychoBench comprises approximately 2,252 carefully curated single-choice questions, crafted to require deep understanding and broad enough to cover various sub-disciplines of psychology. This benchmark provides a comprehensive assessment of an LLM's ability to function as a counselor. Our evaluation shows that advanced models such as GPT-4o, Llama3.3-70B, and Gemma3-27B achieve well above the passing threshold, while smaller open-source models (e.g., Qwen2.5-7B, Mistral-7B) remain far below it. These results suggest that only frontier LLMs are currently capable of meeting counseling exam standards, highlighting both the promise and the challenges of developing psychology-oriented LLMs.

• The paper introduces a novel benchmark, PsychoBench, to assess LLMs' competency in psychological counseling using standardized exam criteria.
• It details a dataset of 2,252 paraphrased questions across counseling domains, validated by experts to ensure accuracy.
• Experimental results show leading LLMs exceed the NCE pass mark, though gaps in empathy and contextual judgment remain.

PsychoBench: Evaluating the Psychology Intelligence of LLMs

Introduction

LLMs have exhibited strong generative capabilities across numerous tasks, yet their application in domains necessitating cognitive acumen, like psychological counseling, is less explored. The paper "PsychoBench: Evaluating the Psychology Intelligence of LLMs" investigates whether LLMs can serve as effective psychological counselors. The research introduces PsychoBench, a benchmark designed to evaluate the ability of LLMs to meet the competencies required for psychological counseling, using the U.S. National Counselor Certification Exam (NCE) as a baseline.

PsychoBench Dataset

PsychoBench is a dataset comprising approximately 2,252 single-choice questions, tailored to meet the standards of the NCE. These questions span counseling methods, abnormal psychology, developmental psychology, and ethics. Data collection involved gathering publicly available exam questions, which were then paraphrased using GPT-based methods for linguistic variability. Human experts validated these questions to ensure consistency and accuracy. This dataset sets a novel standard for evaluating LLM competence in psychological domains.

Experimental Evaluation

Across the PsychoBench dataset, a diverse set of both open-source and proprietary LLMs was evaluated for their ability to correctly answer psychotherapeutic exam-style questions. The models assessed include variations and scaled models like GPT-4o, Llama3.3-70B, and others. The primary evaluation metric is Top-1 Accuracy, signifying the model's ability to select the correct answer from multiple choices.

Leading proprietary models, particularly GPT-4o, exhibited exceptional performance, achieving over 94\% in Top-1 accuracy—far surpassing the 70\% threshold required to pass the NCE. Comparatively, large open-source models such as Llama3.3-70B achieved high accuracy as well but consistently underperformed relative to their proprietary counterparts. Smaller models demonstrated significantly lower proficiency, barely achieving random guess levels of accuracy.

Results and Analysis

The experimental findings underscore a performance hierarchy. GPT-4-class proprietary models and the largest open-source systems consistently surpassed the NCE pass mark, while mid-sized models generally lingered around this threshold. Smaller models demonstrated considerable deficits, highlighting challenges in applying these less powerful architectures to specialized cognitive tasks.

Notably, the results elucidated that despite high Top-1 accuracy, the Top-2 accuracy indicates models sometimes include the correct answer within their top choices, showcasing partial knowledge and necessitating improvements in answer prioritization and confidence calibration.

Implications and Future Directions

The ability of LLMs to perform at or above professional licensure levels suggests their potential utility in mental health support and educational training domains. However, the research highlights significant gaps in empathy, contextual judgment, and emotional reasoning—competencies crucial for real-world psychological counseling. These results suggest the necessity for continued advancements in LLM development, particularly focusing on enhancing domain adaptation, emotional sensitivity, and ethical reasoning capabilities.

Future explorations may focus on integrating multimodality, further refining reasoning calibration, and extending LLM evaluations across differing cultural and linguistic contexts. The release of PsychoBench is a pivotal step toward fostering research at the intersection of LLMs and psychological counseling, aiming to yield safe, effective AI systems that can complement human expertise in mental health care.

Conclusion

"PsychoBench: Evaluating the Psychology Intelligence of LLMs" advances the understanding of LLM capacities in specialized cognitive tasks by introducing a novel evaluation benchmark grounded in professional psychological counseling standards. While leading LLM architectures have demonstrated impressive proficiency, considerable work remains to enhance open-source models to achieve reliable competency equitably. This benchmark serves as a catalyst for further research to ensure AI systems can efficiently and ethically support mental health initiatives.