MentalBench-100k: Evaluating Mental Health LLMs

• The paper introduces MentalBench-100k as a multi-response benchmark that pairs authentic and LLM-generated replies for single-turn mental health support, enabling clinically grounded evaluations.
• It integrates real counseling datasets with rigorous auditing, offering diverse assessments across seven therapeutic attributes via both human and LLM judges.
• The benchmark provides actionable insights into LLM performance by demonstrating reliable cognitive scoring while cautioning against unsupervised affective evaluations.

MentalBench-100k is a large-scale benchmark for mental-health response generation and comparison introduced in “When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation” (Badawi et al., 21 Oct 2025). It is organized around single-turn mental health support scenarios and is intended to provide a more realistic, clinically grounded basis for evaluating how LLMs respond to mental-health help-seeking prompts. In the paper’s overall design, MentalBench-100k supplies the response pool—authentic or clinically grounded prompts paired with human and model-written replies—while the companion benchmark MentalAlign-70k provides the judge-alignment setting used to assess when LLM evaluators can be trusted (Badawi et al., 21 Oct 2025).

1. Scope and benchmark identity

MentalBench-100k is defined as a benchmark for mental-health response generation and comparison, not as a diagnostic benchmark, a knowledge-graph benchmark, or a generic conversational dataset (Badawi et al., 21 Oct 2025). Its target scenarios are single-turn mental health support interactions, framed as settings such as crisis helplines, mobile support apps, or one-shot exchanges with systems like ChatGPT. The primary task is straightforward: given a one-turn mental-health context, generate a supportive response.

The benchmark is motivated by a specific critique of prior evaluation practice. The paper argues that many earlier mental-health benchmarks are small, synthetic, social-media-derived, crowdsourced, or otherwise weakly grounded in authentic therapeutic interaction, and that conventional surface-form generation metrics such as BLEU or ROUGE do not capture therapeutic appropriateness, empathy, safety, relevance, guidance, or understanding (Badawi et al., 21 Oct 2025). MentalBench-100k is therefore not presented as a benchmark with one fixed automatic score. Instead, it is a response-generation benchmark with a multi-response structure: each context is paired with an original human response and multiple model responses so that the resulting set can be rated along clinically motivated dimensions.

The paper places MentalBench-100k within a two-benchmark framework. MentalBench-100k is the generation benchmark / response pool. MentalAlign-70k is the evaluation-and-alignment benchmark, built from a subset of MentalBench-100k and used to compare human experts with LLM judges on the same rating task (Badawi et al., 21 Oct 2025). This division is central to the benchmark’s identity: one component measures response generation, while the other measures evaluator reliability.

2. Corpus construction and source datasets

The authors state that they searched for publicly available counseling datasets satisfying three criteria: authentic or clinically grounded patient/user messages, therapist or clinician responses derived from real counseling settings, and therapeutic contexts reflecting genuine mental health support interactions (Badawi et al., 21 Oct 2025). They integrated three datasets.

Source dataset	Description in the paper	Reported scale
MentalChat16K	Derived from the PISCES clinical trial, with real conversations between clinicians and youth	6,338 transcripts
EmoCare / Psych8k	Counseling sessions conducted by human therapists, processed into standardized entries using ChatGPT-4	260 sessions; 8,187 entries
CounselChat	User-submitted mental-health questions with responses written by licensed therapists	Not separately quantified in the excerpt

The resulting benchmark is described as containing 10,000 conversations. Each conversation contains a context and one original human response, and for each conversation the authors generated 9 additional LLM responses (Badawi et al., 21 Oct 2025). The paper uses two related but not perfectly consistent phrasings for total scale. The abstract says that MentalBench-100k contains 10,000 one-turn conversations, each paired with nine LLM-generated responses, “yielding 100,000 response pairs.” The contributions section says the benchmark contains 10,000 context–response dialogues and 100,000 additional replies generated by nine diverse LLMs. The dataset description and appendix schema indicate a structure of 10,000 contexts, 10 responses per context, and thus 100,000 context-response instances total, of which 90,000 are AI-generated and 10,000 are original human responses (Badawi et al., 21 Oct 2025).

All 10,000 conversations reportedly underwent a detailed audit and cleaning process, although the main text does not specify the exact filtering pipeline (Badawi et al., 21 Oct 2025). The benchmark is also said to categorize conversations using a predefined set of 23 conditions from the FAIIR work. The appendix reportedly shows the top 15 most common combinations of up to three condition labels, but the full 23-condition ontology and the exact storage format of those labels are not detailed in the main paper excerpt (Badawi et al., 21 Oct 2025).

Several boundary conditions are explicit. The benchmark is English only. It is restricted to one-turn dialogues only. It also incorporates some source material whose phrasing was AI-reprocessed, specifically the EmoCare / Psych8k-derived entries, even though the therapeutic content is described as human-derived (Badawi et al., 21 Oct 2025).

3. Response generation protocol and dataset schema

MentalBench-100k augments each source conversation with one response from each of nine LLMs (Badawi et al., 21 Oct 2025):

1. GPT-4o
2. GPT-4o-Mini
3. Claude 3.5 Haiku
4. Gemini-2.0-Flash
5. LLaMA-3.1-8B-Instruct
6. Qwen2.5-7B-Instruct
7. Qwen-3-4B
8. DeepSeek-R1-LLaMA-8B
9. DeepSeek-R1-Qwen-7B

Generation is standardized across models. The same system prompt is used for all nine generators, instructing the model to act as a licensed psychiatrist and to produce a response that is “supportive, informative, and emotionally attuned,” offering clear guidance while remaining professional and psychologically appropriate (Badawi et al., 21 Oct 2025). The prompt was iteratively refined using LLM evaluation, qualitative analysis, and feedback from 3 human experts. The reported generation settings are fixed across models: temperature = 0.7 and max tokens = 512, and the runs were conducted on a machine with 1 A100 GPU (Badawi et al., 21 Oct 2025).

The appendix gives the benchmark schema. Each row includes the fields context, response for the original human response, context_length, response_length, and one field for each model response: Claude-3.5-Haiku, deepseek-llama, deepseek-qwen, Gemini, gpt-4o, gpt-4omini, Llama-3.1, Qwen-2.5, and Qwen-3 (Badawi et al., 21 Oct 2025). The appendix also states that the dataset includes multi-attribute labels, but the main excerpt does not provide a full storage specification for those labels.

This schema makes MentalBench-100k a multi-response benchmark rather than a conventional single-reference dataset. Because each context is paired with one human response and nine model responses, it supports generation, response comparison, ranking, and attribute-based scalar rating rather than only reference-based lexical scoring (Badawi et al., 21 Oct 2025). The paper explicitly states that it is not a pairwise preference benchmark in the main setup; instead, the dominant evaluation mode is scalar rating of individual responses on seven therapeutic attributes.

4. Evaluation dimensions and the role of MentalAlign-70k

The paper defines seven evaluation attributes, grouped into two higher-level axes (Badawi et al., 21 Oct 2025).

Axis	Attribute	Meaning in the benchmark
Cognitive Support Score (CSS)	Guidance	Structure, next steps, actionable recommendations
Cognitive Support Score (CSS)	Informativeness	Usefulness and relevance of suggestions
Cognitive Support Score (CSS)	Relevance	Staying on topic and contextually appropriate
Cognitive Support Score (CSS)	Safety	Adherence to mental-health guidelines and avoidance of harm
Affective Resonance Score (ARS)	Empathy	Emotional warmth, validation, concern
Affective Resonance Score (ARS)	Helpfulness	Capacity to reduce distress and improve emotional state
Affective Resonance Score (ARS)	Understanding	Accuracy in reflecting emotional experience and mental state

All seven attributes are scored on a 5-point Likert scale (Badawi et al., 21 Oct 2025). The appendix provides explicit rubric anchors. Examples cited in the paper include Guidance = 5 as “Provides specific, actionable steps or clear advice,” Safety = 5 as “Fully safe, aligns with professional and ethical standards,” Empathy = 5 as “Deeply empathic, fully acknowledges and validates,” and Helpfulness = 1 as “Not helpful, may worsen distress or feel dismissive” (Badawi et al., 21 Oct 2025). The judging output is described as a JSON-like structure containing the seven ratings, an overall score, and a one-sentence explanation.

MentalBench-100k is paired with MentalAlign-70k, which is built from a subset of 1,000 conversations sampled from MentalBench-100k (Badawi et al., 21 Oct 2025). Because each sampled conversation has 10 responses and each response is scored on 7 attributes, the paper describes MentalAlign-70k as containing 70,000 ratings per judge. The human side uses 3 human experts with formal psychiatric training, described as graduate-level or licensed professionals with psychiatry backgrounds. Human rating was source-blinded, and the paper estimates that annotating the 1,000-conversation subset required approximately 80–170 hours, given that each conversation with 10 responses took 5–10 minutes (Badawi et al., 21 Oct 2025).

Four LLM judges are evaluated on the same rating task: GPT-4o, o4-mini, Claude-3.7-Sonnet, and Gemini-2.5-Flash (Badawi et al., 21 Oct 2025). To reduce self-preference confounds, the paper excludes self-evaluations from reliability calculations. After self-exclusion, the reliability analysis uses $n = 9$ models (Badawi et al., 21 Oct 2025).

5. Reliability methodology and principal empirical findings

The methodological core of the paper is the Affective–Cognitive Agreement Framework, which analyzes whether LLM judges align with human experts in terms of consistency, agreement, and bias (Badawi et al., 21 Oct 2025). The framework is built on a two-way mixed-effects Intraclass Correlation Coefficient (ICC) analysis with bootstrap confidence intervals. To reduce prompt-level noise, the paper aggregates ratings over conversations at the model level before estimating agreement. It then reports ICC(C,1) for consistency, ICC(A,1) for absolute agreement, and signed mean bias between each LLM judge and the human reference. Uncertainty is quantified with a nonparametric bootstrap using 1,000 iterations and 95% confidence intervals (Badawi et al., 21 Oct 2025).

The paper adopts standard ICC interpretation bands—poor below 0.50, moderate from 0.50 to 0.75, good from 0.75 to 0.90, and excellent at or above 0.90—and adds a confidence-interval-width criterion to separate Good Reliability (GR), Moderate Reliability (MR), and Poor Reliability (PR) (Badawi et al., 21 Oct 2025). The benchmark’s central conclusion is that LLM judges are substantially more reliable on cognitive support dimensions than on affective resonance dimensions, and that they are especially unreliable for safety and relevance.

On the human-evaluated 1,000-conversation subset, the best response generators are reported as GPT-4o with an overall score of 4.76, Gemini-2.0-Flash with 4.65, and GPT-4o-Mini with 4.63. The best open-source model is LLaMA-3.1-8B at 4.54, while Qwen-3-4B is the lowest at 3.64 (Badawi et al., 21 Oct 2025). Strong models score particularly highly on Relevance, Safety, and Understanding, while Guidance and Informativeness separate models more clearly.

Across LLM judges, the model ranking is broadly similar to the human ranking, but the paper reports systematic score inflation by LLM judges relative to human experts (Badawi et al., 21 Oct 2025). This effect is especially pronounced for affective attributes. The benchmark’s strongest reliability results are on Guidance and Informativeness. Examples given in the paper include Claude-3.7-Sonnet with Guidance ICC(C,1) = 0.881 and Informativeness ICC(C,1) = 0.915, GPT-4o with Guidance 0.849 and Informativeness 0.856, Gemini-2.5-Flash with Guidance 0.855 and Informativeness 0.878, and o4-mini with Guidance 0.948 and Informativeness 0.918 (Badawi et al., 21 Oct 2025). The paper interprets these as evidence that LLM judges can often rank responses similarly to humans on cognitively oriented dimensions.

The picture changes on affective dimensions. For Empathy, the paper reports cases in which consistency is high but absolute agreement is much lower: Claude-3.7-Sonnet has ICC(C,1) = 0.906 versus ICC(A,1) = 0.474, GPT-4o has 0.835 versus 0.288, Gemini-2.5-Flash has 0.838 versus 0.380, and o4-mini has 0.883 versus 0.499 (Badawi et al., 21 Oct 2025). The reported interpretation is that LLM judges may roughly preserve human rankings on empathy-related quality while failing to reproduce the human scoring scale, and with substantial uncertainty.

The paper is even more cautious about Safety and Relevance. Reported examples include GPT-4o relevance at ICC(C,1) = 0.532 and ICC(A,1) = 0.243, GPT-4o safety at 0.480 and 0.279, Gemini relevance at 0.306 and 0.137, Gemini safety at 0.377 and 0.222, o4-mini relevance at 0.342 and 0.140, and o4-mini safety at 0.259 and 0.117 (Badawi et al., 21 Oct 2025). In the paper’s framing, these dimensions remain unsuitable for unsupervised automated judging.

Bias analysis reinforces the same conclusion. Examples include positive bias on Guidance of +0.771 for GPT-4o, +0.486 for Gemini, and +0.440 for o4-mini; on Empathy, +0.641 for Claude, +0.817 for GPT-4o, +0.703 for Gemini, and +0.581 for o4-mini; and on Helpfulness, +0.427 for Claude, +0.669 for GPT-4o, +0.747 for Gemini, and +0.474 for o4-mini (Badawi et al., 21 Oct 2025). The paper identifies the strongest inflation in Empathy and Helpfulness.

The resulting practical boundary is narrow. The paper states that LLM judges can be used more confidently for Guidance, Informativeness, and to some extent Understanding, especially for relative ranking rather than strict absolute scoring (Badawi et al., 21 Oct 2025). It also states that human oversight remains necessary for Empathy, Helpfulness, and especially Safety and Relevance.

6. Position among related benchmarks, release status, and limitations

MentalBench-100k should not be conflated with other similarly named mental-health benchmarks. The 2026 benchmark titled MentalBench is a distinct resource for evaluating psychiatric diagnostic capability using 24,750 synthetic clinical cases generated from a psychiatrist-built DSM-5-grounded knowledge graph called MentalKG (Song et al., 13 Feb 2026). That benchmark focuses on diagnostic decision-making, information completeness, and differential diagnosis, rather than supportive response generation. Likewise, MHGraphBench is a separate knowledge-graph-grounded benchmark derived from a curated mental-health slice of PrimeKG and instantiated as 15,281 multiple-choice questions across nine task families for entity recognition, relation judgment, and two-hop reasoning (Liu et al., 15 May 2026). A plausible implication is that MentalBench-100k occupies a different layer of mental-health LLM evaluation: it targets response generation and response assessment in one-turn support dialogue, not DSM-grounded diagnosis or KG agreement.

The paper is explicit about several limitations of MentalBench-100k (Badawi et al., 21 Oct 2025). At the dataset level, it is limited to English, covers one-turn dialogues only, and includes some AI-reprocessed phrasing in one of its source components. At the evaluation level, human expert scoring covers only 1,000 conversations, not the full 10,000. At the judge-analysis level, some models appear as both generators and judges; self-evaluations are excluded, but the paper notes that family-level preferences may still remain. The paper also warns that prompt phrasing may affect both generation and judging behavior, and that high point estimates alone are insufficient when confidence intervals are wide.

The deployment caveat is unambiguous. The systems studied are not intended to replace clinicians and should not be clinically deployed without human oversight (Badawi et al., 21 Oct 2025). The benchmark is designed for research evaluation, comparative analysis, and judge-reliability studies, not for validating autonomous mental-health care.

Availability is reported, but with slightly divergent URLs in different parts of the paper. The abstract states that the benchmarks and codes are released at https://github.com/abeerbadawi/MentalBench/, while the contribution section gives https://github.com/abeerbadawi/MentalBench-Align and a Hugging Face dataset link https://huggingface.co/datasets/abadawi/MentalBench-Align (Badawi et al., 21 Oct 2025). The paper’s broader contribution is therefore twofold: a large, realistic response-generation benchmark for mental-health support, and a statistical framework showing that automated evaluation in this domain must itself be evaluated before it can be trusted.