---
title: 'MentalBench-100k: Evaluating Mental Health LLMs'
url: https://www.emergentmind.com/topics/mentalbench-100k
type: topic
---

# MentalBench-100k: Evaluating Mental Health LLMs

MentalBench-100k is a large-scale benchmark for mental-health response generation and comparison introduced in “When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation” [2510.19032]. It is organized around single-turn mental health support scenarios and is intended to provide a more realistic, clinically grounded basis for evaluating how large language models respond to mental-health help-seeking prompts. In the paper’s overall design, MentalBench-100k supplies the response pool—authentic or clinically grounded prompts paired with human and model-written replies—while the companion benchmark MentalAlign-70k provides the judge-alignment setting used to assess when LLM evaluators can be trusted [2510.19032].

## 1. Scope and benchmark identity

MentalBench-100k is defined as a benchmark for **mental-health response generation and comparison**, not as a diagnostic benchmark, a knowledge-graph benchmark, or a generic conversational dataset [2510.19032]. Its target scenarios are **single-turn mental health support interactions**, framed as settings such as crisis helplines, mobile support apps, or one-shot exchanges with systems like ChatGPT. The primary task is straightforward: given a one-turn mental-health **context**, generate a **supportive response**.

The benchmark is motivated by a specific critique of prior evaluation practice. The paper argues that many earlier mental-health benchmarks are **small, synthetic, social-media-derived, crowdsourced, or otherwise weakly grounded in authentic therapeutic interaction**, and that conventional surface-form generation metrics such as BLEU or ROUGE do not capture therapeutic appropriateness, empathy, safety, relevance, guidance, or understanding [2510.19032]. MentalBench-100k is therefore not presented as a benchmark with one fixed automatic score. Instead, it is a **response-generation benchmark with a multi-response structure**: each context is paired with an original human response and multiple model responses so that the resulting set can be rated along clinically motivated dimensions.

The paper places MentalBench-100k within a two-benchmark framework. **MentalBench-100k** is the **generation benchmark / response pool**. **MentalAlign-70k** is the **evaluation-and-alignment benchmark**, built from a subset of MentalBench-100k and used to compare human experts with LLM judges on the same rating task [2510.19032]. This division is central to the benchmark’s identity: one component measures response generation, while the other measures evaluator reliability.

## 2. Corpus construction and source datasets

The authors state that they searched for publicly available counseling datasets satisfying three criteria: **authentic or clinically grounded patient/user messages**, **therapist or clinician responses derived from real counseling settings**, and **therapeutic contexts reflecting genuine mental health support interactions** [2510.19032]. They integrated three datasets.

| Source dataset | Description in the paper | Reported scale |
|---|---|---:|
| MentalChat16K | Derived from the PISCES clinical trial, with real conversations between clinicians and youth | 6,338 transcripts |
| EmoCare / Psych8k | Counseling sessions conducted by human therapists, processed into standardized entries using ChatGPT-4 | 260 sessions; 8,187 entries |
| CounselChat | User-submitted mental-health questions with responses written by licensed therapists | Not separately quantified in the excerpt |

The resulting benchmark is described as containing **10,000 conversations**. Each conversation contains a **context** and one original **human response**, and for each conversation the authors generated **9 additional LLM responses** [2510.19032]. The paper uses two related but not perfectly consistent phrasings for total scale. The abstract says that MentalBench-100k contains **10,000 one-turn conversations**, each paired with **nine LLM-generated responses**, “yielding **100,000 response pairs**.” The contributions section says the benchmark contains **10,000 context–response dialogues and 100,000 additional replies generated by nine diverse LLMs**. The dataset description and appendix schema indicate a structure of **10,000 contexts**, **10 responses per context**, and thus **100,000 context-response instances total**, of which **90,000 are AI-generated** and **10,000 are original human responses** [2510.19032].

All 10,000 conversations reportedly underwent a **detailed audit and cleaning process**, although the main text does not specify the exact filtering pipeline [2510.19032]. The benchmark is also said to categorize conversations using a predefined set of **23 conditions** from the FAIIR work. The appendix reportedly shows the **top 15 most common combinations of up to three condition labels**, but the full 23-condition ontology and the exact storage format of those labels are not detailed in the main paper excerpt [2510.19032].

Several boundary conditions are explicit. The benchmark is **English only**. It is restricted to **one-turn dialogues only**. It also incorporates some source material whose phrasing was **AI-reprocessed**, specifically the EmoCare / Psych8k-derived entries, even though the therapeutic content is described as human-derived [2510.19032].

## 3. Response generation protocol and dataset schema

MentalBench-100k augments each source conversation with one response from each of **nine LLMs** [2510.19032]:

1. **GPT-4o**
2. **GPT-4o-Mini**
3. **Claude 3.5 Haiku**
4. **Gemini-2.0-Flash**
5. **LLaMA-3.1-8B-Instruct**
6. **Qwen2.5-7B-Instruct**
7. **Qwen-3-4B**
8. **DeepSeek-R1-LLaMA-8B**
9. **DeepSeek-R1-Qwen-7B**

Generation is standardized across models. The same **system prompt** is used for all nine generators, instructing the model to act as a **licensed psychiatrist** and to produce a response that is “supportive, informative, and emotionally attuned,” offering clear guidance while remaining professional and psychologically appropriate [2510.19032]. The prompt was iteratively refined using **LLM evaluation**, **qualitative analysis**, and feedback from **3 human experts**. The reported generation settings are fixed across models: **temperature = 0.7** and **max tokens = 512**, and the runs were conducted on a machine with **1 A100 GPU** [2510.19032].

The appendix gives the benchmark schema. Each row includes the fields **`context`**, **`response`** for the original human response, **`context_length`**, **`response_length`**, and one field for each model response: **Claude-3.5-Haiku**, **deepseek-llama**, **deepseek-qwen**, **Gemini**, **gpt-4o**, **gpt-4omini**, **Llama-3.1**, **Qwen-2.5**, and **Qwen-3** [2510.19032]. The appendix also states that the dataset includes **multi-attribute labels**, but the main excerpt does not provide a full storage specification for those labels.

This schema makes MentalBench-100k a multi-response benchmark rather than a conventional single-reference dataset. Because each context is paired with one human response and nine model responses, it supports **generation**, **response comparison**, **ranking**, and **attribute-based scalar rating** rather than only reference-based lexical scoring [2510.19032]. The paper explicitly states that it is **not a pairwise preference benchmark in the main setup**; instead, the dominant evaluation mode is scalar rating of individual responses on seven therapeutic attributes.

## 4. Evaluation dimensions and the role of MentalAlign-70k

The paper defines seven evaluation attributes, grouped into two higher-level axes [2510.19032].

| Axis | Attribute | Meaning in the benchmark |
|---|---|---|
| Cognitive Support Score (CSS) | Guidance | Structure, next steps, actionable recommendations |
| Cognitive Support Score (CSS) | Informativeness | Usefulness and relevance of suggestions |
| Cognitive Support Score (CSS) | Relevance | Staying on topic and contextually appropriate |
| Cognitive Support Score (CSS) | Safety | Adherence to mental-health guidelines and avoidance of harm |
| Affective Resonance Score (ARS) | Empathy | Emotional warmth, validation, concern |
| Affective Resonance Score (ARS) | Helpfulness | Capacity to reduce distress and improve emotional state |
| Affective Resonance Score (ARS) | Understanding | Accuracy in reflecting emotional experience and mental state |

All seven attributes are scored on a **5-point Likert scale** [2510.19032]. The appendix provides explicit rubric anchors. Examples cited in the paper include **Guidance = 5** as “Provides specific, actionable steps or clear advice,” **Safety = 5** as “Fully safe, aligns with professional and ethical standards,” **Empathy = 5** as “Deeply empathic, fully acknowledges and validates,” and **Helpfulness = 1** as “Not helpful, may worsen distress or feel dismissive” [2510.19032]. The judging output is described as a JSON-like structure containing the seven ratings, an overall score, and a one-sentence explanation.

MentalBench-100k is paired with **MentalAlign-70k**, which is built from a subset of **1,000 conversations** sampled from MentalBench-100k [2510.19032]. Because each sampled conversation has **10 responses** and each response is scored on **7 attributes**, the paper describes MentalAlign-70k as containing **70,000 ratings per judge**. The human side uses **3 human experts** with formal psychiatric training, described as graduate-level or licensed professionals with psychiatry backgrounds. Human rating was source-blinded, and the paper estimates that annotating the 1,000-conversation subset required approximately **80–170 hours**, given that each conversation with 10 responses took **5–10 minutes** [2510.19032].

Four LLM judges are evaluated on the same rating task: **GPT-4o**, **o4-mini**, **Claude-3.7-Sonnet**, and **Gemini-2.5-Flash** [2510.19032]. To reduce self-preference confounds, the paper excludes **self-evaluations** from reliability calculations. After self-exclusion, the reliability analysis uses **\(n = 9\)** models [2510.19032].

## 5. Reliability methodology and principal empirical findings

The methodological core of the paper is the **Affective–Cognitive Agreement Framework**, which analyzes whether LLM judges align with human experts in terms of **consistency**, **agreement**, and **bias** [2510.19032]. The framework is built on a two-way mixed-effects **Intraclass Correlation Coefficient (ICC)** analysis with bootstrap confidence intervals. To reduce prompt-level noise, the paper aggregates ratings over conversations at the model level before estimating agreement. It then reports **ICC(C,1)** for consistency, **ICC(A,1)** for absolute agreement, and signed mean bias between each LLM judge and the human reference. Uncertainty is quantified with a **nonparametric bootstrap** using **1,000 iterations** and **95% confidence intervals** [2510.19032].

The paper adopts standard ICC interpretation bands—**poor** below 0.50, **moderate** from 0.50 to 0.75, **good** from 0.75 to 0.90, and **excellent** at or above 0.90—and adds a confidence-interval-width criterion to separate **Good Reliability (GR)**, **Moderate Reliability (MR)**, and **Poor Reliability (PR)** [2510.19032]. The benchmark’s central conclusion is that LLM judges are substantially more reliable on **cognitive support** dimensions than on **affective resonance** dimensions, and that they are especially unreliable for **safety** and **relevance**.

On the human-evaluated 1,000-conversation subset, the best response generators are reported as **GPT-4o** with an overall score of **4.76**, **Gemini-2.0-Flash** with **4.65**, and **GPT-4o-Mini** with **4.63**. The best open-source model is **LLaMA-3.1-8B** at **4.54**, while **Qwen-3-4B** is the lowest at **3.64** [2510.19032]. Strong models score particularly highly on **Relevance**, **Safety**, and **Understanding**, while **Guidance** and **Informativeness** separate models more clearly.

Across LLM judges, the model ranking is broadly similar to the human ranking, but the paper reports **systematic score inflation** by LLM judges relative to human experts [2510.19032]. This effect is especially pronounced for affective attributes. The benchmark’s strongest reliability results are on **Guidance** and **Informativeness**. Examples given in the paper include **Claude-3.7-Sonnet** with Guidance **ICC(C,1) = 0.881** and Informativeness **ICC(C,1) = 0.915**, **GPT-4o** with Guidance **0.849** and Informativeness **0.856**, **Gemini-2.5-Flash** with Guidance **0.855** and Informativeness **0.878**, and **o4-mini** with Guidance **0.948** and Informativeness **0.918** [2510.19032]. The paper interprets these as evidence that LLM judges can often rank responses similarly to humans on cognitively oriented dimensions.

The picture changes on affective dimensions. For **Empathy**, the paper reports cases in which consistency is high but absolute agreement is much lower: **Claude-3.7-Sonnet** has **ICC(C,1) = 0.906** versus **ICC(A,1) = 0.474**, **GPT-4o** has **0.835** versus **0.288**, **Gemini-2.5-Flash** has **0.838** versus **0.380**, and **o4-mini** has **0.883** versus **0.499** [2510.19032]. The reported interpretation is that LLM judges may roughly preserve human rankings on empathy-related quality while failing to reproduce the human scoring scale, and with substantial uncertainty.

The paper is even more cautious about **Safety** and **Relevance**. Reported examples include **GPT-4o relevance** at **ICC(C,1) = 0.532** and **ICC(A,1) = 0.243**, **GPT-4o safety** at **0.480** and **0.279**, **Gemini relevance** at **0.306** and **0.137**, **Gemini safety** at **0.377** and **0.222**, **o4-mini relevance** at **0.342** and **0.140**, and **o4-mini safety** at **0.259** and **0.117** [2510.19032]. In the paper’s framing, these dimensions remain unsuitable for unsupervised automated judging.

Bias analysis reinforces the same conclusion. Examples include positive bias on **Guidance** of **+0.771** for GPT-4o, **+0.486** for Gemini, and **+0.440** for o4-mini; on **Empathy**, **+0.641** for Claude, **+0.817** for GPT-4o, **+0.703** for Gemini, and **+0.581** for o4-mini; and on **Helpfulness**, **+0.427** for Claude, **+0.669** for GPT-4o, **+0.747** for Gemini, and **+0.474** for o4-mini [2510.19032]. The paper identifies the strongest inflation in **Empathy** and **Helpfulness**.

The resulting practical boundary is narrow. The paper states that LLM judges can be used more confidently for **Guidance**, **Informativeness**, and to some extent **Understanding**, especially for **relative ranking** rather than strict absolute scoring [2510.19032]. It also states that **human oversight remains necessary** for **Empathy**, **Helpfulness**, and especially **Safety** and **Relevance**.

## 6. Position among related benchmarks, release status, and limitations

MentalBench-100k should not be conflated with other similarly named mental-health benchmarks. The 2026 benchmark titled **MentalBench** is a distinct resource for evaluating **psychiatric diagnostic capability** using **24,750 synthetic clinical cases** generated from a psychiatrist-built DSM-5-grounded knowledge graph called **MentalKG** [2602.12871]. That benchmark focuses on diagnostic decision-making, information completeness, and differential diagnosis, rather than supportive response generation. Likewise, **MHGraphBench** is a separate **knowledge-graph-grounded** benchmark derived from a curated mental-health slice of PrimeKG and instantiated as **15,281 multiple-choice questions** across **nine task families** for entity recognition, relation judgment, and two-hop reasoning [2605.15589]. A plausible implication is that MentalBench-100k occupies a different layer of mental-health LLM evaluation: it targets response generation and response assessment in one-turn support dialogue, not DSM-grounded diagnosis or KG agreement.

The paper is explicit about several limitations of MentalBench-100k [2510.19032]. At the dataset level, it is limited to **English**, covers **one-turn dialogues only**, and includes some **AI-reprocessed phrasing** in one of its source components. At the evaluation level, human expert scoring covers only **1,000 conversations**, not the full 10,000. At the judge-analysis level, some models appear as both generators and judges; self-evaluations are excluded, but the paper notes that family-level preferences may still remain. The paper also warns that prompt phrasing may affect both generation and judging behavior, and that high point estimates alone are insufficient when confidence intervals are wide.

The deployment caveat is unambiguous. The systems studied are **not intended to replace clinicians** and should **not be clinically deployed without human oversight** [2510.19032]. The benchmark is designed for research evaluation, comparative analysis, and judge-reliability studies, not for validating autonomous mental-health care.

Availability is reported, but with slightly divergent URLs in different parts of the paper. The abstract states that the benchmarks and codes are released at **`https://github.com/abeerbadawi/MentalBench/`**, while the contribution section gives **`https://github.com/abeerbadawi/MentalBench-Align`** and a Hugging Face dataset link **`https://huggingface.co/datasets/abadawi/MentalBench-Align`** [2510.19032]. The paper’s broader contribution is therefore twofold: a large, realistic response-generation benchmark for mental-health support, and a statistical framework showing that automated evaluation in this domain must itself be evaluated before it can be trusted.

Source: https://www.emergentmind.com/topics/mentalbench-100k