MentalAlign-70k Benchmark

• The paper introduces MentalAlign-70k, a benchmark that uses 70k scalar ratings to compare LLM judgments with those of clinical experts.
• It organizes evaluation around cognitive attributes (Guidance, Informativeness) and affective attributes (Empathy, Helpfulness, Understanding) using a two-way mixed-effects ANOVA.
• Results reveal that while LLM judges are reliable on cognitive metrics, they suffer from calibration and precision issues on affective and safety ratings, warranting human oversight.

Searching arXiv for the primary paper and closely related work on MentalAlign-70k. MentalAlign-70k is a judge-alignment benchmark for mental-health dialogue evaluation introduced in "When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation" (Badawi et al., 21 Oct 2025). It is designed to answer a specific question: when LLMs are used as automatic judges of therapeutic dialogue quality, how closely do they agree with human clinical experts, and on which aspects can they be trusted? The benchmark reuses a 1,000-conversation subset of MentalBench-100k, pairs each conversation with 10 candidate responses, and scores each response on seven therapeutic attributes. Three human clinical experts and four high-performing LLM judges each produce $1{,}000 \times 10 \times 7 = 70{,}000$ scalar ratings, enabling statistically grounded comparison between human and machine evaluation in a safety-sensitive domain.

1. Corpus design and benchmark composition

MentalAlign-70k is not a separate dialogue dataset. It is built on top of MentalBench-100k, which consolidates three clinically grounded sources. MentalChat16K is derived from the PISCES clinical trial and contains 6,338 transcripts of real conversations between clinicians and youth. EmoCare / Psych8k comprises 260 real counseling sessions conducted by human therapists and processed into 8,187 entries; the content is human-derived but standardized or rephrased by GPT-4. CounselChat contains user-submitted questions with licensed therapist responses.

From these sources, 10,000 authentic context-response pairs were curated for MentalBench-100k. MentalAlign-70k selects a representative subset of 1,000 one-turn conversations. Each conversation is paired with 10 responses: the original human therapist answer and 9 LLM-generated answers produced under a consistent “you are a licensed psychiatrist” system prompt, temperature 0.7, and max 512 tokens. The nine generation models are GPT-4o, GPT-4o-Mini, Claude-3.5-Haiku, Gemini-2.0-Flash, LLaMA-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, Qwen-3-4B, DeepSeek-R1-LLaMA-8B, and DeepSeek-R1-Qwen-7B.

At the dataset row level, the structure includes the context, the original response, nine columns for LLM responses, and optional metadata such as condition labels and lengths. In the rating task, judges see the context and one anonymized candidate response, without source information, and assign scores for seven attributes on a 5-point Likert scale, together with a short textual justification.

The “70k” designation refers to the 70,000-rating-per-judge setup rather than to the number of dialogue examples. This distinction is conceptually important because the benchmark’s novelty lies in evaluator alignment, not merely corpus size.

2. Therapeutic attribute ontology: CSS and ARS

MentalAlign-70k organizes evaluation around seven attributes grouped into two higher-level constructs.

The Cognitive Support Score (CSS) covers four cognitive or informational qualities relevant to mental-health support:

• Guidance: ability to offer structure, next steps, and actionable recommendations.
• Informativeness: usefulness and relevance of information to the user’s mental-health concern.
• Relevance: degree to which the response stays on-topic and is contextually appropriate.
• Safety: adherence to mental-health guidelines and avoidance of harmful suggestions.

These attributes are aggregated as

$$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$

The Affective Resonance Score (ARS) covers three emotional or relational qualities:

• Empathy: emotional warmth, validation, and concern expressed.
• Helpfulness: the capacity of the response to reduce distress and improve the user’s emotional state.
• Understanding: how accurately the response reflects the user’s emotional experience and mental state.

These attributes are aggregated as

$$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$

The paper states that both CSS and ARS are grounded in psychotherapy research and mental-health rating instruments such as CTRS and PANAS, but simplified into a Likert 1–5 format aligned with psychiatric community practice. The grouping also encodes a substantive distinction between cognitive support and affective attunement. A plausible implication is that the benchmark does not treat “therapeutic quality” as a unitary scalar; instead, it operationalizes clinically distinct dimensions that may exhibit different judge-reliability profiles.

3. Judges, annotation protocol, and reference standard

MentalAlign-70k compares four LLM judges with three human clinical experts. The LLM judges are GPT-4o, GPT-4o-mini (called O4-Mini), Claude-3.7-Sonnet, and Gemini-2.5-Flash. Each receives a standardized evaluation prompt telling it that it is a “licensed psychiatrist” evaluating a response and asking it to output integer scores for each of the seven attributes plus an explanation. The human side consists of three domain experts with graduate-level or licensed psychiatry or mental-health backgrounds, blind to whether a response was written by a human or by a model and blind to model identity.

All judges use the same 5-point Likert scale and the same rubric. This common protocol is essential because the benchmark is intended to isolate alignment between judges rather than differences induced by incompatible rating instructions. The human ratings are treated as reference, not “perfect ground truth,” but as the normative baseline for alignment analysis. The paper does not report explicit numeric inter-rater reliability among the human raters; disagreements among humans are instead absorbed into the ANOVA formulation and reflected in residual variance.

When the benchmark computes agreement between an LLM judge and the human group, it excludes self-evaluation cases in which the judge rates responses produced by its own model family. Thus, for agreement analysis, the response-model dimension becomes $n = 9$ rather than $n = 10$. This exclusion is intended to reduce self-preference confounds, though the paper also notes that some residual role-overlap concerns remain because certain model families appear as both generators and judges.

4. Affective-Cognitive Agreement Framework

The benchmark’s statistical core is the Affective-Cognitive Agreement Framework, which uses intraclass correlation coefficients with confidence intervals to quantify agreement, consistency, and bias between LLM judges and human experts (Badawi et al., 21 Oct 2025). Let $m = 1{,}000$ denote conversations, $n$ the number of response models, $k = 7$ judges, and $a = 7$ attributes. If $Y_{cija}$ is the 1–5 score assigned by judge $$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$0 to response $$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$1 in conversation $$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$2 on attribute $$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$3, the analysis first aggregates to model-level means

$$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$4

For each attribute, the resulting matrix of model means by judge is analyzed using a two-way mixed-effects ANOVA:

$$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$5

where $$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$6 is the grand mean for attribute $$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$7, $$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$8 is the random effect of response model $$\mathrm{CSS} = \frac{\text{Guidance} + \text{Informativeness} + \text{Relevance} + \text{Safety}}{4}.$$9, $$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$0 is the judge effect, $$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$1 is the judge-response interaction, and $$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$2 is residual error. From this model the framework derives the ANOVA mean squares $$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$3, $$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$4, and $$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$5.

Two ICC variants are used. ICC(C,1) measures consistency, or rank agreement:

$$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$6

ICC(A,1) measures absolute agreement and penalizes scale differences:

$$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$7

The interpretation thresholds are: less than 0.50 poor, 0.50–0.75 moderate, 0.75–0.90 good, and at least 0.90 excellent. Because the effective number of models is small, the framework supplements point estimates with a nonparametric bootstrap over models using 1,000 bootstrap samples and 95% confidence intervals. Confidence-interval width is used as a precision criterion: width $$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$8 indicates Good Reliability (GR), $$\mathrm{ARS} = \frac{\text{Empathy} + \text{Helpfulness} + \text{Understanding}}{3}.$$9 indicates Moderate Reliability (MR), and width $n = 9$0 indicates Poor Reliability (PR).

Bias is quantified as the average difference between an LLM judge’s mean score and the human mean score for the same model and attribute:

$n = 9$1

A normalized absolute bias is also defined as

$n = 9$2

using the 1–5 scale range. Positive $n = 9$3 indicates leniency or inflation relative to humans; negative values indicate harsher scoring.

5. Empirical reliability profile

The benchmark reports systematic inflation by LLM judges across all four LLM judges and attributes (Badawi et al., 21 Oct 2025). On Empathy, the reported biases are +0.641 for Claude-Sonnet, +0.817 for GPT-4o, +0.703 for Gemini-2.5-Flash, and +0.581 for O4-Mini. On Guidance, the reported biases are +0.248 for Claude, +0.771 for GPT-4o, +0.486 for Gemini, and +0.440 for O4-Mini. Informativeness is more mixed, with Claude at $n = 9$4, GPT-4o at +0.461, Gemini at +0.115, and O4-Mini at $n = 9$5. Safety and Relevance biases are described as relatively small, approximately +0.1–0.4, but still positive. The general pattern is therefore leniency, especially on affective attributes.

The strongest reliability results occur on the cognitive attributes Guidance and Informativeness. For Guidance, ICC(C,1) values are 0.881 for Claude, 0.849 for GPT-4o, 0.855 for Gemini, and 0.948 for O4-Mini; the associated CI widths are 0.216, 0.324, 0.398, and 0.233 respectively. For Informativeness, ICC(C,1) values are 0.915 for Claude, 0.856 for GPT-4o, 0.878 for Gemini, and 0.918 for O4-Mini; the corresponding CI widths are 0.142, 0.310, 0.439, and 0.340. The paper interprets these results as good to excellent consistency, with absolute agreement often lower than consistency because of scale inflation, especially for GPT-4o.

The affective attributes show a different pattern. For Empathy, ICC(C,1) values remain high—0.906 for Claude, 0.835 for GPT-4o, 0.838 for Gemini, and 0.883 for O4-Mini—but the confidence intervals are wide, with widths approximately 0.529, 0.560, 0.517, and 0.469, and ICC(A,1) values drop to 0.474, 0.288, 0.380, and 0.499. For Helpfulness, Claude has ICC(C,1) = 0.900 with width 0.258 and ICC(A,1) = 0.742, whereas GPT-4o, Gemini, and O4-Mini exhibit lower precision or weaker agreement. For Understanding, Claude records ICC(C,1) = 0.791 and ICC(A,1) = 0.806, GPT-4o records 0.823 and 0.485, Gemini records 0.362 and 0.180, and O4-Mini records 0.871 and 0.592.

The paper interprets these affective results as reduced precision and reduced calibration. Even when rank consistency appears high, wide confidence intervals and low absolute agreement imply that the true agreement with clinicians may vary substantially. This suggests that LLM judges can overestimate how empathetic or helpful a response is, and do so with unstable precision.

The weakest results are on Relevance and Safety. Relevance yields ICC(C,1) values of 0.730, 0.532, 0.306, and 0.342 for Claude, GPT-4o, Gemini, and O4-Mini, with CI widths of 0.594, 0.559, 0.755, and 0.605. Safety yields ICC(C,1) values of 0.685, 0.480, 0.377, and 0.259, with CI widths of 0.628, 0.741, 0.790, and 0.621. The corresponding ICC(A,1) values are also low. The benchmark therefore concludes that automated safety judging cannot be trusted yet.

6. Trust criteria, limitations, and relation to adjacent work

The benchmark proposes a practical guidance matrix for deciding when an LLM judge may be usable. High ICC, narrow confidence intervals, and low bias indicate a reliable judge that may be usable for automated evaluation with minimal human oversight. High ICC with narrow intervals but moderate bias indicates a consistent ranker that requires calibration. Low ICC with narrow intervals indicates a judge that is reliably unreliable. Any setting with wide confidence intervals is treated as insufficient evidence. Applied to MentalAlign-70k, the “trusted” or more usable region is Guidance and Informativeness for most judges, especially Claude and O4-Mini; Helpfulness and Understanding occupy a caution zone; Empathy requires human oversight because of wide confidence intervals and inflation; Safety and Relevance are not trusted and require human oversight.

Several limitations are explicit. The data scope is English-only and one-turn, with no coverage of multi-turn conversational dynamics. Only 1,000 conversations from the 10,000-example generation benchmark are human-rated. Some source responses, particularly in EmoCare standardization, have been rephrased by LLMs. Prompt sensitivity is not systematically explored beyond one prompt configuration. Cultural and linguistic generalizability is limited because the datasets are primarily drawn from Western and English clinical contexts. The benchmark is also computationally and financially expensive because it combines multi-model response generation with multi-judge, multi-attribute scoring.

Within the broader literature, the benchmark is complemented by work on alignment rather than evaluator calibration. "Multi-Objective Alignment of LLMs for Personalized Psychotherapy" (Beikzadeh et al., 17 Feb 2026) does not introduce a dataset named MentalAlign-70k, but it provides a closely related methodological blueprint: survey-derived personas, therapeutic preference modeling, criterion-specific reward models, and multi-objective direct preference optimization for empathy, safety, active listening, self-motivated change, trust/rapport, and patient autonomy. That paper reports that multi-objective DPO achieves a better empathy-safety balance than single-objective optimization and that blinded clinician evaluation finds LLM-evaluator agreement comparable to inter-clinician reliability. A plausible implication is that MentalAlign-70k and multi-objective psychotherapy alignment address adjacent layers of the same problem: the former asks when LLMs can be trusted as evaluators of therapeutic dialogue, while the latter asks how models can be optimized against clinically meaningful therapeutic criteria.

MentalAlign-70k’s main significance is therefore methodological. It converts large-scale mental-health response benchmarking into a reliability-aware evaluation setting in which judge performance is decomposed into consistency, absolute agreement, precision, and bias. The resulting picture is differentiated rather than uniform: LLM judges are reasonably reliable on cognitive attributes such as guidance and informativeness, substantially less calibrated on affective resonance, and not yet reliable on safety-critical judgment.