PsychoLexQuery: Persian Therapy Evaluation

• PsychoLexQuery is a single-turn dataset focused on authentic Persian psychotherapy-style dialogue that emphasizes cultural fit, therapeutic routing, and reasoning.
• It is constructed from real, anonymized user questions sourced from Persian counseling forums, ensuring ecological validity and cultural authenticity.
• Evaluation uses a 10-point Likert scale to assess empathy, coherence, therapeutic alignment, and structured reasoning against multiple baseline models.

Searching arXiv for papers directly relevant to PsychoLexQuery and its surrounding framework. PsychoLexQuery is a single-turn evaluation dataset for Persian psychotherapy-style dialogue, introduced within the PsychoLexTherapy framework to test whether a model can answer a real user’s mental-health question with empathy, coherence, cultural fit, and therapy-oriented reasoning (Abbasi et al., 4 Oct 2025). It is constructed from publicly available Persian counseling interactions rather than synthetic prompts, and it functions as the single-turn counterpart to the multi-turn PsychoLexDialogue resource in the same framework. In this setting, PsychoLexQuery is not a general psychological QA benchmark; it is a targeted evaluation resource for culturally grounded therapeutic response generation in Persian.

1. Definition and institutional role

PsychoLexQuery serves as the single-turn benchmark in a three-stage system development process consisting of: assessing SLMs psychological knowledge with PsychoLexEval; designing and implementing the reasoning-oriented PsychoLexTherapy framework; and constructing two evaluation datasets—PsychoLexQuery and PsychoLexDialogue—to benchmark against multiple baselines (Abbasi et al., 4 Oct 2025). Its immediate purpose is to evaluate whether a small LLM can respond appropriately to a one-shot counseling-style message without relying on multi-turn memory.

Within that design, PsychoLexQuery occupies a specific methodological niche. It evaluates response generation to a single user message, whereas PsychoLexDialogue extends the same problem into longer therapeutic interactions. The distinction is operationally important because, for the single-turn benchmark, memory is inactive, while in multi-turn evaluation the long-term memory module is activated. This means PsychoLexQuery isolates the contribution of therapeutic routing, structured reasoning, and culturally aligned response formulation from the separate contribution of structured memory.

The dataset is therefore best understood as an evaluation layer for psychotherapy simulation rather than as a corpus for general mental-health classification or ordinary instruction following. Its emphasis is on whether a generated answer is therapeutically coherent in Persian, not merely fluent or emotionally warm.

2. Data acquisition and anonymization

PsychoLexQuery was built from real Persian user questions collected from Persian-language psychology forums and counseling platforms, specifically EhyaCenter+, Moshaverfa, and Simiaroom (Abbasi et al., 4 Oct 2025). The construction procedure used web crawling and manual extraction of publicly available posts in which users describe psychological concerns and counselors or psychologists respond.

The paper states three reasons for this sourcing strategy: ecological validity, because the questions are real rather than synthetic; cultural authenticity, because the language, concerns, and social context reflect Persian/Iranian user experience; and therapy relevance, because the prompts come from genuine counseling situations. These properties distinguish PsychoLexQuery from evaluation sets assembled from examination material or automatically generated scenarios.

Because the source material is sensitive, the preprocessing pipeline applied extensive anonymization. Names were replaced with placeholders such as “Person A”; locations were generalized into coarse categories such as “large city”; timestamps were abstracted to month/year level; phone numbers, emails, and social media handles were removed; and specific job or project identifiers were generalized. The stated goal was to preserve cultural realism while protecting privacy. This design makes the dataset suitable for model evaluation in a mental-health setting where confidentiality is not incidental but structurally central.

3. Size, topics, and task characteristics

The paper states that PsychoLexQuery contains approximately 4,000 user questions (Abbasi et al., 4 Oct 2025). The topics span common mental-health concerns in Persian contexts, including relationship difficulties, anxiety, depression, family conflict, academic pressure, workplace stress, self-esteem issues, social anxiety, grief and loss, and behavioral issues in children/adolescents. Figure and table descriptions indicate that relationship difficulties were the dominant category and that multiple emotional themes commonly co-occurred in a single question.

This topical profile matters for interpretation. PsychoLexQuery is not limited to one diagnostic domain, one counseling school, or one user demographic scenario. Instead, it samples the heterogeneous problem space of counseling-style help seeking as it appears in Persian online platforms. A plausible implication is that strong performance on the dataset requires robust handling of mixed affective and interpersonal content rather than narrow optimization for one symptom label.

The task formulation is also narrower than generic open-ended dialogue. A system must answer a real counseling-style question in a way that is judged on empathy, coherence and structure, cultural fit, therapeutic alignment, content accuracy, adaptability, linguistic fluency, clarity, and human-likeness on a 10-point Likert scale in the automatic evaluation protocol. That specification makes PsychoLexQuery a response-quality benchmark with explicit therapeutic and cultural criteria, not simply a next-turn prediction dataset.

4. Integration with PsychoLexTherapy

PsychoLexQuery is inseparable from the architecture it was designed to test. PsychoLexTherapy is intended to simulate how a therapist reasons, rather than merely generating generic empathetic text, by first selecting a therapeutic orientation and then following a structured reasoning path aligned with that orientation (Abbasi et al., 4 Oct 2025). The framework explicitly models Cognitive Behavioral Therapy (CBT), Reality Therapy (RT), and Person-Centered Therapy (PCT).

The routing mechanism is the Therapy Approach Selector. It chooses a path according to message cues: CBT for negative automatic thoughts, distortions, or hopelessness; PCT for empathy, acceptance, and emotional reflection; and RT for needs, responsibility, choices, and action. After selection, the system executes an explicit multi-step path. The CBT path performs extraction of automatic thoughts, inference of emotional consequences, projection of behavioral tendencies, generation of balanced alternatives, derivation of adaptive behaviors, and synthesis into a final response. The RT path identifies needs and wants, analyzes current behaviors, evaluates consequences, plans alternative behaviors, and integrates them into a final response. The PCT path performs empathic reflection, exploratory questioning for self-awareness, and synthesis into a warm, non-directive response.

In the PsychoLexQuery setting, these structured paths are evaluated without long-term memory. That design choice is methodologically important because it tests whether therapeutic structure alone improves single-turn response quality. The broader framework uses small LLMs rather than large cloud-only systems. Candidate models were restricted to under 10 billion parameters for technical feasibility and privacy, with model families including Gemma-3, Qwen-3, LLaMA-3.2, and Mistral, and Gemma-4B chosen as the base model because it balanced sufficient psychological competence, manageable compute requirements, and local deployment/privacy. This suggests that PsychoLexQuery was designed for evaluation under privacy-preserving, on-device therapeutic simulation constraints rather than purely maximal-scale inference.

5. Evaluation protocol and reported findings

The single-turn PsychoLexQuery benchmark compares PsychoLexTherapy with four baselines: Simple Prompt, Simple Prompt + Therapy Selector, Empathy Chain, and Empathic Agents (Abbasi et al., 4 Oct 2025). The Simple Prompt baseline uses a generic empathetic prompt with no explicit therapeutic reasoning. Simple Prompt + Therapy Selector adds routing to CBT/RT/PCT logic without a full structured reasoning chain. Empathy Chain uses a two-phase process consisting of a reasoning trace followed by a final empathetic answer, with CBT, PCT, and RT variants. Empathic Agents uses multi-agent debate, where multiple therapy-aligned agents generate candidate responses, critique one another, and a final decision agent synthesizes the output.

Automatic evaluation uses GPT-5 as an LLM-as-a-judge. In the single-turn setting, outputs are rated on a 10-point Likert scale across the nine criteria already listed. The study also includes a human preference evaluation in which 3 psychology graduate students, blinded to system identity, were shown 5 anonymized outputs and asked to rank them from 1 (best) to 5 (worst).

The reported result is that PsychoLexTherapy outperformed all baselines on automatic LLM-as-a-judge evaluation and was also ranked highest in the human study (Abbasi et al., 4 Oct 2025). The paper characterizes the baseline pattern as follows: Simple Prompt was fluent but weak therapeutically; Simple + Selector was slightly better but still shallow; Empathy Chain sometimes improved reasoning but could become mechanical or verbose; Empathic Agents showed strong performance due to multiple perspectives; and PsychoLexTherapy achieved the best overall balance of empathy, structure, cultural fit, and therapeutic alignment.

The authors attribute this outcome to the combination of explicit therapeutic routing, structured reasoning paths, Persian cultural constraints, and internal quality-control logic. In this interpretation, PsychoLexQuery operationalizes a distinction between generic empathy and therapy-oriented reasoning: success on the dataset requires not only emotionally resonant language but also mode selection and structured response organization.

6. Relation to the broader PsychoLex ecosystem and research significance

PsychoLexQuery belongs to a broader line of psychology-focused LLM resources but occupies a different layer of the stack. The earlier PsychoLex framework introduced PsychoLexQA, PsychoLexEval, and PsychoLexLLaMA as bilingual Persian-English resources for psychology-specific instruction tuning and multiple-choice evaluation; PsychoLexEval contains 3,430 rows and is designed to test psychological knowledge and reasoning rather than psychotherapy-style response generation (Abbasi et al., 2024). In contrast, PsychoLexQuery evaluates responses to real Persian user questions in a counseling setting. The difference is methodological: PsychoLexEval emphasizes domain knowledge and MCQA-style correctness, whereas PsychoLexQuery emphasizes therapeutic response quality under culturally grounded conditions.

This distinction aligns with a broader trend in psychological LLM research. Psyche-R1 argues that reliable psychological systems should jointly integrate empathy, psychological expertise, and reasoning, rather than optimizing only one or two of these properties (Dai et al., 14 Aug 2025). PsychoLexQuery can be read as an evaluation resource built around that same triad, but specialized for Persian psychotherapy-style dialogue and for single-turn assessment. This suggests that the dataset’s significance lies less in raw scale than in the specificity of what it measures: therapeutic appropriateness under real-world Persian mental-health prompting.

The limitations are also explicit. PsychoLexQuery is a single-turn dataset, so it does not directly measure continuity, personalization over time, or long-horizon therapeutic coherence. Those dimensions are addressed instead by PsychoLexDialogue, which is built by expanding PsychoLexQuery into 3,400 dialogues, 16 thematic categories, and an average of 10–14 turns. More generally, the benchmark evaluates psychotherapy simulation, not clinical efficacy, and the surrounding work does not claim readiness for autonomous clinical care.

In encyclopedic terms, PsychoLexQuery is most accurately characterized as a culturally grounded Persian evaluation dataset for one-shot psychotherapeutic response generation. Its defining properties are the use of real user questions, strong anonymization, explicit linkage to therapy-specific reasoning paths, and evaluation protocols that privilege empathy, coherence, cultural fit, and therapeutic alignment over generic fluency alone (Abbasi et al., 4 Oct 2025).