Multi-Level Narrative Evaluation Outperforms Lexical Features for Mental Health

Abstract: How people narrate their experiences offers a window into how the mind organizes them. Computational approaches to therapeutic writing have evolved from lexical counting to neural methods, yet remain fragmented: dictionary tools miss discourse structure, while embeddings conflate local coherence with global organization. No existing framework maps these techniques onto the hierarchical processes through which narratives are constructed. Here we introduce a three-level framework - micro-level lexical features, meso-level semantic embeddings, and macro-level LLM narrative evaluation - and show, across 830 Chinese therapeutic texts spanning depression, anxiety, and trauma, that macro-level evaluation substantially outperforms lexical and embedding features for mental health prediction. This challenges the field's emphasis on word-counting: formal structural features (Labov's story grammar, RST coherence, propositional composition) demonstrate that narrative organization per se carries predictive signal, while clinically-grounded narrative dimensions capture how psychological states are expressed through discourse. Semantic embeddings add minimal independent value but yield incremental gains in multi-level classification. By grounding computational levels in discourse processing theory, this framework identifies macro-structural organization as the primary locus of clinical signal and generates testable hypotheses for intervention design and longitudinal research.

• The paper introduces a hierarchical multi-level framework integrating LIWC-based lexical analysis, semantic embeddings, and LLM-driven narrative evaluation.
• It demonstrates that macro-level narrative features, such as formal structure and clinical dimensions, yield superior predictive performance with notable R² and AUC metrics.
• Findings highlight that narrative organization captures psychological distress more effectively than traditional lexical frequency and local semantic embeddings.

Multi-Level Narrative Evaluation Outperforms Lexical Features for Mental Health

Framework and Hierarchical Model

This paper introduces a principled, theory-driven multi-level analytical architecture for predicting mental health status from therapeutic writing. The model consists of three computational layers, explicitly mapped to text comprehension and production processes: micro-level lexical analysis using LIWC, meso-level semantic coherence via sentence embeddings, and macro-level narrative evaluation with LLMs (GPT-4o). The macro-level layer is further differentiated into formal structural and clinically-grounded narrative dimensions, corresponding to Labov's story grammar, Rhetorical Structure Theory (RST), and clinical features of narrative expression. This hierarchical mapping grounds computational techniques in discourse processing theory, particularly Kintsch and Van Dijk's situation model, textbase, and surface linguistic levels.

Dataset and Methodology

A corpus of 830 Chinese therapeutic writing samples spanning ages 9–50 and diverse clinical/community settings is analyzed. Mental health severity (depression, anxiety, trauma) is measured using standardized psychometric tools (BDI-II, PHQ-9, BAI, GAD-7, RCADS, PTSD symptom scales) with cross-study normalization. The three computational layers extract features via: (1) LIWC for micro-level lexical and psycholinguistic markers; (2) OpenAI embeddings for meso-level semantic coherence (sentence-to-sentence and sentence-to-document similarity); (3) GPT-4o for macro-level evaluation, operationalizing narrative structure (Labov components, RST relations, propositional composition) and clinical narrative dimensions (cognitive bias, distress, affective tone, agentic integration, spatio-temporal grounding). Models are trained and validated on regression and classification tasks using stratified 5-fold cross-validation and evaluated by R², AUC, Balanced Accuracy, Macro-F1, and SHAP-based feature attribution.

Results and Numerical Performance

Macro-level narrative evaluation yields the highest predictive performance, especially in regression tasks—a finding that directly contradicts the traditional emphasis on lexical frequency analysis. The full model (B+L1+L2+L3) produces depression regression $R^2 = 0.332$, anxiety regression $R^2 = 0.235$, depression classification AUC $= 0.699$, anxiety AUC $= 0.718$, and trauma AUC $= 0.676$. Layer 3 alone offers near full-model performance (depression $R^2 = 0.295$, anxiety $R^2 = 0.204$, trauma AUC $= 0.692$), with formal narrative organization and clinically-grounded narrative expression dominating SHAP importance scores. Micro-level lexical features (L1) improve classification incrementally; meso-level semantic coherence embeddings (L2) have negligible standalone utility ($R^2$ near zero, AUC ≤ 0.566). Integration of layers leads to incremental gains in classification but plateauing regression metrics until macro-level features are included.

Condition-specific narrative signatures are observed: temporal disorganization and cognitive composition are discriminative for depression, spatial grounding and uncertainty markers for anxiety, and macro-structural coherence for trauma. Age is a strong covariate but its importance likely reflects sample heterogeneity rather than developmental effects.

Structural and Clinical Implications

The evidence establishes macro-structural discourse features—not lexical or semantic density—as the primary locus of clinical signal in therapeutic writing. Formal properties of narrative (temporal, spatial grounding, compositional balance, Labovian closure) predict symptom severity independent of expressed clinical content. Clinically-grounded dimensions such as cognitive bias and affective tone reflect the narrative organization of psychological distress, extending beyond surface-level symptom markers. LLMs, when used as structured evaluators, offer transparent and externally auditable extraction of these dimensions, bridging qualitative clinical judgment with quantitative modeling.

The minimal impact of semantic embeddings highlights a methodological weakness: embeddings capture only local semantic similarity and lack sequential logic modeling, which leads to ceiling effects in constrained clinical writing corpora.

Methodological Contributions

The multi-level framework ensures theoretical interpretability and operational transparency. Each feature is explicitly mapped to psychological hypotheses, replacing black-box feature extraction. The use of LLMs as prompt-engineered evaluators rather than classifiers enables replicable, JSON-formatted outputs with supporting textual evidence, enhancing analytical auditability. The hierarchical decomposition clarifies the boundaries and clinical relevance of each linguistic level: macro-structural features are robust correlates of psychopathology, superseding micro-level and meso-level information.

Limitations and Future Directions

Sample heterogeneity, cultural and linguistic specificity, cross-sectional design, and the need for human expert validation pose limitations. The prominence of age in feature importance is likely due to sub-sample context rather than developmental trajectory. Longitudinal and ablation studies are necessary to determine causal relationships between narrative fragmentation and distress, and to dissect the independent utility of macro-level structural versus clinical narrative features.

Cultural dependency of narrative conventions must be addressed in future research for broader generalization. Translational impact is contingent upon experimental evidence for intervention efficacy—particularly for scaffolding narrative dimensions (temporal for depression, spatial for anxiety) in therapeutic writing.

Conclusion

This paper demonstrates that macro-level narrative organization, as evaluated by structured LLMs, is the dominant carrier of clinical signal in therapeutic writing for depression, anxiety, and trauma. The findings challenge reliance on lexical frequency and semantic density, identifying narrative structural coherence and clinically-grounded expression as the primary predictive features. The theoretical, methodological, and practical implications suggest that narrative evaluation frameworks grounded in discourse processing theory should guide future AI developments in computational psychiatry, both for diagnosis and personalized intervention design.