- The paper introduces a three-stage pipeline combining high-recall zero-shot LLM screening, structured evidence extraction, and supervised classification, achieving AUPRCs of 0.816–0.887 across internal, external, and prospective hospital datasets.
- Evidence augmentation improved AUPRC from 0.859 to 0.900, while logistic regression outperformed fine-tuned BERT models and provided a more interpretable, retrainable final classifier.
- The approach transferred without site-specific retraining, extracted primary self-harm methods with 95.0% clinician agreement, and highlighted calibration and documentation-style differences as key deployment challenges.
Motivation and problem setting
Self-harm surveillance systems that rely on ICD-10 diagnostic codes suffer from low sensitivity (13.8%–65%), reflecting delayed code assignment, coding-practice variation, and the lossy translation of clinical narratives into structured codes. Free-text ED triage notes, recorded at first contact, offer a richer and earlier signal. Prior NLP work has demonstrated strong binary detection of self-harm from triage notes (e.g., an AUPRC of 0.84 with gradient boosting at a single site), but two gaps remain: cross-hospital transferability is underexplored, with models developed at one hospital often degrading elsewhere, and most systems produce only binary labels without structured supporting evidence or method-level detail.
This paper addresses both gaps with a three-stage pipeline applied to triage notes from three Victorian hospitals: (1) zero-shot LLM screening for high recall, (2) zero-shot LLM extraction of structured self-harm evidence (act, injury, method, timing, intent, plus keywords and summary), and (3) a supervised classifier combining the raw note with Stage 2 outputs. Development used Royal Melbourne Hospital (RMH) data from 2012–2017; evaluation covered RMH held-out and prospective (2018–2022) sets, external validation at Latrobe Regional Health (LRH) and Sunshine Hospital (SunH), and prospective external validation at LRH.
Data and annotation
The corpus comprises roughly 1.4 million de-identified triage notes across three sites, with self-harm prevalence between 1.06% and 1.68%. Labels were assigned by trained suicide-prevention researchers with inter-annotator agreement of κ=0.91. Triage notes are short (median 21 tokens at RMH; 43–47 tokens at the external sites), which motivates the evidence-augmentation design: brief, abbreviated text leaves little context for a standalone classifier.
The RMH development set was divided into four label-stratified chunks to support sequential prompt development and prevent leakage between stages; prompts were fixed on Chunks 1–2 before any test-set contact. Both LLM stages used gpt-oss-20b deployed locally on a single A100 80GB GPU, with self-consistency majority voting over three runs (Stage 1) and five runs (Stage 2).
Stage 1 screening achieved recall of 0.967–0.984 across all sites and periods, reducing each corpus to only 4.9%–6.5% of notes while raising prevalence to 21.2%–26.7%. This reduction is what makes repeated Stage 2 inference computationally tractable at scale.
Stage 2 alone was a poor binary classifier: recall was high (0.933–0.961) but precision low (0.389–0.497), yielding F1 scores around 0.55–0.65. The authors are explicit that zero-shot LLM classification produces too many false positives for direct surveillance use; its value lies in the structured intermediate outputs, which feed a supervised Stage 3 classifier that learns the decision boundary from labelled data. Combined Stages 1+2 retained recall of 0.902–0.946 on full corpora.
Evidence augmentation and model selection
Feature ablation on RMH Chunk 4 showed that augmenting TF-IDF + MiniLM note representations with Stage 2 outputs improved AUPRC from 0.859 to 0.900. Notably, LLM free-text reasoning carried more discriminative signal than extracted evidence spans (AUPRC 0.897 vs. 0.879 when added individually). An exhaustive ablation over all 127 indicator subsets found that using all indicators maximised AUPRC (0.900), while the method + summary combination achieved the best F1 (0.841) at nearly identical AUPRC (0.896), and was selected as the final configuration.
A further notable finding is that logistic regression outperformed all fine-tuned BERT variants (BERT-base, ClinicalBERT, SciBERT) given these features — e.g., LR AUPRC 0.896 vs. ClinicalBERT 0.878. The authors attribute this to the feature space already containing explicit, standardised evidence signals, leaving little for a deeper contextual model to add, and note the practical benefit of an interpretable, easily retrainable final stage.
Trained solely on RMH Chunk 3 with no site-specific adaptation or threshold recalibration, the full pipeline achieved:
| Test set |
AUPRC |
F1 |
Precision |
Recall |
| RMH test (in-distribution) |
0.887 |
0.827 |
0.840 |
0.815 |
| RMH prospective (2018–2022) |
0.881 |
0.823 |
0.853 |
0.800 |
| LRH test (external) |
0.884 |
0.826 |
0.813 |
0.838 |
| LRH prospective (external) |
0.879 |
0.817 |
0.801 |
0.836 |
| SunH test (external) |
0.816 |
0.772 |
0.710 |
0.846 |
Quarterly analysis showed no meaningful degradation over time, and the F1 threshold selected on RMH transferred without retuning. Against baselines trained on the full pre-screening corpus, the proposed approach achieved the highest AUPRC everywhere; gains were largest at external sites (SunH: 0.816 vs. 0.737 for ClinicalBERT and 0.720 for note-only LR), suggesting the staged design improves robustness under both hospital and temporal shift.
Same-hospital upper-bound comparisons clarify where transfer cost arises. At LRH, cross-hospital performance was essentially equivalent to local training. At SunH, AUPRCs were similar (0.816 cross-hospital vs. 0.824 same-hospital), but the operating point shifted: precision dropped (0.710 vs. 0.805) while recall rose (0.846 vs. 0.771). The authors conclude the lower SunH performance reflects documentation-style and demographic differences rather than transfer failure per se, and that threshold recalibration may suffice to tailor the precision–recall trade-off locally.
Computational cost
At RMH volumes of 260–300 presentations/day, the full pipeline requires approximately 2.0–2.3 GPU-minutes per day on a single A100, with estimated energy use of roughly 30 Wh (Stage 1) and 219 Wh per 1,000 notes processed in Stages 1 and 2 respectively, and negligible cost for the CPU-based Stage 3. The staged design is thus deployable at realistic ED throughput.
Method-level surveillance and clinical validation
Beyond binary classification, the approach extracts the primary self-harm method from a 15-category list. On 1,016 clinician-annotated RMH cases (annotated by a clinical psychologist blinded to model outputs), the LLM-extracted method matched the clinician-assigned primary method in 95.0% of cases, with weighted F1 of 0.95 but macro F1 of only 0.76 due to rare categories. Overdosing/self-poisoning dominated (64.3%) followed by cutting (23.8%). Frequent categories performed near-ceiling (self-burning F1 = 1.00; overdose 0.98; hanging and jumping 0.97), whereas "method not reported" and "other" were frequently confused with each other or misassigned to overdosing (F1 = 0.38 and 0.40). Only the method field was clinically validated; the evidence and reasoning texts and remaining indicators were not independently reviewed.
Error analysis
Confusion matrices, probability distributions, and calibration curves identified SunH as carrying a heavier false-positive burden, with negative-class probabilities showing a right tail absent at RMH and LRH. Uncalibrated probabilities misaligned with observed positive rates at all sites, worst at SunH; Platt scaling fitted on RMH transferred adequately to LRH but poorly to SunH, while per-site isotonic calibration improved alignment everywhere. Calibration was not incorporated into the final model, so reported probabilities should be treated as uncalibrated.
Qualitative review of high-confidence false positives revealed two patterns: possible reference-label disagreement (e.g., overdoses with documented suicidal intent labelled negative in the reference data) and annotation-rule exclusions the model could not know (self-harm described as historical rather than current presentation; suicidal ideation without an act; overdoses assigned to a mutually exclusive alcohol/other-drug category by GCS criteria). Near-threshold errors were mostly genuine negatives such as recreational drug use. Some fraction of apparent model error therefore reflects ambiguity in the ground truth itself.
Limitations and open questions
The paper concedes several constraints directly. Evaluation covers three hospitals within a single Australian state, so generalisation to other health systems remains untested. All LLM stages rely on a single model (gpt-oss-20b); whether smaller or more efficient LLMs would perform comparably is unresolved. Only the method output received clinical validation. LLM inference carries non-trivial GPU, energy, and carbon costs despite staging. Finally, this is a retrospective study of surveillance potential, not a deployment study; prospective operational evaluation, including whether site-specific recalibration suffices at sites like SunH, remains open.
Conclusion
This study demonstrates that a staged architecture — high-recall zero-shot LLM screening, zero-shot structured evidence extraction, and a simple supervised classifier augmented with LLM-derived evidence and reasoning — can detect self-harm in ED triage notes with AUPRCs of 0.816–0.887 across internal, external, and prospective evaluations, largely without site-specific retraining. The evidence-augmentation step outperforms both note-only baselines and fine-tuned BERT classifiers, and the extracted primary-method labels agree with blinded clinician annotation at 95%, enabling method-specific surveillance beyond binary case identification. The main residual challenges are the precision shift observed at one external site, uncalibrated predicted probabilities, and validation of the remaining extracted indicators.