---
title: 'HSMP-BERT: Hierarchical Clinical Extraction'
url: https://www.emergentmind.com/topics/hierarchical-structured-matching-prediction-bert-hsmp-bert
type: topic
---

# HSMP-BERT: Hierarchical Clinical Extraction

Hierarchical Structured Matching Prediction BERT (HSMP-BERT) is a prompt-based model for fine-grained extraction of structured clinical information from Hebrew free-text radiology reports in Crohn’s disease. It was developed for a low-resource-language setting in which multi-organ findings are sparsely represented, and it combines a Hebrew RoBERTa backbone, Section Matching Prediction (SMP) pretraining over report sections, prompt-based organ–finding queries, and hierarchical tree-based inference. In the reported administrative database study, the model was applied to 9,683 reports from Crohn’s patients imaged during 2010–2023 across Israeli providers, with a radiologist-annotated subset used to evaluate extraction into 90 structured labels per subject [2509.04519].

## 1. Clinical setting and label space

The stated goal of HSMP-BERT is to structure free-text Hebrew radiology reports on Crohn’s disease into granular organ–finding labels such as wall thickening in a specific gastrointestinal organ. The source corpus comprises 9,683 radiology reports, and a subset of 512 reports was manually annotated by radiologists for findings across six gastrointestinal organs and 15 pathologies, yielding 90 structured labels per subject [2509.04519].

The extraction problem is explicitly multi-label. Each report is assessed for all organ–finding pairs, and the reported analysis retained only two classes: presence or absence of a finding. In that formulation, “organ not visible” and “resected” were collapsed into the negative class. For evaluation, only organ–finding pairs with more than 15 positive samples were considered, resulting in 24 final targets [2509.04519].

This setup places HSMP-BERT at the intersection of clinical NLP, low-resource-language modeling, and structured information extraction. A plausible implication is that the model is designed not merely to classify whole reports, but to support downstream cohort analysis at the level of clinically meaningful organ–pathology combinations.

## 2. Backbone, section-aware pretraining, and prompt formulation

The reported base model is Hebrew RoBERTa (HeRo), which was further domain-adaptively pretrained on the radiology corpus. HSMP-BERT’s section-aware component is Section Matching Prediction (SMP), introduced to teach the model the logical structure connecting the “Findings” and “Impression” sections in radiology reports [2509.04519].

The SMP objective uses paired Findings and Impression sections and predicts whether they match. The prediction head is defined as
$$
q_\mathcal{M}(n_k \mid x_i) = \mathrm{softmax}(W_{smp} h_{[CLS]} + b_{smp}),
$$
where \(n_k \in \{\text{Match}, \text{NotMatch}\}\) and \(h_{[CLS]}\) is the pooled representation. The reported loss is binary cross-entropy [2509.04519].

Prompt-based extraction is then built on top of this section-aware representation. Each structured label is verbalized as a clinical-style natural language prompt. The prompt template is
$$
\mathcal{T}(x_i) = [CLS]~x^F_i~[SEP]~\textrm{``There is \{finding\} in the \{organ\}.''}~[EOS].
$$
For each report \(x^F_i\) and label \(y_j\), the model scores compatibility between report and prompt using prompt-based inference, without requiring explicit output layers for each class. During SMP-tuning, pairs of \((x^F_i,\text{prompt})\) are labeled 1 when correct and 0 otherwise, again optimizing a binary cross-entropy loss [2509.04519].

A key technical feature is that report structure is not treated as incidental metadata. Instead, the alignment between Findings and Impression is used as a pretraining signal, and the organ–finding schema is expressed through prompts rather than through a conventional fixed-output multi-label head.

## 3. Hierarchical inference and structured prediction

HSMP-BERT’s hierarchy appears in inference as a tree-structured query-routing procedure. The reported sequence is scan-level, then organ-level, then finding-level: first, “Is the scan normal?”; second, “Is there any abnormality in the ileum?”; third, “Is there wall thickening in the ileum?” [2509.04519].

The routing mechanism is conditional. If a branch is negative, its child prompts are skipped. This tree-based querying reduces unnecessary model evaluations and is the basis of the method’s efficiency claims. The paper reports that hierarchical inference cuts runtime by \(5.1\times\) versus traditional inference, with average per-report time reduced from 1.77 seconds under flat inference to 0.35 seconds under hierarchical inference. Total runtime is reported as 4:46:00 for flat inference versus 0:55:36 for hierarchical inference; BERT calls are reduced from 2,180,436 to 395,750; and processed tokens are reduced from 574 M to 114 M [2509.04519].

This design distinguishes HSMP-BERT from flat prompt evaluation over the full Cartesian product of labels. A common misconception is to equate any “hierarchical” BERT model with hierarchy in the encoder itself. In HSMP-BERT, the reported hierarchy is operationalized through tree-based querying over clinically ordered decisions, rather than through hyperbolic geometry or layer-wise classifier attachment.

## 4. Evaluation protocol and reported performance

The reported split is multilabel-stratified, with 66% train+validation and 33% test, preserving label prevalence. Performance was evaluated using accuracy, F1, Cohen’s \(\kappa\), AUC, PPV, NPV, and recall; the detailed evaluation framework also lists balanced accuracy and precision [2509.04519].

On the 24 organ–finding combinations with more than 15 positives, HSMP-BERT achieved mean F1 \(0.83 \pm 0.08\) and Cohen’s \(\kappa\) \(0.65 \pm 0.17\). The detailed results also report accuracy \(0.94 \pm 0.04\), PPV \(0.71 \pm 0.16\), NPV \(0.97 \pm 0.02\), and balanced accuracy \(0.83 \pm 0.10\). The paper states that performance was consistent across both common findings, such as ileum wall thickening, and rare findings, such as sigmoid comb sign [2509.04519].

The principal baselines were SMP zero-shot and standard fine-tuning. In the abstract, HSMP-BERT is reported to outperform the SMP zero-shot baseline, which obtained F1 \(0.49 \pm 0.07\) and \(\kappa\) \(0.06 \pm 0.07\), and standard fine-tuning, which obtained F1 \(0.30 \pm 0.27\) and \(\kappa\) \(0.27 \pm 0.34\), with paired \(t\)-test \(p < 10^{-7}\) [2509.04519]. The detailed comparison further reports \(\Delta\)F1 \(= 0.33\) and \(\Delta \kappa = 0.59\) versus SMP zero-shot, with \(p < 10^{-15}\), and \(\Delta\)F1 \(= 0.53\) and \(\Delta \kappa = 0.38\) versus standard fine-tuning, with \(p < 10^{-8}\) [2509.04519].

The paper also provides single-label examples. Ileum wall thickness reached F1 \(= 0.93\) and \(\kappa = 0.86\), while rectum wall thickness, described as rare with 6% prevalence, reached F1 \(= 0.88\), \(\kappa = 0.77\), and NPV \(= 0.9\) [2509.04519].

## 5. Population-level application in Crohn’s disease imaging

After evaluation on expert annotations, HSMP-BERT was applied to the full corpus of 9,683 reports from 7,389 patients, with stratification by sex and age. The reported large-scale output revealed associations among ileal wall thickening, stenosis, and pre-stenotic dilatation, as well as age- and sex-specific trends in inflammatory findings [2509.04519].

The ileum was reported as the most affected organ, more in males than females and more in adults than pediatric cases. The detailed prevalence figures are 57.7% in males versus 44.8% in females, and 52.6% in adults versus 45.1% in pediatric cases. Wall thickening and wall enhancement were the most common pathologies. Inflammation was higher in males than females, with reported frequencies of 48.4% versus 38.2%, and was also higher in adults. Stenosis and pre-stenotic dilatation were more frequent in adults, whereas DWI abnormalities were more frequent in pediatric cases [2509.04519].

The paper states that the model output on the full corpus closely mirrors statistics in the expert-labeled set, supporting validity and deployment suitability. In this sense, HSMP-BERT is presented not only as a per-report extraction model but also as a mechanism for population-level analysis of Crohn’s disease imaging patterns [2509.04519].

## 6. Relation to other hierarchical BERT formulations

HSMP-BERT belongs to a broader family of work that introduces hierarchy into BERT-like systems, but adjacent studies place the hierarchy at different levels of the modeling stack. In “Layer-wise Guided Training for BERT: Learning Incrementally Refined Document Representations,” different BERT layers are guided to predict labels at different depths of a label hierarchy for large-scale multilabel text classification, with the reported “last-six” configuration assigning classifiers \(f_7\) to \(f_{12}\) to hierarchy levels \(L_1\) to \(L_6\) and improving both classification results and parameter utilization [2010.05763].

In “Hierarchical Multitask Learning Approach for BERT,” hierarchy is introduced during pre-training by solving tasks at different transformer layers instead of only at the last layer, transferring information from NSP to masked LM, and adding the bigram shift task to encode word order information. The reported conclusion is that imposing a task hierarchy in pre-training improves the performance of embeddings [2011.04451].

In “A Sentence-level Hierarchical BERT Model for Document Classification with Limited Labelled Data,” hierarchy is document-internal: a token-level RoBERTa encoder produces sentence vectors, and a sentence-level BERT encoder models document structure. That model is targeted at long-document classification with limited labeled data rather than structured clinical extraction [2106.06738]. In “Language Models as Hierarchy Encoders,” hierarchy is made explicit in the embedding geometry by placing encoder outputs inside a Poincaré ball and optimizing hyperbolic clustering and centripetal losses; that work focuses on subsumption and transitive inference rather than report-to-label extraction [2401.11374].

These comparisons clarify the specific meaning of hierarchy in HSMP-BERT. Its reported contribution is not a generic hierarchy-aware encoder, but a combination of section-aware pretraining, prompt-based label verbalization, and scan-to-organ-to-finding inference routing for structured extraction from Hebrew radiology text [2509.04519].

Source: https://www.emergentmind.com/topics/hierarchical-structured-matching-prediction-bert-hsmp-bert