German CheXpert: Automated CXR Labeling

• German CheXpert is a framework for automatically extracting structured labels from German chest X-ray reports using rule-based NLP and deep learning augmentation.
• It employs a three-stage pipeline—mention extraction, classification with a modified NegEx algorithm, and mention aggregation—to ensure accurate label prediction.
• The system integrates a secure multi-reader annotation tool and demonstrates high performance, with deep learning models achieving an AUC of 0.939 for pneumothorax detection.

German CheXpert is a computational framework for the automatic extraction of structured labels from German-language chest X-ray (CXR) radiology reports. Modeled after the original English-language CheXpert pipeline, its purpose is to generate high-throughput, clinically meaningful annotation for CXR image-text datasets, thereby enabling scalable development and evaluation of deep learning models for medical image classification in German clinical settings. The pipeline leverages rule-based NLP tailored to German medical terminology, and can be augmented with deep learning for multi-finding label prediction and transfer learning approaches. German CheXpert provides an integrated solution, including a secure web-based multi-reader annotation tool, for creating gold-standard datasets, iteratively expanding knowledge bases, and powering weak supervision of deep neural networks (Wollek et al., 2023, Wollek et al., 2023).

1. Pipeline Architecture and Label Extraction Logic

German CheXpert adapts CheXpert’s original three-stage pipeline—mention extraction, mention classification, and mention aggregation—to German clinical language:

1. Mention Extraction: Input reports are processed using class-specific phrase lists for 14 target findings. Initial lists are compiled by radiologists and expanded via iterative annotation. Language-aware, case-insensitive, token-based regex searches identify candidate phrases for each class.
2. Mention Classification: The system uses a modified German NegEx algorithm to detect negation and uncertainty cues. Lists of pre- and post-negation signals (e.g., “kein,” “nicht”) and uncertainty expressions (e.g., “unwahrscheinlich,” “verdacht”) are maintained. For each mention, a sliding window of ±k tokens around the match is scanned. The assignment procedure is:
   • Positive if no cue is within span,
   • Negative if a negation cue appears,
   • Uncertain if an uncertainty cue appears.
3. Mention Aggregation: For each observation $i$, all mention-level classifications $M_{i,j}$ are aggregated to yield the class label $L_i$:

$$L_{i} = \begin{cases} \text{positive}, &\text{if}~ \exists j: M_{i, j} = \text{positive} \ \text{uncertain}, &\text{if}~ (\nexists j: M_{i,j} = \text{positive}) \wedge (\exists j: M_{i,j} = \text{uncertain}) \ \text{negative}, &\text{otherwise.} \end{cases}$$

The “no finding” label is defaulted to negative except when any other class (excluding “support devices”) is positive or uncertain.

No neural modules or learnable loss functions are embedded in the rule-based labeler; model-driven components are downstream only.

2. Datasets, Preprocessing, and Annotation Interface

Core Datasets

• DS 1 (Gold Standard Annotations): 1,086 German thoracic CXR reports (2020–2021), annotated by radiologists for 14 CheXpert classes. 186 reports used for phrase curation; the remainder for primary annotation and evaluation.
• DS 2 (Pneumothorax Classification): 6,434 internal CXR/report pairs; 1,568 images manually reviewed and labeled positive for pneumothorax.

Web Annotation Interface

A secure, locally deployed annotation tool accelerates radiologist labeling and phrase list expansion:

• Dual-pane layout: left side displays report text and view position, right side lists all 14 findings with radio-button inputs for label assignment (positive, negative, uncertain, none).
• “ADD NEW” function supports interactive phrase list augmentation.
• Phrases auto-recognized but marked “none,” and classes assigned labels without known phrases, are flagged for radiologist validation.
• Supports multi-user sessions and immediate update of phrase knowledge base.

Preprocessing and Augmentation

• Text: Lowercased, sentence-split, token-based regular expressions for mention search.
• Images: Resized to $224 \times 224$, normalized to ImageNet mean/std; augmented with 10-crop (five crops per image plus horizontal flips) for robust training.

3. Rule-Based Labeler and Deep Learning Augmentation

Rule-Based Components

• Phrase Lists and Pattern Matching: For each finding, extensive dictionaries ($D_k$) of German keywords and synonyms are maintained and dynamically refined.
• Negation/Uncertainty Strategy: Adapted NegEx-based windowed search for cue detection; all label assignments are fully deterministic.
• Implementation: Highly parallelized, enabling annotation of 100 reports in approximately 1.84 seconds on standard hardware.

Deep Learning Model

A weakly-supervised German BERT model (“bert-base-german-cased,” $H=768$, 12 layers, 12 heads, 512-token limit) is pre-trained on over 66,000 rule-labeled reports (DS₀) for all 14 tasks, then fine-tuned on up to 1,091 manually labeled reports (DS₁). The network features 14 task-specific linear heads (4-way softmax for most findings, binary for “no finding”). Optimization uses AdamW, typical learning rates are $2\times10^{-5}$ for BERT and $3\times10^{-3}$ for image classifiers.

Downstream Imaging Model

• Backbone: DenseNet-121 pretrained on ImageNet.
• Output: Single neuron with sigmoid activation for binary classification (e.g., pneumothorax).
• Objective: Binary cross-entropy loss.
• Augmentation: See above for 10-crop strategy.

4. Evaluation Methodology and Performance Metrics

Tasks and Metrics

The system is evaluated on the following axes (primarily on DS₁):

• Mention Extraction: Detects any finding mention, lumping positive, negative, uncertain as “detected.”
• Negation Detection: Reports correct identification of explicitly negated findings.
• Uncertainty Detection: Isolates the model’s ability to recognize uncertain language.
• Binary Sensitivity/Specificity: After mapping uncertain→positive and none/blank→negative.

Metrics include F1 score, sensitivity (recall), specificity, accuracy, and area-under-ROC (AUC), with bootstrapped 95% confidence intervals for classification tasks.

Task	Rule-Based (RB)	Deep Learning (DL)	Manual Reference
Mention F1	0.91	0.94	–
Negation F1	0.82	0.89	–
Uncertainty F1	0.51	0.61	–
Pneumothorax AUC (DS2)	0.858 [0.832–0.882]	0.939 [0.925–0.952]	0.934 [0.918–0.949]

The deep learning model consistently outperforms the rule-based pipeline on mention extraction, negation, and uncertainty F1, and approaches manual-expert performance on downstream pneumothorax classification (Wollek et al., 2023, Wollek et al., 2023).

5. Strengths, Limitations, and Domain Suitability

Contributions

• Provides a reliable, transparent, and rapidly updatable rule-based labeler for German CXR reports across 14 critical findings.
• Delivers a mature annotation workflow that consolidates gold-standard creation and weak supervision.
• Demonstrates, for the first time at scale for German medical text, that deep learning models trained on weakly labeled data can outperform models trained with public benchmark data, and nearly match expert-annotated gold standards.

Limitations

• Only pneumothorax was validated end-to-end with full imaging model retraining; other findings require further downstream evaluation.
• Rule-based methods are bounded by the coverage of explicit synonyms and expression patterns; they cannot encode deep semantic or paraphrastic variability (“Herz normgröße”) and may yield inconsistent labels for ill-defined or rare classes.
• Multi-study reports can produce label–image misalignments.
• Registration of uncertainty remains challenging, especially for rare or hedged cases.

This suggests that while the rule-based layer achieves high throughput, it benefits from continual validation and augmentation. A plausible implication is that hybridization with transformer-based representations will further enhance label coverage and precision for semantically diverse clinical texts.

6. Implications, Extensions, and Future Directions

The German CheXpert approach provides a scalable solution for large-scale, weakly supervised dataset curation and annotation in German-speaking clinical contexts. The adoption of transformer-based clinical LLMs—such as BERT variants—promises to further alleviate synonymy and semantic drift limitations inherent to rule-based systems. Ongoing and prospective efforts include:

• Integrating transformer models directly into the labeling pipeline for improved synonym and morphology handling.
• Refining and localizing CheXpert’s ontology and label definitions for German radiological reporting conventions.
• Extending evaluation to all 14 CheXpert classes with full downstream imaging model retraining and multi-institutional validation.
• Fostering multi-center, multi-language annotation and classification pipelines for international dataset harmonization (Wollek et al., 2023, Wollek et al., 2023).

In summary, German CheXpert delivers a high-throughput, modular, and extensible framework for extracting structured, clinically meaningful labels from German radiology reports, facilitating both rapid data curation and robust deep learning model development for chest X-ray interpretation.