---
title: RadGraph Annotation
url: https://www.emergentmind.com/topics/radgraph-annotation
type: topic
---

# RadGraph Annotation

RadGraph annotation is a methodology for systematically structuring and extracting clinical entities and their semantic relations from free-text radiology reports. Designed initially for chest X-ray reports, the framework underpins datasets, benchmark models, and information extraction pipelines widely adopted in medical natural language processing (NLP) research. At its core, RadGraph provides a graph-based formalism mapping unstructured language to normalized, relational clinical facts, facilitating downstream applications ranging from automated report understanding to style-sensitive text generation and disease progression tracking [2106.14463][2308.05046][2310.17811].

## 1. RadGraph Annotation Schema

The RadGraph schema formalizes the essential entities and directed relations comprising the clinical content of radiology reports. Entities and relations are annotated in a way that captures not only observed findings but also their clinical assertions and context.

### Entity Types

- **Anatomy**: Any reference to an anatomic region or substructure (e.g., "lung," "pleura," "mediastinum").
- **Observation:Definitely Present**: Findings or pathological states explicitly marked as present (e.g., "consolidation," "effusion").
- **Observation:Uncertain**: Descriptors reflecting diagnostic uncertainty or differentials (e.g., "possible pneumothorax," "cannot exclude lymphadenopathy").
- **Observation:Definitely Absent**: Findings explicitly negated by the radiologist (e.g., "no pneumothorax," "without focal consolidation").

### Relation Types

- **located_at (Observation → Anatomy)**: Links a clinical observation to its anatomical site (e.g., "pleural effusion" located_at "right lung base").
- **modify**: Associates modifying entities to their targets, restricted to same-type pairs (Anatomy→Anatomy or Observation→Observation); includes degree/descriptor modifiers and spatial/anatomical qualifiers (e.g., "moderate" modify "pleural effusion").
- **suggestive_of (Observation → Observation)**: Encodes inferential relationships in which one observation implies or suggests another (e.g., "ground-glass opacity" suggestive_of "early pneumonia") [2106.14463][2310.17811].

### Span and Nesting

Entities are annotated over continuous spans using an IOB tagging scheme internally, with non-overlapping spans except where full-type containment is necessary and cross-type nesting is disallowed (e.g., Anatomy and Observation remain strictly disjoint).

## 2. Annotation Process and Quality Assurance

Annotation is performed by expert annotators—typically board-certified radiologists—on standardized platforms such as Datasaur.ai. The process includes multiple calibration and adjudication steps to ensure clinical and linguistic consistency.

- **Pilot Phase**: Iterative pilots (~15 reports each), refining schema based on annotator disagreements and edge-case notes.
- **Span Guidelines**: Rigid non-overlapping constraints, with granularity bias: in ambiguous segmentations, annotate smaller spans and resolve semantic relations via explicit edges.
- **Ambiguity Resolution**: Written notes and group calls clarify cases such as device labeling and rare negations.
- **Adjudication**: Dual-annotation for test reports, recording all disagreements to serve as a human agreement benchmark.
- **Inter-Annotator Agreement**: Expressed as Cohen’s κ, reporting values of 0.974 (MIMIC-CXR, entities) and 0.841 (MIMIC-CXR, relations) [2106.14463]; for extended schemas, pairwise Cohen’s κ values reach 0.9943–0.9963 [2308.05046].
- **Edge Labeling**: Relations are annotated only when unambiguous; prominence is given to direct, syntactically evident edges.

## 3. Dataset Composition and Schema Evolution

The RadGraph dataset includes manually annotated and model-inferred corpora at scale, with subsequent releases (e.g., RadGraph2) introducing hierarchical labeling and new clinical event types.

| Dataset           | Reports | Entities          | Relations          | Notes                         |
|-------------------|---------|-------------------|--------------------|-------------------------------|
| RadGraph-dev      | 500     | 14,579            | 10,889             | Manually labeled, MIMIC-CXR   |
| RadGraph-test     | 100     | ~1,300–1,500/annot. | ~900–1,100/annot. | Doubly labeled, MIMIC & CheXpert  |
| Inference dataset | 220,763 | ~6 million        | ~4 million         | Auto-labeled, MIMIC-CXR       |
| RadGraph2         | 800     | 23,457            | 17,373             | Explicit "Change" entities    |

The entity taxonomy in RadGraph2 expands the schema to annotate disease and device progression: hierarchical entity types (e.g., "CHAN-CON-WOR" for condition worsening, "CHAN-DEV-AP" for new device), with explicit multi-level probabilities and a tree-structured loss during model training [2308.05046].

## 4. Benchmark Models and Evaluation Metrics

RadGraph Benchmark employs the DYGIE++ framework for joint entity and relation extraction. Architectural details include:

- **Encoder**: PubMedBERT or other domain-pretrained BERT variants.
- **Span Representation**: Candidate spans up to length 3 tokens.
- **Joint Modeling**: Simultaneous prediction of entity types and directed graph edges.
- **Hierarchical Loss (RadGraph2)**: Conditional probability chains over entity taxonomy, optimizing both leaf and internal nodes [2308.05046].

### Evaluation

All metrics use strict span and type matching for entities, and strict endpoint and type matching for relations.

| Task                    | Dataset     | Micro-F1 (Entities) | Micro-F1 (Relations) |
|-------------------------|-------------|---------------------|----------------------|
| RadGraph Benchmark      | MIMIC-CXR   | 0.94                | 0.82                 |
|                         | CheXpert    | 0.91                | 0.73                 |
| Human benchmark (κ)     | MIMIC-CXR   | 0.99                | 0.95                 |
|                         | CheXpert    | 0.93                | 0.75                 |
| RadGraph2 - HGIE (rel.) | MIMIC-CXR   | —                   | ~0.879               |

Performance is summarized as micro- and macro-F1 across classes, with error analysis referenced to the doubly labeled test set [2106.14463][2308.05046][2310.17811].

## 5. Practical Annotation Examples

Annotation is concretized as token-span selection followed by relation arc drawing. For instance:

Report: “Moderate right pleural effusion with adjacent consolidation; no pneumothorax.”

- Entities:
    1. "Moderate" → Observation:Definitely Present
    2. "right"    → Anatomy
    3. "pleural effusion" → Observation:Definitely Present
    4. "adjacent" → Observation:Definitely Present
    5. "consolidation" → Observation:Definitely Present
    6. "no pneumothorax" → Observation:Definitely Absent

- Relations:
    - modify("Moderate"→"pleural effusion")
    - located_at("pleural effusion"→"right")
    - modify("adjacent"→"consolidation")
    - "no pneumothorax" stands as an isolated negative finding

Extended RadGraph2 schema additionally labels temporal and comparative changes, e.g.,
- "new" → CHAN-CON-AP
- "persists" → CHAN-NC
- "increased" → CHAN-CON-WOR, with modify edges to the changed entity [2106.14463][2308.05046].

## 6. Applications and Extensions

RadGraph annotation underlies various downstream applications:

- **Large-scale automatic labeling**: Auto-annotation of hundreds of thousands of chest radiograph reports via trained RadGraph Benchmark or HGIE models facilitates secondary research in medical NLP and imaging [2106.14463][2308.05046].
- **Style-aware report generation**: RadGraph graphs can be extracted from images by a vision-language encoder-decoder, then verbalized into free text by prompting large language models, enabling disentanglement of report content from radiologist style [2310.17811].
- **Progression tracking and device monitoring**: RadGraph2 enables automated detection and linkage of disease or device evolution across serial studies, via a hierarchical change taxonomy [2308.05046].
- **Evaluation metrics**: Graph F1 scores (node and edge matching) serve as objective measures of model content fidelity in both extraction and generation settings, with human annotation agreement (Cohen’s κ > 0.80) validating reproducibility [2106.14463][2310.17811].

## 7. Significance and Future Directions

RadGraph annotation provides a clinically precise, machine-actionable representation of radiological findings, supporting reproducible research standards and interoperability. Its iterative, radiologist-in-the-loop protocol, strict consensus metrics, and extensible schema allow adaptation to new modalities (e.g., CT, MRI), multi-institutional data, and integration with vision-language learning frameworks.

The introduction of hierarchical schemas in RadGraph2 and the coupling with large language models for style-aware text generation represent substantive advances. Ongoing challenges include maintaining high inter-annotator agreement for subtle relation types, adapting schemas to additional imaging contexts, and optimizing for both extraction fidelity and generation fluency [2308.05046][2310.17811].

Source: https://www.emergentmind.com/topics/radgraph-annotation