REVTAF: Retrieval-Enhanced Radiology Reporting

• REVTAF is a radiology report generation framework that integrates hyperbolic retrieval and cross-modal fusion to address class imbalance and weak supervision.
• The Learnable Retrieval Enhancer retrieves semantically relevant reference reports using hyperbolic embeddings and ranking-based supervision, improving performance on rare findings.
• The Fine-grained Visual–Text Alignment and Fusion module employs optimal transport-based cross-attention and consistency constraints to enhance clinical accuracy and natural-language generation.

Searching arXiv for papers related to “REVTAF” and its expansions to ground the article in the relevant literature. REVTAF denotes the “Learnable Retrieval Enhanced Visual-Text Alignment and Fusion” framework for radiology report generation, introduced to address two coupled limitations of automated chest X-ray reporting: long-tailed disease distributions and insufficient cross-modal fusion under weak image–report level supervision. The framework combines a Learnable Retrieval Enhancer (LRE), which retrieves semantically relevant reference reports using hyperbolic representations and ranking-based supervision, with a Fine-grained Visual–Text Alignment and Fusion module (FVTAF), which integrates global and local textual prompts through optimal transport-based cross-attention and a consistency constraint across attention maps. In reported experiments, REVTAF improves both natural-language generation and clinical efficacy metrics on MIMIC-CXR and IU X-Ray, and is presented as outperforming several state-of-the-art baselines and mainstream multimodal LLMs in the specific task setting (Zhou et al., 10 Jul 2025).

1. Definition and problem setting

REVTAF is a radiology report generation framework designed for paired medical images and reports, with an emphasis on chest X-ray datasets such as MIMIC-CXR and IU X-Ray (Zhou et al., 10 Jul 2025). Its stated purpose is to mitigate two issues that are structurally prominent in this domain.

The first issue is class imbalance. Radiology corpora are described as long-tailed, with normal findings dominating and many abnormalities appearing rarely. This biases conventional encoder–decoder report generators toward frequent normal patterns and weakens their ability to describe subtle or rare abnormalities accurately. The second issue is insufficient cross-modal fusion under weak supervision. Because the training signal usually consists only of image–report pairs rather than region–sentence correspondences, learned alignment between visual and textual content tends to be coarse, and decoders may incorporate irrelevant textual information into the generation process (Zhou et al., 10 Jul 2025).

REVTAF addresses these limitations by coupling retrieval with structured fusion. The LRE adaptively retrieves a semantically related reference report, forming Global Reference Prompts (GRPs), while MedKLIP-derived entity prompts provide Local Reference Prompts (LRPs). FVTAF then aligns and fuses the visual representation with both prompt types. This design is intended to improve performance particularly on underrepresented, tail-class inputs, while preserving finer-grained clinical alignment (Zhou et al., 10 Jul 2025).

A plausible implication is that REVTAF treats retrieval not as an external memory added after visual encoding, but as a representation-shaping mechanism that conditions the subsequent alignment and decoding stages.

2. System architecture and data flow

REVTAF uses a staged multimodal pipeline with distinct visual, textual, retrieval, fusion, and decoding components (Zhou et al., 10 Jul 2025). The visual encoder is a ResNet-101 pretrained on ImageNet, which maps an input image $I \in \mathbb{R}^{H \times W}$ to visual features

$$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$

Two textual prompt sources are then constructed. The first source is local, entity-specific prompting. For $M=75$ predefined entities, MedKLIP encodes triplets of the form $(\text{entity}_m,\text{position}_m,\text{exist}_m)$ into

$F_t^l \in \mathbb{R}^{M \times d_l},$

with $$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$ (Zhou et al., 10 Jul 2025). The second source is global, report-level prompting. LRE retrieves a reference report $R_{\text{ref}}$ from a database $\mathrm{DB}=\{R_1,\dots,R_{N_D}\}$ and encodes its sentences via MedKLIP into

$$F_t^g \in \mathbb{R}^{N \times d_g} = f_{\text{medklip}}(R_{\text{ref}}),$$

where $N$ is the maximum sentence count in the database (Zhou et al., 10 Jul 2025).

FVTAF applies two cross-attention branches, one for GRPs and one for LRPs, producing fused features and cross-attention maps:

$$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$0

for the global branch, and analogously $$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$1 for the local branch, where $$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$2, $$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$3, and $$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$4 (Zhou et al., 10 Jul 2025). Each branch then uses residual projection:

$$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$5

and similarly for $$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$6. The final fused representation is

$$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$7

The decoder $$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$8 generates the report from the fused features and an auxiliary disease-prompt branch $$F_v \in \mathbb{R}^{H_v \times W_v \times d_v} = f_e(I).$$9, following PromptMRG:

$M=75$0

During training, LRE uses intra-batch label-based supervision derived from CheXbert, FVTAF computes the multi-source cross-attention maps and their consistency loss, and the decoder is trained with token-level cross-entropy. During inference, LRE performs nearest-neighbor retrieval in learned hyperbolic space, the retrieved report becomes the source of GRPs, and the decoder generates the final report from the fused representation (Zhou et al., 10 Jul 2025).

3. Learnable Retrieval Enhancer

The Learnable Retrieval Enhancer is the component primarily responsible for retrieval quality and tail-class support (Zhou et al., 10 Jul 2025). Its central premise is that anatomical and clinical semantics are hierarchical, and that hyperbolic geometry is a better fit for such hierarchies than Euclidean geometry.

REVTAF adopts the Poincaré ball model. For curvature $M=75$1, the $M=75$2-dimensional ball is

$M=75$3

The framework uses Möbius addition

$M=75$4

and hyperbolic distance in the Poincaré ball

$M=75$5

with an equivalent form for $M=75$6 also given in the paper description (Zhou et al., 10 Jul 2025).

MedKLIP produces entity-specific logits $M=75$7 for a batch of size $M=75$8, and a Hyperbolic Neural Network maps them into hyperbolic embeddings

$M=75$9

with $(\text{entity}_m,\text{position}_m,\text{exist}_m)$0 in the implementation (Zhou et al., 10 Jul 2025). These embeddings are trained using an intra-batch ranking objective. Each report is mapped by CheXbert into $(\text{entity}_m,\text{position}_m,\text{exist}_m)$1 disease categories with four statuses, yielding a 72-dimensional binary status vector $(\text{entity}_m,\text{position}_m,\text{exist}_m)$2. The Hamming distance between reports $(\text{entity}_m,\text{position}_m,\text{exist}_m)$3 and $(\text{entity}_m,\text{position}_m,\text{exist}_m)$4 is

$(\text{entity}_m,\text{position}_m,\text{exist}_m)$5

A similarity score is then defined by the negative hyperbolic distance,

$(\text{entity}_m,\text{position}_m,\text{exist}_m)$6

and a listwise softmax ranking distribution is formed:

$(\text{entity}_m,\text{position}_m,\text{exist}_m)$7

If $(\text{entity}_m,\text{position}_m,\text{exist}_m)$8 denotes the nearest neighbor of sample $(\text{entity}_m,\text{position}_m,\text{exist}_m)$9 under the ground-truth Hamming metric, the ranking loss is

$F_t^l \in \mathbb{R}^{M \times d_l},$0

During training, this label-derived similarity guides report retrieval directly. During inference, nearest-neighbor search in learned hyperbolic space retrieves the most similar database image, and its report becomes the retrieved reference report $F_t^l \in \mathbb{R}^{M \times d_l},$1 (Zhou et al., 10 Jul 2025). The reported interpretation is that this mechanism supplies stronger contextual text priors for rare findings, which would otherwise be poorly represented in the training distribution.

This suggests that retrieval in REVTAF serves a dual role: it regularizes the latent geometry during training and acts as a semantic conditioning source at inference.

4. Fine-grained visual–text alignment and fusion

FVTAF is the second major component of REVTAF and is intended to address coarse alignment under weak supervision (Zhou et al., 10 Jul 2025). It operates on two prompt sources simultaneously: report-level GRPs and entity-level LRPs.

The global and local branches each compute cross-attention-based fused representations and attention maps. Their outputs are concatenated after residual fusion:

$F_t^l \in \mathbb{R}^{M \times d_l},$2

To constrain these branches toward semantically and spatially coherent behavior, REVTAF introduces a Fine-grained Cross-modal Consistency objective. Semantic similarity between global sentence embeddings and local entity embeddings is computed with cosine similarity,

$F_t^l \in \mathbb{R}^{M \times d_l},$3

followed by sigmoid normalization to $F_t^l \in \mathbb{R}^{M \times d_l},$4 (Zhou et al., 10 Jul 2025).

Spatial correlation between the corresponding attention maps is measured with an IoU-based quantity,

$F_t^l \in \mathbb{R}^{M \times d_l},$5

where intersection and union are implemented through elementwise minimum and maximum, respectively (Zhou et al., 10 Jul 2025). The resulting consistency loss is

$F_t^l \in \mathbb{R}^{M \times d_l},$6

REVTAF further replaces conventional cross-attention with an optimal transport-based mechanism termed multi-prompt Sinkhorn Attention (MPSA). Given visual queries $F_t^l \in \mathbb{R}^{M \times d_l},$7 and textual keys $F_t^l \in \mathbb{R}^{M \times d_l},$8, a cost matrix $F_t^l \in \mathbb{R}^{M \times d_l},$9 is formed, for example

$$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$0

With marginals $$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$1 and $$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$2, the entropic regularized transport plan $$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$3 minimizes

$$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$4

Defining $$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$5, Sinkhorn iterations update

$$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$6

yielding

$$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$7

If $$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$8 is the standard attention matrix, then MPSA uses

$$F_t^l(m)=f_{\text{medklip}}([\text{entity}_m,\text{position}_m,\text{exist}_m])$$9

The role assigned to OT in the paper is to provide globally normalized, geometry-aware reweighting that filters irrelevant prompts more effectively than standard dot-product attention under weak supervision (Zhou et al., 10 Jul 2025). The consistency constraint then aligns the two prompt sources at the level of both semantics and spatial attention, reducing contradictions between sentence-level and entity-level guidance.

5. Objective function, optimization, and implementation

REVTAF trains the report generator with token-level cross-entropy. For a report $R_{\text{ref}}$0,

$R_{\text{ref}}$1

and

$R_{\text{ref}}$2

The full training objective is given as

$R_{\text{ref}}$3

with $R_{\text{ref}}$4 and $R_{\text{ref}}$5 (Zhou et al., 10 Jul 2025). The description explicitly notes that there is no separate OT regularizer in the reported formulation.

The implementation details reported for REVTAF are specific. The visual encoder is ResNet-101 pretrained on ImageNet, and MedKLIP is used as the textual encoder for both LRPs and GRPs. Hyperbolic mapping is performed by an HNN with embedding dimension $R_{\text{ref}}$6. MedKLIP prompt channels have dimension 768. Images are resized to 224, training uses batch size 18, optimization uses AdamW with weight decay 0.05 and initial learning rate $R_{\text{ref}}$7 under a cosine learning-rate schedule, and the model is trained for 6 epochs (Zhou et al., 10 Jul 2025).

The reported computational profile includes the following elements. LRE intra-batch pairwise hyperbolic distances scale as $R_{\text{ref}}$8, and listwise ranking over the batch is $R_{\text{ref}}$9. Inference retrieval over a database of size $\mathrm{DB}=\{R_1,\dots,R_{N_D}\}$0 is naively $\mathrm{DB}=\{R_1,\dots,R_{N_D}\}$1, though the description states that an index such as FAISS can reduce this to sublinear time in practice. For FVTAF, forming the cost matrix is $\mathrm{DB}=\{R_1,\dots,R_{N_D}\}$2, Sinkhorn iterations cost $\mathrm{DB}=\{R_1,\dots,R_{N_D}\}$3 per branch for $\mathrm{DB}=\{R_1,\dots,R_{N_D}\}$4 iterations, and attention and fusion cost $\mathrm{DB}=\{R_1,\dots,R_{N_D}\}$5. Training time is reported as approximately 10 hours on an NVIDIA A800 80GB GPU for the described setting (Zhou et al., 10 Jul 2025).

6. Empirical evaluation and significance

REVTAF is evaluated on MIMIC-CXR and IU X-Ray (Zhou et al., 10 Jul 2025). The MIMIC-CXR split contains 270,790 training, 2,130 validation, and 3,858 test samples. IU X-Ray contains 7,470 images and 3,955 reports, and the evaluation protocol follows prior work in which models trained on MIMIC-CXR are tested on IU X-Ray (Zhou et al., 10 Jul 2025).

The evaluation uses both language-generation and clinical-efficacy metrics. Natural-language generation is measured with BLEU-1/2/3/4, METEOR, and ROUGE-L. Clinical efficacy is measured with CheXbert-derived Precision, Recall, and F1 over 14 disease categories (Zhou et al., 10 Jul 2025).

The reported main results are summarized below.

Dataset	Key reported results	Avg
MIMIC-CXR	BLEU-1 0.465, BLEU-4 0.182, METEOR 0.199, ROUGE-L 0.336, Precision 0.628, Recall 0.613, F1 0.592	0.397
IU X-Ray	BLEU-1 0.420, BLEU-4 0.107, METEOR 0.176, ROUGE-L 0.309, Precision 0.286, Recall 0.282, F1 0.273	0.251

Across datasets, the paper reports average improvements of 7.4% on MIMIC-CXR and 2.9% on IU X-Ray over state-of-the-art baselines (Zhou et al., 10 Jul 2025). It further states that comparisons with mainstream multimodal LLMs, including GPT-4, GPT-4o, GPT-4o-mini, and GPT-4.5, highlight REVTAF’s superiority in radiology report generation, and reports an inference time of approximately 3.36 s per sample in the test setting described (Zhou et al., 10 Jul 2025).

The ablation study on MIMIC-CXR attributes gains to both principal components. The baseline average is 0.369. Adding LRE only yields 0.384, adding FVTAF only yields 0.383, and combining LRE with FVTAF yields 0.397 (Zhou et al., 10 Jul 2025). The description further characterizes the LRE-only gain as +1.8% NLG and +0.9% CE, the FVTAF-only gain as +1.7% NLG and +0.9% CE, and the joint gain as +2.7% NLG and +2.9% CE (Zhou et al., 10 Jul 2025).

These results are used in the paper to argue that retrieval and fine-grained fusion are complementary rather than redundant. A plausible implication is that the retrieval stage improves prompt relevance while the fusion stage determines whether that retrieved information is injected in a clinically coherent way.

7. Relation to prior work, limitations, and interpretive issues

REVTAF is positioned against two broad baseline families: retrieval-augmented report generators that use simpler similarity mechanisms, and multimodal fusion models that rely on conventional attention under weak supervision (Zhou et al., 10 Jul 2025). Its specific claimed novelties are threefold.

First, LRE uses hyperbolic ranking rather than Euclidean similarity or CLIP-style feature pooling. The paper’s description argues that hyperbolic embeddings better capture hierarchical semantics and therefore improve retrieval relevance, especially for tail classes (Zhou et al., 10 Jul 2025). Second, FVTAF introduces simultaneous use of entity-level LRPs and report-level GRPs, rather than relying on a single textual prompt source. Third, the framework adds an FCC constraint at the attention-map level and uses OT-based attention to suppress irrelevant prompts during fusion (Zhou et al., 10 Jul 2025).

Several limitations are stated explicitly. Domain shift is identified as a risk: hyperbolic embeddings and retrieval behavior learned on one institution’s data may not generalize across hospitals without adaptation. Retrieval errors may inject spurious or contradictory text if the wrong GRPs are retrieved; OT and FCC are said to mitigate but not eliminate this failure mode. Residual bias from class imbalance may remain despite improvements on tail classes. Robustness depends in part on the Sinkhorn regularization setting and tokenization. Finally, clinical safety requires that generated reports be used as decision support rather than autonomous diagnosis, with human review by radiologists (Zhou et al., 10 Jul 2025).

A potential misconception is to treat REVTAF as primarily a retrieval system. The architecture described in the paper is more accurately a retrieval-conditioned fusion-and-generation system: retrieval supplies report-level contextual priors, but the principal technical burden of clinical faithfulness is shared with the alignment and fusion machinery. Another potential misconception is to view the reported comparison with general multimodal LLMs as a broad statement about medical AI overall. The concrete claim supported by the paper is narrower: within the evaluated radiology report generation setup, REVTAF achieved stronger reported NLG and clinical efficacy metrics and lower inference time than the listed GPT-series baselines (Zhou et al., 10 Jul 2025).

In the context of radiology report generation research, REVTAF is therefore best understood as a domain-specialized, retrieval-augmented multimodal architecture that couples hyperbolic semantic retrieval with OT-constrained fine-grained cross-modal fusion to improve report generation under long-tailed label distributions and weak supervision (Zhou et al., 10 Jul 2025).