Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learning

Abstract: Recent reinforcement learning (RL) approaches have advanced radiology report generation (RRG), yet two core limitations persist: (1) report-level rewards offer limited evidence-grounded guidance for clinical faithfulness; and (2) current methods lack an explicit self-improving mechanism to align with clinical preference. We introduce clinically aligned Evidence-aware Self-Correcting Reinforcement Learning (ESC-RL), comprising two key components. First, a Group-wise Evidence-aware Alignment Reward (GEAR) delivers group-wise, evidence-aware feedback. GEAR reinforces consistent grounding for true positives, recovers missed findings for false negatives, and suppresses unsupported content for false positives. Second, a Self-correcting Preference Learning (SPL) strategy automatically constructs a reliable, disease-aware preference dataset from multiple noisy observations and leverages an LLM to synthesize refined reports without human supervision. ESC-RL promotes clinically faithful, disease-aligned reward and supports continual self-improvement during training. Extensive experiments on two public chest X-ray datasets demonstrate consistent gains and state-of-the-art performance.

• The paper introduces ESC-RL, a novel framework that integrates evidence-aware rewards and self-correcting preference learning to improve radiology report generation.
• It employs a group-wise alignment mechanism (GEAR) to enforce accurate evidence localization and minimizes false findings using IoU-based losses and mean-square-error penalties.
• The framework achieves state-of-the-art performance on MIMIC-CXR and IU-Xray, enhancing lexical and clinical fidelity in generated reports.

Evidence-aware Self-Correcting Reinforcement Learning for Radiology Report Generation

Introduction and Motivation

Automated radiology report generation (RRG) is a critical subfield in medical AI, aiming to reduce expert workload and improve clinical workflows. Although recent advances in RRG have led to improved metrics by leveraging multimodal architectures, reinforcement learning (RL), and LLMs, several persistent limitations undermine clinical utility: (1) RL approaches typically utilize coarse, report-level rewards, which fail to enforce fine-grained evidence alignment and often propagate hallucinated or omitted findings; (2) preference-based RL methods rely on noisy preference datasets without explicit mechanisms for self-correction, resulting in models that do not continually adapt to evolving clinical standards.

The ESC-RL framework directly responds to these issues by integrating clinically aligned, disease-aware rewards with a novel self-correcting module for preference learning. The primary objective is to enforce robust visual-textual alignment at the level of disease evidence and enable automatic report refinement, thereby achieving superior clinical accuracy, error suppression, and adaptability.

Figure 1: Overview of the proposed Evidence-aware Self-Correcting Reinforcement Learning (ESC-RL) framework.

ESC-RL Framework Architecture

Group-wise Evidence-aware Alignment Reward (GEAR)

GEAR is foundational for precise, clinically relevant reinforcement learning. It introduces explicit, evidence-aware supervisory signals by partitioning disease predictions into true positives (TPs), false negatives (FNs), and false positives (FPs) using disease-status vectors extracted from the output and ground-truth reports (CheXbert). This group-wise decomposition enables targeted optimization:

• TPs: Enforce strict consistency between predicted and ground-truth disease-grounded response maps (DRMs) via IoU-based losses, ensuring accurate evidence localization.
• FNs: Penalize missing or overlooked findings by minimizing the mean-square-error between predicted and ground-truth DRMs, thus driving evidence recovery.
• FPs: Discourage unsupported or hallucinated evidence by directly suppressing the response energies of predicted DRMs in absent findings.

Through this design, GEAR provides direct, disease-specific policy gradients that promote granular clinical faithfulness, outperforming conventional report-level or keyword-based reinforcement schemes.

Self-correcting Preference Learning (SPL)

To address label noise and support continual self-improvement, SPL automatically constructs a disease-aware preference dataset by sampling multiple candidate reports per image and partitioning them by disease label status. This is summarized in a two-stage process:

• Dataset Construction & Scoring: Candidate and ground-truth descriptions are paired and scored with preference labels using an LLM. Preference labels capture fine-grained correctness at a disease-description level.
• Sample Filtering & Selection: A lightweight predictor is trained using a dual-threshold mechanism on KL-divergence between ground-truth and predicted preferences, filtering out unreliable or mislabeled observations.

This is used as a gating mechanism for report refinement: only disease-level descriptions presumptively supported by evidence and preference consensus are retained.

Figure 2: Illustration of the Self-correcting Preference Learning (SPL) module.

LLM-guided Report Re-integration

SPL’s output is fed into a report re-integration pipeline. A sequence of candidate disease-level observations is provided as input to the LLM, together with the trustworthy preference dataset and a structured prompt (see Figure 3 in the appendix), instructing the model to assemble a final, refined report. The LLM is constrained to discard unreliable or unsupported statements and to maintain strict adherence to validated clinical findings.

Figure 4: An example of re-integrating multiple observations into a refined report.

Empirical Evaluation

Experimental Protocol

ESC-RL was benchmarked on MIMIC-CXR and IU-Xray, two large-scale chest X-ray datasets, and comprehensively compared against classical, strong RL, and LLM-based RRG baselines. The evaluation suite includes lexical (BLEU-1/4, ROUGE-L, BERTScore) and radiology-specific metrics (RadCliQ, RadGraphF1, CheXbertF1, GREEN), as well as clinical efficacy (CE) statistics.

Strong Numerical Results

ESC-RL achieved consistently superior results, setting new SOTA performance across all principal axes of evaluation. Notably:

• On MIMIC-CXR, ESC-RL produced BLEU-1/4 scores (0.487/0.199), RadGraphF1 (0.304), CheXbertF1 (0.608), and GREEN (0.394), outperforming strong recent models such as REVTAF and OISA by margins up to 2.2% (BLEU-1), 1.6% (CheXbertF1), and 5.0% (GREEN) in absolute terms.
• On IU-Xray, ESC-RL maintained its dominance, with parallel gains in lexical fidelity and clinical fact extraction.

Figure 5 illustrates qualitative superiority: ESC-RL reports show more comprehensive, granular, and accurate disease findings, avoiding both factual omissions and hallucinations, compared to R2GenRL, PromptMRG, and REVTAF.

Figure 5: Qualitative comparison of reports generated by R2GenRL, PromptMRG, REVTAF, and ESC-RL. Incorrect statements are marked; sentences are mapped to reference descriptions with color coding.

Ablation Analysis

Systematic ablations isolate the impact of each component. Both GEAR and SPL deliver standalone gains, and their interaction under RL maximizes every evaluation metric. GEAR’s evidence-aware shaping is most critical for radiology-specific metrics, while SPL robustly diminishes error propagation from noisy observations and boosts end-to-end clinical alignment.

Figure 6: Ablation study of reward weight ($\gamma$) and LLM selection in the SPL module.

Theoretical and Practical Implications

ESC-RL presents new directions for RL-based clinical NLG:

• Reward modeling: GEAR demonstrates that vision-LLMs supporting DRM extraction can be reliably exploited for dense, localizable, and disease-aware reward shaping, which may generalize to other sequence generation tasks requiring explicit evidence alignment.
• Self-correcting policy optimization: SPL’s architecture introduces a robust scaffold for dynamic preference dataset construction and curriculum formation, allowing continuous adaptation without manual relabeling or annotation.
• LLM-in-the-loop refinement: The modular prompt-based integration of LLMs in the report post-processing step is shown to deliver both fluency and factual consistency, primarily affecting non-clinical (lexical) metrics, while core gains are derived from the RL framework itself.

Practically, the ESC-RL pipeline maintains computational efficiency, modularity (plug-and-play for vision-language backbone swaps), and delivers inference and training latency comparable to prior work—making it compatible with real-world deployment scenarios.

Limitations and Future Work

Current validation is restricted to chest X-rays; extension to other imaging modalities (CT, MRI) or anatomies will require further confirmation. The framework depends on the accuracy and robustness of external disease extractors and DRM generators (here, CheXbert and MAVL). Integrating more advanced multimodal foundation models or further automating extractor tuning could mitigate potential bottlenecks. Future work should consider joint optimization of the entire pipeline and broader generalization studies across domains.

Conclusion

ESC-RL introduces a clinically aligned, evidence-aware RL framework that integrates group-wise reward shaping and self-correcting preference learning for automated radiology report generation. By enforcing fine-grained alignment and leveraging automatic, LLM-guided report refinement, ESC-RL achieves superior lexical and clinical fidelity across standard benchmarks and sets a new expectation for reinforcement learning in high-stakes clinical NLG tasks.