CogRad: A Cognitively-Inspired Multi-Agent Framework for Radiology Report Generation

Abstract: Automated radiology report generation (RRG) can ease radiologist workload, yet most existing systems produce a report in a single forward pass, with no mechanism to check a claim against the image or revisit a finding once stated. We present CogRad, a cognitively inspired multi-agent framework that structures generation around four stages of a radiologist's reading process. A Scout agent discovers anatomical regions directly from image patches via slot attention and assigns region and disease-level triage scores; an Investigator agent concentrates representational capacity on the regions Scout flags as suspicious; a Writer agent compiles these signals into a disease gated visual prefix for a LLM; and a Verifier agent supervises training with a visual entailment loss and, at inference, re-examines its own draft sentence by sentence, regenerating any report it judges insufficiently grounded. On CheXpert Plus, CogRad attains a BLEU-4 of 0.316 and a CIDEr of 0.322, the best scores among the methods we compare against. On IU X-Ray, it attains a BLEU-4 of 0.201 and a CIDEr of 0.724, leading every baseline on every standard NLG metric. We further evaluate CogRad with RadGraph F1, CheXbert F1, and a hallucination analysis to assess clinical accuracy beyond standard text-overlap metrics, complemented by ablation studies and Grad-CAM-based visualizations that characterize each agent's contribution and the model's visual grounding.

• The paper introduces a four-agent, cognitively-inspired framework that iteratively generates and verifies radiology reports.
• The model utilizes specialized agents (Scout, Investigator, Writer, Verifier) to achieve higher BLEU and CIDEr scores on CheXpert Plus and IU X-Ray datasets.
• The framework enhances clinical explainability and implements a self-correction mechanism to align generated text with visual evidence.

CogRad: Cognitively-Inspired Multi-Agent Radiology Report Generation

Introduction and Motivation

CogRad addresses a central challenge in radiology report generation (RRG): achieving clinically accurate, textually fluent, and visually grounded reports directly from chest X-ray images. Existing VLM and LLM-based RRG methods typically pursue a single-pass generation paradigm, constraining their ability to check, revise, or verify generated statements against the medical image after the generation step. This workflow diverges from that of practicing radiologists, who iterate between global surveys, focused investigations, structured synthesis, and cross-verification steps during case interpretation.

To bridge this gap, CogRad implements a cognitively-inspired, multi-agent architecture that decomposes the reporting process into four explicit, communicative agents: Scout, Investigator, Writer, and Verifier. Each agent is instantiated as a neural module, and all inter-agent exchanges occur via structured visual tensors rather than only text. This framework enables iterative and evidence-driven report generation, closely emulating expert clinical workflows.

Figure 1: Overview of the CogRad framework and its four specialized agents.

Architecture: Cognitively-Inspired Multi-Agent Pipeline

The four-agent pipeline is designed as follows:

• Scout Agent: Utilizes slot-attention to identify potentially pathological regions by dynamically allocating a fixed number of learnable region slots over input patch features. The output comprises learned region embeddings, triage abnormality scores for each region, and disease-level probability estimates.
• Investigator Agent: Allocates focused representational capacity to regions with the highest abnormality scores, leveraging self-attention to model inter-regional dependencies and cross-attention to aggregate features from the raw patch grid. Regions with the highest focus weights are refined over multiple iterations to enable fine-grained feature extraction.
• Writer Agent: Constructs a hierarchically organized, disease-gated visual prefix comprising disease tokens, regional summaries, and global representations for the LLM (LLaMA-2-7B, LoRA-tuned). A confidence gating mechanism down-weights or masks disease contexts deemed unlikely, reducing the risk of hallucinated statements.
• Verifier Agent: Provides both a training-time, per-token visual entailment loss and an inference-time, sentence-level verifier. During inference, sentences in the draft report below a confidence threshold trigger selective re-examination, prompting a reallocation of attention, further feature extraction, and report regeneration.

This dataflow maintains continuous visual-semantic grounding and enables targeted self-correction cycles until each statement passes automated visual verification.

Datasets and Experimental Protocols

CogRad is evaluated on two established datasets:

• CheXpert Plus: Large-scale, high-heterogeneity clinical dataset with over 223,000 image-report pairs and 14 disease labels, drawn primarily from acute care and ICU populations.
• IU X-Ray: Smaller, more templated dataset with 3,955 reports and 7,470 images, presenting an outpatient case mix.

Both patient-wise and official splits were employed to preclude leakage and preserve clinical relevance.

Figure 2: Overview of the two evaluation datasets.

Results: Quantitative Performance and Ablation

Across both datasets, CogRad achieves strong numerical results on standard NLG metrics—surpassing recent baselines on all reported scores:

• CheXpert Plus: Highest BLEU-4 (0.316) and CIDEr (0.322) among all compared approaches.
• IU X-Ray: Maintains top performance with BLEU-4 (0.201) and the highest CIDEr (0.724).

Ablation studies indicate that both the Scout (disease classification supervision) and Verifier (visual entailment loss) agents provide additive value. Notably, the Verifier is more influential on CheXpert Plus (larger, more heterogenous), while the Scout's impact increases in the smaller IU X-Ray setting. The full model consistently outperforms all ablations.

Figure 3: Ablation results on IU X-Ray.

Figure 4: Ablation results on CheXpert Plus. All three runs are independently trained.

Performance profiles plotted across all dataset-metric combinations reinforce these findings; the integration of all four cognitively-inspired modules delivers the most uniform improvement.

Figure 5: Performance profiles of CogRad and its ablated variants across both datasets.

Clinical Accuracy, Hallucination, and Explainability

Beyond textual overlap, the entity-level correctness of CogRad's outputs is quantified via RadGraph F1, CheXbert F1, and hallucination rate. The results surface an important and nontrivial dissociation: improvements in n-gram overlap metrics (BLEU, ROUGE, CIDEr) do not always correspond to gains in entity-level clinical accuracy. For CheXpert Plus, despite substantial improvement in NLG metrics, entity F1 and RadGraph F1 remain comparatively modest, highlighting persistent limitations in factual consistency; hallucination rates remain non-negligible and are concentrated on common findings and device mentions, rather than rare or spurious entities.

In terms of explainability, the Verifier's cross-modal attention maps double as GradCAM-style heatmaps, facilitating transparent attribution of textual statements to image regions at both token and sentence granularity.

Figure 6: Clinical explainability analysis for X-ray image diagnosis.

The confidence scores provided by the Verifier enable a self-checking feedback loop, which is both interpretable and operationalized for selective report regeneration. The proposed inference-time self-correction mechanism mimics human double-reading workflows and has direct implications for trustworthy clinical deployment.

Limitations and Future Directions

Despite robust gains on benchmarking datasets, two principal limitations emerge:

• The gap between NLG scores and entity-level factual accuracy (as reflected by persistent hallucination and miss rates) suggests that n-gram metrics are insufficient proxies for clinical reliability.
• The self-correction loop currently acts at the sentence level; fine-grained, region-sentence alignment could allow for more targeted interventions.

Potential future advances include (a) direct incorporation of clinical entity F1 metrics into the training objective, (b) region-aware feedback for more precise selective generation, and (c) extension of the hallucination analysis to explicitly map entity claims to anatomical sites.

Theoretical and Practical Implications

The CogRad architecture demonstrates that end-to-end, cognitively structured multi-agent frameworks can yield improvements in the factual reliability, interpretability, and verifiability of reports generated from medical images. Methodologically, the structured communication via tensors supports robust cross-agent collaboration, and the visual prefix modeling with disease gating offers a generalizable scheme for conditioning LLMs in medical and other sensitive domains.

Practically, CogRad's closed-loop self-verification and correction protocol enhances robustness—a critical requirement for clinical decision-support integration. The explicit, interpretable grounding of statements to image regions and the alignment with radiologist workflow paradigms make this architecture an instructive blueprint for future agentic AI in medical domains.

Conclusion

CogRad substantiates that a multi-agent, cognitively mapped, and visually grounded RRG pipeline can surpass strong baselines on traditional and clinical accuracy metrics, albeit with uneven translation between linguistic and entity-level accuracy. The analysis highlights both the need for, and the limitations of, current evaluation pipelines, and establishes a clear direction for enhancing model calibration, trustworthiness, and explainability in medical vision-language generation systems.

(2607.03853)