PET2Rep: PET Radiology Report Benchmark

• PET2Rep is a benchmark that evaluates vision-language models for generating structured PET radiology reports with detailed metabolic insights.
• It leverages real-world 3D whole-body PET/CT image-report pairs covering dozens of organs to mimic oncologic clinical practices.
• The evaluation employs both standard NLG metrics and a novel Clinical Efficiency metric to quantify accuracy in reporting key radiotracer uptake abnormalities.

Searching arXiv for the PET2Rep paper and closely related context. PET2Rep is a large-scale comprehensive benchmark for evaluation of general and medical vision-LLMs (VLMs) for radiology report generation for positron emission tomography (PET) images. Introduced in "PET2Rep: Towards Vision-LLM-Drived Automated Radiology Report Generation for Positron Emission Tomography" (Zhang et al., 6 Aug 2025), it is presented as the first dedicated dataset for PET report generation with metabolic information, built around whole-body image-report pairs that cover dozens of organs and mirror the clinical comprehensiveness of oncologic PET interpretation. Its central purpose is not only to support automated report generation, but also to expose the mismatch between strong generic language generation performance and clinically adequate PET reporting.

1. Definition and problem setting

PET2Rep is framed around a specific limitation in contemporary multimodal medical AI: existing applications of VLMs in the medical domain have predominantly focused on structural imaging modalities, while the unique characteristics of molecular PET imaging have largely been overlooked (Zhang et al., 6 Aug 2025). The benchmark therefore targets radiology report generation for PET images rather than for chest radiography or anatomically localized CT tasks.

The problem formulation is shaped by the imaging modality itself. PET is described as a cornerstone of modern oncologic and neurologic imaging because it illuminates dynamic metabolic processes that transcend the anatomical focus of traditional imaging technologies. In PET2Rep, this distinction is operationalized through reports that must capture not only anatomic findings but also radiotracer uptake patterns, especially for key organs. The benchmark is therefore oriented toward functional assessment as much as toward visual captioning.

A recurrent misconception in automated report generation is that PET reporting is a straightforward extension of existing structural-imaging benchmarks. PET2Rep is organized to refute that assumption. Whole-body PET/CT reporting requires multi-organ summarization, metabolic interpretation, and structured documentation at a scale that differs materially from single-region image-description tasks. The paper’s results further argue that current state-of-the-art VLMs remain considerably short of fulfilling practical needs in this setting (Zhang et al., 6 Aug 2025).

2. Dataset composition and image-report pairing

PET2Rep is introduced as the first large-scale, clinically sourced, whole-body PET/CT image-report paired benchmark tailored for VLM development and evaluation in radiology report generation (Zhang et al., 6 Aug 2025). It contains 565 cases of 3D whole-body $^{18}$F-FDG PET/CT collected from real clinical settings, including multiple medical centers. The stated motivation for this sourcing is to ensure data authenticity and to avoid biases from public archives or toy tasks.

Each case includes the complete imaging protocol, after patients’ proper preparation, together with physician-generated structured radiology reports. Coverage extends across the entire body from head to proximal limbs, so that reports address findings across dozens of organs and better reflect oncological practice, where systemic evaluation is crucial for cancer detection and staging.

Aspect	PET2Rep characteristic
Scale and source	565 cases of 3D whole-body $^{18}$F-FDG PET/CT from real clinical settings
Coverage	Entire body from head to proximal limbs; dozens of organs
Report pairing	Physician-generated structured radiology reports
Input representation	Separate PET/CT and fused PET/CT representative coronal slices

The inclusion of metabolic information is a defining feature. PET2Rep reports systematically describe radiotracer uptake patterns such as increased, decreased, absent, and normal uptake for key organs. The benchmark therefore encodes a form of clinical semantics that is neglected by traditional radiology benchmarks centered on anatomical abnormality description alone.

Preprocessing is standardized. CT images are resampled to PET resolution and z-score normalized. PET images are normalized to Standardized Uptake Value (SUV), with

$\text{SUV} = \frac{C(t)}{\text{ID}/\text{BW}},$

where $C(t)$ is tissue radioactivity concentration, $\text{ID}$ is the injected dose, and $\text{BW}$ is body weight (Zhang et al., 6 Aug 2025). For 3D-to-2D mapping, representative coronal slices are selected in a manner described as clinical-practice inspired, because most evaluated VLMs are 2D-based. Inputs include both separate PET/CT images and fused PET/CT images produced via superposition, reflecting real-world radiology workflows.

3. Report schema and representation of clinical content

The report side of PET2Rep is deliberately constrained. Reports are constructed using a radiologist-designed template and are fully structured, head-to-toe, and strictly objective, enumerating abnormalities rather than diagnostic impressions (Zhang et al., 6 Aug 2025). This design choice has methodological significance: it reduces stylistic variance and makes the evaluation problem more explicitly about factual content extraction and structured summarization.

The objective format is especially relevant for PET because the benchmark is not merely testing free-form language fluency. It is testing whether a model can map multimodal whole-body input to a report that correctly localizes and categorizes uptake abnormalities across many anatomical structures. The paper states that the benchmark uniquely captures whole-body image-report pairs that cover dozens of organs and fill a critical gap in existing benchmarks.

This structure also clarifies what counts as success and failure. A well-formed report in PET2Rep is one that follows the prescribed organization and accurately describes metabolic uptake patterns in the relevant organs. By contrast, grammatically polished but clinically incorrect reports remain inadequate. This is the foundation for the benchmark’s dual emphasis on natural language generation metrics and clinically targeted evaluation (Zhang et al., 6 Aug 2025).

4. Evaluation methodology and the Clinical Efficiency metric

PET2Rep evaluates models with both standard natural language generation metrics and a new Clinical Efficiency (CE) metric (Zhang et al., 6 Aug 2025). The standard metrics include BLEU, METEOR, and ROUGE-L. The paper explicitly argues that such metrics assess text similarity but overlook clinical correctness, since two reports with opposite medical findings can still obtain similar overlap-based scores.

The CE metric is designed around PET’s modality-specific requirement to describe radiotracer uptake patterns. For 19 key organs or anatomic structures, the generated report is parsed to assign one of four states: Increased Uptake, Decreased Uptake, Absent Uptake, or Normal. If an organ is not mentioned, it is classified as “Normal,” reflecting real reporting practices.

For each organ and each abnormal class—Increased Uptake, Decreased Uptake, and Absent Uptake—the benchmark computes true positives $TP_c$, false positives $FP_c$, and false negatives $FN_c$, and then derives

$\text{Precision}_c = \frac{TP_c}{TP_c + FP_c},$

$^{18}$0

and

$^{18}$1

The final CE metrics are macro-averaged across the three positive abnormality classes (Zhang et al., 6 Aug 2025).

The benchmark also includes manual radiologist review. The reported rubric covers medical accuracy, key findings completeness, expression clarity, clinical usability, and overall rating, with each dimension scored from 1 to 5. This makes PET2Rep notable not only as a dataset but also as an evaluation protocol in which automatic metrics and expert judgment are jointly used.

A central implication of the evaluation design is that PET2Rep treats clinical adequacy as partially independent from lexical similarity. This suggests a broader methodological point: benchmark design in medical VLMs must align the target metric with clinically consequential information, rather than with surface-form correspondence alone.

5. Benchmarking of vision-LLMs

PET2Rep reports a head-to-head comparison of 30 cutting-edge general-purpose and medical-specialized VLMs (Zhang et al., 6 Aug 2025). The evaluation is conducted in zero-shot fashion, with no PET-specific fine-tuning, and uses prompt templates that specify modality, clinical task, and report structure. Multiple input strategies are considered, including separated PET/CT images and fused images.

The model pool includes examples such as Qwen2.5-VL, InternVL3, LLaVA-V1.5, Deepseek-VL2, Yi-VL, MedGemma, LLaVA-Med, RadFM, HuatuoGPT-Vision, Lingshu, Qilin-Med-VL, and closed-source commercial models such as Gemini 2.5 Pro, GPT-4o, Moonshot-v1, and Qwen-VL-Max. The emphasis is comparative rather than architectural: PET2Rep is intended to evaluate a diverse cross-section of current VLM practice under a unified PET reporting task.

The main finding is negative in a precise sense. The results show that current state-of-the-art VLMs perform poorly on PET report generation and fall considerably short of practical needs (Zhang et al., 6 Aug 2025). Most models fail to exceed even a template baseline that blindly fills out a structured report. Only a small number of models, including examples from the Lingshu and MedGemma series, are described as approaching but not clearly surpassing baseline performance.

The benchmark also reports a disparity between natural language generation metrics and CE metrics. Some models generate grammatically correct or well-worded reports and obtain relatively high NLG scores, yet miss key findings and therefore score much lower on clinical efficiency. This is one of the paper’s strongest arguments against reading language-overlap performance as evidence of clinical readiness.

Manual radiologist evaluation leads to an even stricter conclusion: even the best model outputs are rated not clinically usable and fail to meet real-world standards across the evaluated dimensions (Zhang et al., 6 Aug 2025).

6. Failure modes, limitations, and research implications

PET2Rep identifies several insufficiencies in current VLMs. These include lack of clinical generalization, overfitting to training distributions, inadequacy of NLG metrics, difficulty integrating functional and structural information, and absence of 3D context (Zhang et al., 6 Aug 2025). The benchmark therefore functions not only as an evaluation suite but also as a failure-analysis framework for multimodal medical generation.

The paper organizes model failures into five categories: correct template outputs, irrelevant information, unstructured outputs, refusal to answer, and chaotic or empty outputs. Correct template outputs are characterized as rare. More commonly, models generate off-topic details, ignore the requested report structure, refuse the task for sensitivity or ethics reasons, or produce blank or nonsensical responses. These categories are important because they show that failure is not reducible to small factual errors; it often occurs at the level of task framing, report organization, and modality understanding.

Another misconception directly addressed by the results is that larger models within the same family necessarily outperform smaller ones. PET2Rep reports that larger models do not always do so; in some instances they hallucinate more severely or fill reports with invented or off-topic data (Zhang et al., 6 Aug 2025). The limitation is therefore not simply model scale.

The recommendations for future research follow directly from these findings. The paper argues for rich, real-world benchmarks; a shift from NLG to clinically meaningful efficacy assessment; model development beyond scaling; incorporation of 3D volumetric data and clinical workflow; and broader multilingual and cross-center generalization. It also proposes inclusion of quantitative metrics such as SUV and lesion volume in both data and evaluation. A plausible implication is that PET2Rep is intended as an infrastructure benchmark for domain adaptation in PET-centered multimodal AI, rather than as a static leaderboard alone.

In this sense, PET2Rep occupies a specific place in the evolution of automated radiology reporting. It formalizes whole-body PET report generation as a benchmark in which metabolic information, structured reporting, and clinical correctness are all first-class evaluation targets. Its principal significance lies in making the gap between current VLM capabilities and realistic PET reporting requirements measurable and reproducible (Zhang et al., 6 Aug 2025).