RadEyeVideo for Chest X-Ray Analysis

• RadEyeVideo is a video-based prompting method that transforms radiologists’ sequential eye-fixation data into a short video, maintaining both spatial and temporal gaze information for chest X-ray analysis.
• It preserves the order and duration of gaze fixations, overcoming the limitations of static heatmaps and textual prompts in capturing diagnostic focus.
• Integration with video-capable LVLMs like LLaVA-OneVision has shown improved report accuracy and diagnostic performance, making it a scalable, human-centered approach in medical imaging.

RadEyeVideo is a video-based prompting method for chest X-ray analysis that augments general-domain Large Vision-LLMs (LVLMs) with radiologists’ eye-fixation trajectories encoded as a short video sequence rather than as static heatmaps or textual prompts. The method is designed to preserve both the spatial distribution and the sequential order of gaze, thereby exposing the model to the areas of interest and the order in which they are examined during image interpretation. In the formulation reported by the original work, RadEyeVideo is applied to chest X-ray report generation and disease diagnosis using video-capable open-source LVLMs, including VideoLLaMA3, LongVA, and LLaVA-OneVision, without fine-tuning the model weights on gaze data (Kim et al., 12 Jul 2025).

1. Conceptual framing and scope

RadEyeVideo is explicitly described as not being a brand-new neural network but rather a prompting scheme that extends existing general-domain LVLMs with video-input capability (Kim et al., 12 Jul 2025). Its input pipeline consists of a chest X-ray image $I$, a radiologist fixation sequence $G$, and a textual prompt $T$. The fixation sequence is converted into a gaze video $V$, and the tuple $(I, V, T)$ is then provided to a video-capable LVLM to generate either a report or a diagnosis.

The central methodological claim is that prior gaze-conditioning approaches often overlook the sequential order of eye movements. RadEyeVideo addresses this by representing fixations as a video sequence of “red dot” gaze frames. This preserves both temporal and spatial gaze dynamics. The reported motivation is that sequential gaze may contain clinically relevant structure that is lost when fixation information is collapsed into a static heatmap or rendered as a textual summary (Kim et al., 12 Jul 2025).

The approach is situated within chest X-ray analysis rather than general medical imaging. The reported downstream tasks are report generation and disease diagnosis on a dataset derived from MIMIC-CXR, REFLACX, and EyeGaze. The paper further states that the method is intended as a scalable human-centered approach for utilizing LVLMs in medical image analytics (Kim et al., 12 Jul 2025).

2. Eye-gaze video representation

The gaze representation begins from an ordered fixation sequence

$G = \{g_1,\dots,g_n\},$

where each fixation is

$g_i = (x_i, y_i, t_i),$

with $t_i$ measured in seconds and a fixed gaze radius $r = 5$ px (Kim et al., 12 Jul 2025).

The conversion rule uses $\mathrm{fps} = 10$. Each fixation $G$0 is rendered as a number of consecutive frames

$G$1

In each frame $G$2, the original chest X-ray is overlaid with a red dot at the current fixation coordinate $G$3. This construction preserves where the gaze dwells and for how long it dwells there (Kim et al., 12 Jul 2025).

To match the LVLM’s fixed-length video input, the method uniformly samples $G$4 frames from the full rendered fixation video: $G$5 The paper notes an example fixed-length setting of $G$6 frames. This produces a compact video token sequence that remains compatible with pre-existing video-capable LVLM interfaces (Kim et al., 12 Jul 2025).

A key distinction from alternative gaze encodings follows directly from this representation. Static heatmaps encode spatial emphasis but not ordering, whereas the red-dot video retains sequential order and dwell time. The paper further reports that split-duration text prompts risk being mis-interpreted. This suggests that the representational gain of RadEyeVideo is not merely additional supervision, but a change in modality that better matches the temporal inductive bias of video LVLMs (Kim et al., 12 Jul 2025).

3. Integration with video-capable LVLMs

RadEyeVideo relies on the internal video-processing machinery already present in LVLMs such as VideoLLaMA3, LongVA, and LLaVA-OneVision. The architecture described in the source includes a spatial encoder applied to each image or frame, a temporal or spatio-temporal encoder implemented as a stack of transformer blocks with self-attention over space and time, and cross-attention layers in the LLM that fuse visual embeddings with token embeddings (Kim et al., 12 Jul 2025).

The spatio-temporal self-attention block is summarized as

$G$7

where $G$8 aggregates token features across spatial patches and time frames, and $G$9, $T$0, and $T$1 are learned projections (Kim et al., 12 Jul 2025).

After $T$2 video-encoder layers, the resulting video tokens $T$3 are injected into the language backbone through cross-attention: $T$4 The paper also gives the cross-modal fusion at the level of query, key, and value projections as

$T$5

No additional gating module is introduced; the standard residual cross-attention is used (Kim et al., 12 Jul 2025).

The multimodal input consists of three elements: the static chest X-ray image $T$6, the gaze video $T$7, and a textual prompt $T$8, optionally with up to three in-context exemplar reports $T$9. For report generation, the paper factorizes the process into findings generation and impressions summarization: $V$0 For diagnosis, the formulation is

$V$1

Because the method is zero- or few-shot prompting rather than parameter fine-tuning, RadEyeVideo can be attached to existing general-domain LVLMs without retraining the backbone on gaze annotations (Kim et al., 12 Jul 2025).

4. Dataset, tasks, and evaluation protocol

The experiments are conducted on MIMIC-Eye-Video, described as being derived from MIMIC-CXR, REFLACX, and EyeGaze. The dataset statistics reported in the source are 3,693 unique chest X-ray images with eye-tracking and 6,896 total dictated reports (Kim et al., 12 Jul 2025).

Two disjoint evaluation splits are specified. The Alpha split contains 3,134 images and 3,699 reports with an average of 1.18 radiologists per image. The Beta split contains 549 images and 3,197 reports with an average of 5.82 radiologists per image. The gaze collection protocols are stated to follow Karargyris et al. (PhysioNet Eye Gaze 2020) and Lanfredi et al. (REFLACX 2021) (Kim et al., 12 Jul 2025).

Two task families are evaluated. The first is report generation, divided into findings generation and impressions summarization. The second is disease diagnosis. Inference is performed with batch size $V$2, temperature $V$3 or $V$4 if $V$5 is not supported, and maximum output length $V$6 tokens for report generation or $V$7 tokens for diagnosis (Kim et al., 12 Jul 2025).

The evaluation metrics are micro-averaged across five key findings and include CheXbert F1 for the top-5 common abnormalities, RadGraph-XL relation/entity overlap, and RaTEScore for radiology style and clinical correctness. The paper also reports a scaled score relative to CheXagent: $V$8 This scaled formulation is used to compare specialized medical LVLMs and general-domain LVLMs under a common normalization (Kim et al., 12 Jul 2025).

5. Empirical results and comparative behavior

The paper reports that when prompted with eye-gaze videos, model performance improves by up to $V$9 in the report generation task and on average $(I, V, T)$0 for both tasks using scaled evaluation metrics (Kim et al., 12 Jul 2025). It further states that LLaVA-OneVision enhanced with RadEyeVideo surpasses task-specific medical LVLMs such as MAIRA-2 and CheXagent.

The comparative scores reported in the source are summarized below.

Model / setting	Overall scaled score
CheXagent	100.0
CXR-LLaVA	202.7
MAIRA-2	152.2
VideoLLaMA3 (no gaze)	151.3
VideoLLaMA3 + HeatMap	122.7
VideoLLaMA3 + FixationText	142.7
VideoLLaMA3 + RadEyeVideo	131.1
LongVA (no gaze)	133.5
LongVA + RadEyeVideo	135.6
LLaVA-OneVision (no gaze)	139.4
LLaVA-OneVision + RadEyeVideo	154.6

More specific deltas are also given. For VideoLLaMA3, adding RadEyeVideo yields diagnosis improvements of $(I, V, T)$1 but report decreases of $(I, V, T)$2, and the overall scaled score drops by $(I, V, T)$3 relative to the no-gaze version (Kim et al., 12 Jul 2025). For LongVA, RadEyeVideo increases the overall scaled score from $(I, V, T)$4 to $(I, V, T)$5, including a $(I, V, T)$6 Alpha improvement in report generation. For LLaVA-OneVision, the addition of RadEyeVideo increases the overall score from $(I, V, T)$7 to $(I, V, T)$8, with report improvements of $(I, V, T)$9 on Alpha and $G = \{g_1,\dots,g_n\},$0 on Beta, and diagnosis improvements of $G = \{g_1,\dots,g_n\},$1 on Alpha and $G = \{g_1,\dots,g_n\},$2 on Beta (Kim et al., 12 Jul 2025).

These results indicate that the effect of gaze-video prompting is backbone-dependent. The paper explicitly notes that gains are largest on LLaVA-OneVision, showing that the backbone’s ability to process spatio-temporal input is critical. A plausible implication is that RadEyeVideo is most effective when the host LVLM has sufficiently strong temporal modeling capacity to exploit the sequential fixation structure (Kim et al., 12 Jul 2025).

6. Interpretation, limitations, and broader significance

A recurring theme in the paper is that RadEyeVideo preserves the sequential order of gaze fixations and their durations, unlike static heatmaps, which lose ordering information. This is presented as the primary reason the method can convey domain expert knowledge more faithfully to video-capable LVLMs (Kim et al., 12 Jul 2025). The work therefore contributes to a line of research in which expert behavioral traces are treated as first-class multimodal signals rather than auxiliary annotations.

The method is also computationally lightweight relative to retraining a specialized medical LVLM. The reported additional cost is only about $G = \{g_1,\dots,g_n\},$3 more GPU memory than a static heatmap prompt, or about $G = \{g_1,\dots,g_n\},$4 GB more when using LLaVA-OneVision. The paper characterizes this as far less costly than fully re-training a task-specific medical LVLM (Kim et al., 12 Jul 2025).

The reported failure modes remain clinically relevant. Example errors include minor localization errors in reports and cases in which abnormality locations can be mis-stated. The limitations section also identifies the relatively small eye-tracking dataset of 3.7k images, the fact that MIMIC-CXR may not capture the full diversity of clinical fluoroscopy, and the restriction of the study to chest X-ray rather than CT, MRI, or other multimodal tasks such as visual question answering and anatomical detection (Kim et al., 12 Jul 2025).

In a broader methodological sense, RadEyeVideo demonstrates that general-domain LVLMs can exceed task-specific medical LVLMs when supplied with structured expert signals in a modality they can natively process. The source explicitly states that the method enhances LLaVA-OneVision to surpass MAIRA-2 and CheXagent, both trained on large chest X-ray data (Kim et al., 12 Jul 2025). This suggests that, in at least some clinical settings, performance may depend not only on domain-specific pretraining but also on whether expert attention dynamics are represented in a form compatible with the model’s spatio-temporal interface.