ScanDiff: Diffusion Model for Scanpath Prediction

• ScanDiff is a diffusion-based generative model that predicts human gaze scanpaths as stochastic samples, conditioned on both image data and textual task descriptions.
• It unifies free-viewing and goal-directed search using cross-attention between DINOv2 image features and CLIP text embeddings to capture scanpath variability.
• Evaluations show ScanDiff achieves superior performance and diversity metrics on free-viewing and visual search benchmarks compared to deterministic models.

Searching arXiv for the specified paper and closely related scanpath-diffusion work to ground the article in current literature. ScanDiff is a diffusion-based generative model for human gaze scanpath prediction in natural images that produces sequences of fixations, including positions and durations, conditioned on both the visual stimulus and a textual description of the viewing task. It was introduced as a unified architecture for free-viewing and goal-directed visual search, with the explicit aim of modeling scanpath variability rather than a single averaged trajectory. In this formulation, a scanpath $\mathbf{s} = \{f_1,\dots,f_N\}$ consists of fixations $f_i = (r_i,m_i)$, where $r_i = (x_i,y_i) \in \mathbb{R}^2$ is the 2D fixation location and $m_i \in \mathbb{R}^+$ is fixation duration (Cartella et al., 30 Jul 2025).

1. Problem setting and conceptual rationale

Scanpath prediction addresses the ordered sequence of eye fixations and saccades generated while viewing a visual stimulus. The problem is consequential because it describes how visual attention unfolds over time, with stated applications in human–computer interaction, autonomous driving, cognitive robotics and active perception, and vision–language tasks such as VQA, referring expressions, and image captioning. ScanDiff is motivated by the claim that many deep scanpath models, including CNN+RNN and Transformer variants, generate averaged behavior, exhibit limited stochasticity, and are usually specialized either for free-viewing or for task-driven settings rather than both in one framework (Cartella et al., 30 Jul 2025).

The central criticism addressed by ScanDiff is that deterministic objectives and evaluation against aggregated ground-truth behavior encourage the generation of a single “mean” scanpath. In the reported problem formulation, this appears as very high similarity among generated scanpaths themselves and, in some cases, higher mutual similarity between generated scanpaths than between real human scanpaths. ScanDiff instead treats scanpaths as samples from a stochastic distribution conditioned on image and task. This suggests a shift in emphasis from point prediction toward distribution learning.

The model is presented as a unified architecture in which free-viewing is expressed by an empty string condition $c$, whereas visual search uses a target description such as “laptop.” The same training regime and the same architecture are used in both cases; only the conditioning changes. A plausible implication is that the model is intended to absorb differences between exploratory and goal-directed gaze as conditional structure rather than as separate tasks.

Within the broader literature on diffusion-based gaze generation, ScanDiff is described as the first application of diffusion models to both free-viewing and goal-directed scanpaths in natural scenes, whereas previous diffusion gaze work covered reading or 360° imagery. A directly related example is ScanDL, a diffusion model for synthetic scanpaths on texts, which addresses sentence reading rather than natural-scene viewing (Bolliger et al., 2023).

2. Representation and architecture

ScanDiff represents a scanpath as a matrix $\mathbf{s} \in \mathbb{R}^{N \times 3}$, with normalized coordinates $x_i,y_i \in [0,1]$ and fixation duration $m_i$ in seconds. A learned linear projection

$$g_\theta : \mathbb{R}^{N \times 3} \to \mathbb{R}^{N \times d}$$

maps the scanpath to a latent sequence

$$\mathbf{z}_0 = g_\theta(\mathbf{s}) \in \mathbb{R}^{N \times d}.$$

A key design choice is that diffusion is applied in this embedded latent space rather than directly in coordinate space, with the stated purpose of capturing temporal and spatial structure more effectively (Cartella et al., 30 Jul 2025).

The visual backbone is a pre-trained DINOv2 ViT-B/14 with registers. Images are resized to $f_i = (r_i,m_i)$0, patch size is $f_i = (r_i,m_i)$1, the resulting feature map has $f_i = (r_i,m_i)$2 patches, and the visual embedding dimension is $f_i = (r_i,m_i)$3. Task text is encoded by a pre-trained CLIP ViT-B/32 text encoder with $f_i = (r_i,m_i)$4. Visual and textual embeddings are projected to the common latent dimension $f_i = (r_i,m_i)$5, the text embedding is repeated spatially, concatenated with visual features, and projected again to form a joint multimodal representation

$f_i = (r_i,m_i)$6

The denoiser is an encoder-only Transformer with $f_i = (r_i,m_i)$7 layers, 8 attention heads per layer, and hidden dimension $f_i = (r_i,m_i)$8. Each layer comprises self-attention over noisy scanpath tokens, cross-attention with query from scanpath tokens and key/value from $f_i = (r_i,m_i)$9, and a feed-forward network. The model uses learnable positional encodings to encode fixation order and sinusoidal diffusion timestep embeddings to encode the diffusion step. This architecture follows the Transformer-based diffusion design associated in the paper with Peebles and Xie, but its conditioning mechanism is specifically cross-attention rather than raw concatenation of image and scanpath tokens.

A separate reconstruction head $r_i = (x_i,y_i) \in \mathbb{R}^2$0, implemented as three linear layers with ReLU, decodes denoised embeddings back into fixation features $r_i = (x_i,y_i) \in \mathbb{R}^2$1. Variable-length prediction is handled by a validity prediction module

$r_i = (x_i,y_i) \in \mathbb{R}^2$2

which predicts whether each token is valid or padding. The final predicted length is defined as the number of consecutive tokens with high validity at the beginning of the sequence.

Component	Specification	Role
Visual backbone	DINOv2 ViT-B/14 with registers	Extracts image features
Text backbone	CLIP ViT-B/32 text encoder	Encodes task text
Denoiser	Encoder-only Transformer, 6 layers, 8 heads, $r_i = (x_i,y_i) \in \mathbb{R}^2$3	Denoises latent scanpaths
Decoder	Three linear layers with ReLU	Reconstructs fixation features
Length module	Linear validity predictor	Produces variable-length scanpaths

The architectural combination is explicitly multimodal and non-autoregressive. This suggests that diversity is not injected as an add-on after deterministic prediction, but is instead built into the generative mechanism.

3. Diffusion formulation and optimization

The forward process is a Gaussian diffusion applied to the latent scanpath embedding. For a general sample $r_i = (x_i,y_i) \in \mathbb{R}^2$4, the forward transition is

$r_i = (x_i,y_i) \in \mathbb{R}^2$5

In ScanDiff this becomes

$r_i = (x_i,y_i) \in \mathbb{R}^2$6

with a square-root noise schedule and $r_i = (x_i,y_i) \in \mathbb{R}^2$7 steps. After $r_i = (x_i,y_i) \in \mathbb{R}^2$8 steps, $r_i = (x_i,y_i) \in \mathbb{R}^2$9 is approximately isotropic Gaussian. The reverse process is parameterized as

$m_i \in \mathbb{R}^+$0

with $m_i \in \mathbb{R}^+$1, and the denoiser predicts the clean latent $m_i \in \mathbb{R}^+$2 from $m_i \in \mathbb{R}^+$3 and $m_i \in \mathbb{R}^+$4 (Cartella et al., 30 Jul 2025).

The total training loss is

$m_i \in \mathbb{R}^+$5

The diffusion term is a simplified VLB-inspired objective,

$m_i \in \mathbb{R}^+$6

with importance sampling over timesteps following Nichol and Dhariwal. Reconstruction uses an $m_i \in \mathbb{R}^+$7 loss on coordinates and durations,

$m_i \in \mathbb{R}^+$8

while the validity module is trained with binary cross-entropy over padding labels,

$m_i \in \mathbb{R}^+$9

At the terminal step, the prior-alignment term

$c$0

encourages consistency with the Gaussian prior and is reported to improve stability, especially on search tasks.

Training scanpaths are padded or truncated to a maximum length $c$1, with padded positions masked out from reconstruction loss. The reported optimization setup uses AdamW, learning rate $c$2, weight decay $c$3, and batch size 128. Ablations tested $c$4, with $c$5 selected.

The training objective differs from image diffusion in several respects that are explicitly highlighted: the diffused object is a sequence $c$6, temporal order is encoded with positional embeddings, and the loss decomposes into spatial coordinates, fixation durations, and sequence length. The combination of these components indicates that ScanDiff is designed as a structured sequential generator rather than a direct adaptation of image diffusion to gaze coordinates.

4. Unified conditioning for free-viewing and search

Task conditioning is implemented through text. For free-viewing, the condition is the empty string $c$7; for visual search, it is an object name or description such as “fork,” “bowl,” or “keyboard.” The task embedding is produced by CLIP, projected to the latent dimension, tiled over spatial positions, and fused with projected image features to form $c$8. During denoising, scanpath tokens attend to this multimodal representation in every Transformer layer through cross-attention (Cartella et al., 30 Jul 2025).

This mechanism is described as the basis for a unified model of free-viewing and task-driven scanpath generation. In free-viewing, the model relies primarily on bottom-up visual features because there is no strong task signal; in search, denoising is biased toward locations likely to contain the target object. The paper explicitly states that there is no classifier-free guidance scaling or Stable-Diffusion-style text-conditioning trick; guidance is inherent in the architecture through cross-attention and the shared embedding space.

Ablations reported in the paper indicate that CLIP text features are especially effective compared with RoBERTa, attributed to CLIP’s vision–language pretraining. Cross-attention conditioning also outperforms raw concatenation in ablations, with better performance and semantic alignment.

At inference, generation proceeds by encoding $c$9 and $\mathbf{s} \in \mathbb{R}^{N \times 3}$0, sampling

$\mathbf{s} \in \mathbb{R}^{N \times 3}$1

iteratively denoising from $\mathbf{s} \in \mathbb{R}^{N \times 3}$2 to $\mathbf{s} \in \mathbb{R}^{N \times 3}$3, decoding the resulting $\mathbf{s} \in \mathbb{R}^{N \times 3}$4 into $\mathbf{s} \in \mathbb{R}^{N \times 3}$5, predicting validity scores $\mathbf{s} \in \mathbb{R}^{N \times 3}$6, and truncating to the first $\mathbf{s} \in \mathbb{R}^{N \times 3}$7 valid fixations. Because the reverse process is stochastic, repeated sampling from different $\mathbf{s} \in \mathbb{R}^{N \times 3}$8 values yields multiple plausible scanpaths for the same image and task. The sampling strategy is classical DDPM with 1000 steps; no DDIM or ancestral variants are reported.

A common misconception in scanpath prediction is that stronger similarity metrics necessarily imply more human-like behavior. The discussion around ScanDiff rejects that assumption by emphasizing that a model can improve average similarity through a single representative trajectory while still failing to match human variability. In this framework, conditioning and stochastic generation are intended to preserve both task compliance and inter-sample diversity.

5. Evaluation protocol and empirical performance

ScanDiff is evaluated on free-viewing, task-driven search, and zero-shot generalization settings. The free-viewing datasets are COCO-FreeView with 6,202 images and MIT1003 with 1,003 images; training uses the combined COCO-FreeView and MIT1003 data with a 70/15/15 split. Task-driven evaluation uses COCO-Search18, in which 10 subjects perform visual search over 18 categories, with 3,101 target-present and 3,101 target-absent images. OSIE is used for zero-shot free-viewing evaluation and is not used in training (Cartella et al., 30 Jul 2025).

Similarity is measured with MultiMatch, ScanMatch, Sequence Score, and Semantic Sequence Score, all reported as Kullback–Leibler divergences between human–generated and human–human score distributions, so lower is better. Diversity is measured with Recall Sequence Score and the Diversity-aware Sequence Score,

$\mathbf{s} \in \mathbb{R}^{N \times 3}$9

where the denominator penalizes mismatch between generated internal diversity and human internal diversity.

On COCO-FreeView, among models trained under identical settings, ScanDiff achieves the best MM-direction score, very competitive MM-position and MM-average scores, and the best ScanMatch and Sequence Score when fixation durations are considered. On MIT1003, it attains the highest MM-average score among all methods and is particularly strong on duration-aware ScanMatch and Sequence Score. The paper notes that DeepGazeIII sometimes has slightly better spatial-only metrics, but does not cover task-driven settings or diversity analysis in the same way.

On COCO-Search18 target-present search, ScanDiff is reported as state-of-the-art across nearly all metrics. Its MM average KL divergence is $x_i,y_i \in [0,1]$0, corresponding to a 71% reduction versus the next best retrained model, GazeXplain at $x_i,y_i \in [0,1]$1. SemSS is $x_i,y_i \in [0,1]$2 with duration and $x_i,y_i \in [0,1]$3 without duration, both best overall. Sequence Score is $x_i,y_i \in [0,1]$4 with duration and $x_i,y_i \in [0,1]$5 without duration, described as better than or on par with the best baselines.

On target-absent search, which the paper characterizes as more exploratory and more variable, ScanDiff again reports the best MM, SM, SS, and SemSS values: MM average $x_i,y_i \in [0,1]$6; SM $x_i,y_i \in [0,1]$7 with duration and $x_i,y_i \in [0,1]$8 without; SS $x_i,y_i \in [0,1]$9 with duration and $m_i$0 without; SemSS $m_i$1 with duration and $m_i$2 without. Although TPP-Gaze performs slightly better in a few duration-only metrics in this condition, ScanDiff is described as more consistent across metrics.

The diversity analysis is central to the model’s empirical argument. On all datasets and settings, ScanDiff achieves the highest DSS and RSS. The paper gives MIT1003 as an example, with DSS $m_i$3 and RSS $m_i$4, exceeding TPP-Gaze at DSS $m_i$5 and RSS $m_i$6. Qualitative comparisons further report that deterministic models such as Gazeformer often produce nearly identical scanpaths across repeated sampling, whereas ScanDiff produces multiple distinct yet human-like trajectories for the same stimulus.

6. Limitations, interpretation, and research significance

The reported limitations are fourfold. First, ScanDiff models population-level variability rather than personalized gaze behavior; the paper explicitly notes that personality, clinical conditions, and other individual traits are not modeled. Second, the text-conditioning space is limited in the experiments to simple object labels, although more complex conditions such as natural-language queries or multi-step instructions are described as conceptually possible. Third, the use of 1,000 diffusion steps makes inference computationally heavy relative to deterministic models. Fourth, the paper states that deeper analysis of cognitive plausibility remains open, even if the generated scanpaths appear human-like (Cartella et al., 30 Jul 2025).

These limitations define the main future directions identified in the paper: user-specific conditioning, broader vision-and-language integration, and use of the model as a simulator of realistic eye-movement data for training other systems. The authors also suggest speed-up approaches such as DDIM or distillation, but do not explore them experimentally.

In terms of significance, the paper frames ScanDiff as a diffusion-based scanpath model for natural images that unifies free-viewing, target-present search, and target-absent search in a single architecture. Its contributions are stated as the use of diffusion to learn a full stochastic distribution over scanpaths in latent space, explicit prediction of variable-length sequences through token validity, multimodal conditioning via ViTs and CLIP, and diversity-aware evaluation through DSS and RSS. A plausible implication is that ScanDiff repositions scanpath prediction from a deterministic sequence-regression problem to a conditional generative modeling problem in which realism and variability are co-primary objectives.

Relative to earlier diffusion-based gaze modeling, ScanDiff occupies the natural-scene regime, whereas ScanDL operates in reading and generates scanpaths on texts rather than on images (Bolliger et al., 2023). That distinction is methodologically important: ScanDiff integrates DINOv2 image features, CLIP task embeddings, fixation durations, and variable-length prediction, while retaining the more general diffusion premise that scanpaths should be modeled as samples from a learned distribution rather than as a single canonical trajectory.