---
title: Diffusion-Based Gaze Synthesis
url: https://www.emergentmind.com/topics/diffusion-based-gaze-synthesis
type: topic
---

# Diffusion-Based Gaze Synthesis

Diffusion-based gaze synthesis refers to a family of generative frameworks that employ denoising diffusion probabilistic models (DDPMs) to synthesize, impute, manipulate, or redirectionally control human gaze time-series or gaze-conditioned imagery. These models leverage the iterative stochastic inversion of a forward noising process to generate complex, temporally coherent and perceptually realistic gaze trajectories or images. Diffusion approaches have demonstrated state-of-the-art fidelity and diversity in continuous gaze sequence generation, subject-aware bio-signal synthesis, scanpath simulation, image-conditioned gaze vector synthesis, 3D gaze redirection, imputation under missing data, and the incorporation of gaze as a semantic control for downstream visual synthesis tasks.

## 1. Mathematical Foundations of Diffusion-Based Gaze Synthesis

Diffusion-based gaze synthesis methods operate by defining a forward noising process $q(x_t \mid x_{t-1}) = \mathcal N(\sqrt{1-\beta_t} x_{t-1}, \beta_t I)$ for $t=1\dots T$ on the clean gaze trajectory $x_0$ (which may be a sequence of gaze vectors, velocities, or higher-order temporally organized gaze features) [2403.17477][2409.01240][2511.09867][2507.23021][2511.02468]. The process transforms $x_0$ into an isotropic Gaussian latent $x_T$ by repeated application of stochastic Gaussian noise. The reverse denoising process, parameterized by a neural model $\epsilon_\theta$, iteratively reconstructs a synthetic gaze sequence by sampling from a series of conditional Gaussians of the form
$$
p_\theta(x_{t-1} \mid x_t, c) = \mathcal N\left(\mu_\theta(x_t, t, c), \sigma_t^2 I \right)
$$
with explicit dependence on the conditioning variable $c$ (such as user embeddings, image features, or auxiliary sensor data). The objective minimized during training is an MSE (simplified score matching) loss between the true and predicted noise:
$$
\mathcal L = \mathbb E_{x_0,\, t,\, \epsilon}\left\|\epsilon - \epsilon_\theta(x_t, t, c)\right\|_2^2
$$
where $x_t$ is constructed as $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon$.

Many diffusion-based gaze synthesis models augment this with specialized conditioning mechanisms—for example, user identity embeddings (for personalized sequence generation [2409.01240][2511.09867]), image/scene embeddings (image- or scanpath-conditioned generation [2403.17477][2507.23021]), auxiliary sensor tokens (for motion-based imputation [2511.02468]), or explicit orthogonality constraints (for disentangling gaze from other facial or behavioral factors [2511.11231]). Some models introduce additional losses, such as cosine similarity in a user-authenticator embedding space, to enforce subject specificity [2409.01240][2511.09867].

## 2. Key Model Architectures and Conditioning Strategies

Diffusion-based gaze synthesis encompasses a variety of architectural paradigms, unified by the core denoising framework but differentiated in their encoder, conditioner, and denoiser network design:

- **DiffGaze** employs a 1D residual backbone with dual transformers (temporal and spatial) to model continuous gaze sequences on 360° images, conditioned on Spherical-CNN extracts from equirectangular environments [2403.17477].
- **DiffEyeSyn** and its successors utilize a DiffWave-style or U-Net-based denoising network, with conditioning on both identity-stripped gaze dynamics and subject-identity vectors from EKYT authenticator models [2409.01240][2511.09867][2601.21057]. Feature-wise linear modulation (FiLM) and per-block user embedding injection are commonly employed.
- **ScanDiff** fuses Vision Transformer (ViT) feature maps from the stimulus image and language-derived task encoding at every transformer layer to stochastically generate scanpaths as sequences of fixation-state tokens [2507.23021].
- **HAGI++** demonstrates a multi-modal transformer denoiser, using cross-modal self-attention and FiLM layers to fuse time-aligned head orientation and wrist motion signals with gaze data for robust imputation and generation under arbitrary missingness [2511.02468].
- **TextGaze** cascades a sketch diffusion module driven by CLIP-based attention embeddings with a sketch-conditioned face image diffusion module, enabling generation of facial images with specified gaze from free-form textual input [2404.17486].
- **3D Gaussian/DiT-Gaze** combines a head/eye-parameterized 3D Gaussian splatting front-end with a diffusion transformer renderer for fine-grained, disentangled 3D gaze redirection [2511.11231].
- **RadGazeGen** introduces ControlNet-style adapters to condition chest X-ray image diffusion on expert gaze heatmaps and radiomic feature maps [2410.00307].

Most models optimize conditioning fusion via residual connections, cross-attention, or explicit concatenation, and exploit geometric tokenization, time/frequency encoding, and domain-specific normalization (e.g., down-up sampling, velocity clamping, sin-map scaling) in the gaze feature space.

## 3. Applications and Problem Domains

The scope of diffusion-based gaze synthesis includes:

| Application Domain         | Representative Model(s) | Conditioning Modalities                    |
|---------------------------|------------------------|-------------------------------------------|
| Continuous sequence generation | DiffGaze, DiffEyeSyn        | Image, user-embedding, scanpath           |
| User-specific synthesis         | DiffEyeSyn, updated DiffEyeSyn | Identity embedding, dynamics              |
| Scanpath prediction, saliency   | ScanDiff, DiffGaze           | Image, text/task, scene                   |
| Gaze imputation/generation      | HAGI++                      | Head/wrist motion, partial gaze           |
| Controllable gaze face synthesis| TextGaze                     | Text, sketch, 3D pose, facial landmarks   |
| Gaze-guided image synthesis     | RadGazeGen                   | Gaze heatmap, radiomics, text             |
| 3D gaze redirection             | DiT-Gaze                     | 3D Gaussian splatting, latent codes       |

Use cases include large-scale data augmentation for calibration-free tracking and biometrics [2409.01240][2511.09867], privacy-preserving surrogate data [2601.21057], robust scanpath modeling for visual attention studies [2507.23021][2403.17477], and multimodal medical image generation guided by expert radiologist attention [2410.00307]. Advanced models such as HAGI++ enable recovery of natural gaze kinematics even under extreme (100%) data loss by leveraging multimodal wearable inputs [2511.02468].

## 4. Empirical Evaluation and Metrics

Quantitative and qualitative benchmarks are standardized across the literature, with task-specific metrics:

- **Trajectory similarity:** Dynamic Time Warping (DTW), Levenshtein (LEV), Mean Absolute Error (MAE), Root Mean Squared Error (RMSE) for continuous sequence matching [2403.17477][2511.09867].
- **Temporal and scanpath structure:** MultiMatch, ScanMatch, Sequence Score (SS), Semantic SS, Recurrence measures [2507.23021].
- **User specificity:** Cosine similarity in authenticator embedding space [2409.01240][2511.09867].
- **Perceptual/realism:** User studies, median human-vs-synth rating, Fréchet Inception Distance (FID), Inception Score (IS), CLIP-score [2403.17477][2404.17486][2410.00307].
- **Saliency maps:** Normalized Scanpath Saliency (NSS), AUC, CC, SIM, KL divergence for saliency evaluation [2403.17477].
- **Gaze-redirection accuracy:** Angular error of gaze estimator on generated images [2511.11231][2404.17486].
- **Imputation/generation:** Mean angular error (MAE), Jensen–Shannon divergence of gaze velocity distributions in imputation/generation settings [2511.02468].

Models such as DiffGaze narrow the gap to human agreement in continuous scanpath (DTW ↓ 20% over previous SoTA) and match human-level saccade/fixation statistics [2403.17477]. Subject-specific diffusion (DiffEyeSyn variants) reduce spatial error by a factor of 4× over GANs and reach cosine embedding similarity ≈0.92–0.95, near the human within-subject level [2409.01240][2511.09867]. 3D-aware DiT-Gaze achieves gaze-redirection errors of 6.353° (↓4.1% vs. 3DGS) and increased perceptual scores [2511.11231]. HAGI++ attains 25% reduction in MAE compared to strong interpolation baselines and accurate velocity-profile matching, even when gaze is fully missing and inferred purely from head/wrist [2511.02468]. Gaze-guided image synthesis (RadGazeGen) improves localization and structure agreement (SSIM up to 0.7045) and attains lower FID relative to prior multi-control diffusion methods [2410.00307].

## 5. User Specificity, Privacy, and State Signal Attenuation

Recent diffusion-based gaze synthesis frameworks explicitly address the dual requirements of subject specificity and privacy. The introduction of user identity guidance losses and pre-trained authenticator-based embeddings allows fine control over idiosyncratic oculomotor signatures [2409.01240][2511.09867][2601.21057]. Updated models demonstrate that, while high-level kinematic realism and subject-identity cues are preserved, internal state features (such as fatigue or mental workload) are suppressed: correlations between synthetic gaze features and subjective reports drop to non-significant levels, contrasting with original real-gaze signals where $\rho \sim 0.2–0.4$ (p < 0.05) can be detected for state-dependent attributes [2601.21057]. This property enables the use of synthetic gaze for privacy-preserving benchmarking, AR/VR simulation, and data dissemination where state leakage is a concern.

## 6. Emerging Directions: Multimodality, Control, and Synthesis in Context

Contemporary research extends diffusion-based gaze synthesis to encompass multimodal and task-aware synthesis frameworks:

- **Text and semantic control** is effected via cross-attention on CLIP-derived features from free-form text, enabling natural language to control facial gaze in photo-realistic synthesis without direct gaze-value annotation [2404.17486].
- **Image and scanpath co-conditioning** allows for the generation of plausible scanpaths modulated by both visual features and explicit task cues (object reference, search instructions) via transformer cross-modality [2507.23021][2403.17477].
- **Medical imaging applications** now exploit gaze-encoded attention (radiologist fixation heatmaps) as a spatial prior in text-to-image diffusion, improving the anatomical plausibility and regional fidelity of synthesized diagnostics [2410.00307].

Pure gaze sequence generation, as in HAGI++, demonstrates robust gaze inference under substantially missing input, leveraging auxiliary inertial or limb-mounted sensors (head, wrist) for behavioral context [2511.02468]. 3D Gaussian/diffusion fusion (DiT-Gaze) provides a differentiable, geometry-aware representation for fine-grained gaze redirection, with orthogonality constraints for disentangling gaze from expression/pose [2511.11231].

## 7. Limitations and Current Challenges

Despite significant advances, key methodological and practical challenges remain:

- **Dependency on domain-specific embeddings:** User-specific synthesis relies on the representational power of authenticator models (e.g., EKYT) and could be limited by their discriminative granularity [2409.01240][2601.21057].
- **Computational and data costs:** High-resolution diffusion synthesis, especially for images or large datasets, remains computationally intensive [2410.00307][2511.11231]. Gaze conditioning data can be expensive or unavailable for certain modalities (e.g., 3D medical imaging).
- **Generalization:** Cross-domain robustness—transfer to novel settings (AR/VR, head-mounted tracks, lower sampling frequencies)—requires new architectures and foundation models [2409.01240][2410.00307].
- **Interpretability:** While gaze synthesis can mask internal state cues, the interpretability and controllability of synthesized output, particularly in privacy-sensitive scenarios, requires further quantification [2601.21057].

Further research is investigating larger and more robust user-embedding extractors ("foundation" user-authenticator analogues), temporal integrations for multimodal controls, extension to AR/VR environments, and compositional generative frameworks for gaze, motion, and semantic context.

---

**References**

- DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images [2403.17477]
- DiffEyeSyn: Diffusion-based User-specific Eye Movement Synthesis [2409.01240]
- Quantitative and Qualitative Comparison of Generative Models for Subject-Specific Gaze Synthesis: Diffusion vs GAN [2511.09867]
- Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction [2507.23021]
- HAGI++: Head-Assisted Gaze Imputation and Generation [2511.02468]
- TextGaze: Gaze-Controllable Face Generation with Natural Language [2404.17486]
- 3D Gaussian and Diffusion-Based Gaze Redirection [2511.11231]
- RadGazeGen: Radiomics and Gaze-guided Medical Image Generation using Diffusion Models [2410.00307]
- Privatization of Synthetic Gaze: Attenuating State Signatures in Diffusion-Generated Eye Movements [2601.21057]

Source: https://www.emergentmind.com/topics/diffusion-based-gaze-synthesis