---
title: 'GAZED: Gaze-Aware Systems Overview'
url: https://www.emergentmind.com/topics/gazed
type: topic
---

# GAZED: Gaze-Aware Systems Overview

GAZED is not a single standardized term in the arXiv literature. In some works it is the exact name of gaze-guided cinematic editing systems; in others, closely related names such as GazeD denote a specific 3D gaze-estimation model; and in many papers, “gazed” simply refers to the object, region, or point being visually attended by a person. Across these usages, the common technical core is the estimation or exploitation of visual attention as a structured signal: a heatmap, a gaze point, a segmentation mask, a bounding box, a 3D vector, a gaze target, or a shot-selection prior [2010.11886] [2311.15581] [2601.12948] [2308.13318].

## 1. Terminological scope and core definitions

Within the cited literature, “GAZED” has at least three distinct statuses. First, it appears as the exact name of cinematic systems, most notably "GAZED- Gaze-guided Cinematic Editing of Wide-Angle Monocular Video Recordings" and "Real Time GAZED: Online Shot Selection and Editing of Virtual Cameras from Wide-Angle Monocular Video Recordings," where gaze is used to guide virtual camera selection and editing [2010.11886] [2311.15581]. Second, "GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation" uses a near-homographic name for a diffusion-based model that jointly estimates 3D pose and 3D gaze from a single RGB image [2601.12948]. Third, several papers explicitly note that they do **not** define “GAZED” as a named dataset or algorithm; instead, they use “gazed” in the descriptive sense of the object being looked at, or of gaze-grounded prediction more generally [2308.13318] [2307.09662] [2403.17545].

This terminological heterogeneity matters because the underlying research problems differ. In tabletop human–robot interaction, the target is often the **gazed object** on a table. In in-the-wild gaze target detection, the task expands to the gaze heatmap, the precise pixel point, an in/out-of-frame decision, and the class and location of the gazed object. In 3D settings, the problem becomes one of regressing or retrieving a gaze ray or a 3D target under geometric, privacy, or multimodal constraints [2308.13318] [2307.09662] [2409.17886].

Taken together, these works suggest that GAZED is best understood not as a single canonical benchmark or architecture, but as a family of gaze-aware formulations in which attention is treated as an object-aware, context-sensitive, and often geometrically constrained variable.

## 2. Gazed-object perception in tabletop interaction and fixation-driven segmentation

A concrete instance of gazed-object detection appears in tabletop human–robot interaction with the iCub humanoid robot. The system in "iCub Detecting Gazed Objects: A Pipeline Estimating Human Attention" is an online, vision-only pipeline with three pathways: Human Attention Estimation, Object Detection, and Attentive Object Detection. Human attention is estimated by combining OpenPose, a face recognition/detection module from Lombardi et al. (2022), and a Visual Target Detection model based on a spatio-temporal CNN with a ConvLSTM adapted from Chong et al. (2020). The final gazed object is selected by extracting the hottest region from the attention heatmap and matching it to detected object boxes by overlap via IoU, or by nearest-center fallback when there is no overlap [2308.13318].

The same work introduced the ObjectAttention dataset for tabletop HRI. It contains **250 videos**, **22,732 frames**, and **10 participants (4F/6M)**, with **5 sessions per participant**, object counts increasing from **1 to 5**, and a “pringles” distractor placed in at least one trial per session but never gazed at. Fine-tuning the VTD model on ObjectAttention improved performance from **AUC = 87.5% ± 0.9** and **L2 distance = 0.131 ± 0.014** to **AUC = 92.5% ± 1.9** and **L2 distance = 0.089 ± 0.014**. Full-pipeline top-1 accuracy decreased slightly with clutter but remained **around 70% even in the most cluttered settings**, while the distractor caused errors in **~3% of frames** in which it was present [2308.13318].

A more fine-grained formulation is fixation-driven object segmentation. "Personal Fixations-Based Object Segmentation with Object Localization and Boundary Preservation" defines PFOS as mapping an RGB image $I \in \mathbb{R}^{H\times W \times 3}$ and a personal fixation map to a binary mask $M \in \{0,1\}^{H\times W}$. Sparse fixation points are converted into a fixation density map by
$$
FDM = nor\_min\text{-}max(FM \circledast G_\sigma(x,y;\sigma)),
$$
with $\sigma$ corresponding to **1° visual angle (24 px at 800×600)**. The proposed OLBP network combines an Object Localization Module, which interprets the fixation prior via multi-scale dilated convolutions and feature re-weighting, with a Boundary Preservation Module, which predicts and reinjects boundary cues to reduce background leakage [2101.09014].

The PFOS dataset contains **700 images at 800×600**, each with fixation maps from **15 subjects**, for a total of **10,500 personal fixation maps**. It is intentionally challenging: **35.1% constrained** and **64.9% unconstrained** fixation maps. On the test set, OLBP achieved **IoU 73.7%**, **S-measure 81.1%**, **weighted F 80.0%**, **E-measure 88.7%**, and **F-measure 84.3%**, outperforming 17 state-of-the-art baselines and degrading only modestly under added fixation noise [2101.09014].

These two lines of work represent complementary formulations of gazed-object perception. The iCub pipeline maps gaze to a discrete tabletop object through heatmap-to-box fusion, whereas PFOS maps gaze to a dense object mask through fixation-conditioned segmentation. A plausible implication is that later systems can interpolate between these endpoints by using gaze first for object proposal selection and then for boundary refinement.

## 3. Object-aware gaze target detection in natural scenes

The move from pixel-level heatmap regression to object-aware reasoning is explicit in "Object-aware Gaze Target Detection." That model predicts, for each person in an image, a gaze heatmap $H$, a precise gaze point $g=(x,y)$, a watch-outside probability $p_{out}$, and the class and bounding box of the gazed object. Its architecture combines an Object Detector Transformer, a Gaze Cone Predictor, and a Gaze Object Transformer. The detector jointly finds heads and scene objects, the cone predictor estimates a per-head 2D/3D cone with a fixed binocular field of view angle $\alpha = 120^\circ$, and GOT uses cone-consistent masking and an object score matrix $\Sigma$ to model head–object and head–head associations [2307.09662].

This formulation yields a more explicitly explainable output than heatmap-only methods. On GazeFollow, the RGB-only model achieved **AUC = 0.922**, **average distance = 0.072**, and **minimum distance = 0.033**. On VideoAttentionTarget, it achieved **AUC = 0.923**, **distance = 0.102**, and **out-of-frame AP = 0.944**. The paper also reports **up to +2.91% AUC**, **≈50% reduction in gaze distance**, **+9% out-of-frame AP**, and **+11–13% AP gains** for gazed-object classification and localization [2307.09662].

A related but more compact formulation appears in "Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning." This work argues that pixel-level regression is unstable because human gaze is fundamentally object-oriented. It therefore uses a two-stage framework: object-level semantic encoding with object masks from **YOLO11x** and **SAM2-hiera-large**, followed by geometry-guided localization based on a 2D gaze direction, a field-of-view prior, and multi-scale feature fusion from a frozen **DINOv3 ViT-L/16** backbone. The field-of-view mechanism thresholds directional responses at a best-performing **$\theta = 120^\circ$**, and the training loss combines heatmap BCE, a direction loss $L_{dir} = 1 - \hat g \cdot g^{gt}$, and an in/out loss where available [2606.29334].

The resulting model reports **AUC of 0.961 on GazeFollow, 0.948 on VideoAttentionTarget, 0.987 on ChildPlay, and 0.977 on GOO-Real**, while maintaining **7.1M** trainable parameters. The ablations indicate that object-aware encoding, multi-scale fusion, and geometry are all useful, with deep features dominant and shallow features refining localization [2606.29334].

Both papers replace purely spatial heatmap regression with object-centric structure. In the former, the structure is learned by explicit head–object association in a DETR-like architecture; in the latter, it is induced through object masks, semantic tokens, and a field-of-view prior. This suggests a broader shift in GAZED-style systems from “where is the peak?” to “which entity is being selected, under what geometric constraints?”

## 4. Three-dimensional gaze, privacy preservation, and diffusion-based modeling

Three-dimensional gaze target detection introduces explicit geometry and, increasingly, privacy constraints. "Upper-Body Pose-based Gaze Estimation for Privacy-Preserving 3D Gaze Target Detection" avoids using face images at inference time. Its inputs are a scene RGB image with blurred face, an aligned depth map, a head location mask, 2D/3D eye positions, and **13 upper-body joints from hip upward**. The method predicts a unit 3D gaze direction, constructs alignment heatmaps
$$
V_i = \hat{\mathbf g}\cdot \frac{\mathbf q_i}{\|\mathbf q_i\|_2}, \qquad \hat V_i = \mathrm{ReLU}(V_i)^\alpha,\ \alpha=3,
$$
and retrieves the 3D target by maximizing ray–point alignment in a local neighborhood around the 2D heatmap argmax [2409.17886].

The benchmark is GFIE, with **71,799 RGB-D frames**, **61 subjects**, and non-overlapping train/validation/test splits of **59,217 / 6,281 / 6,281**. On the GFIE test set, the privacy-preserving method achieved **0.284 m, 15.9°, 0.983, 0.083** for 3D distance, 3D angle error, 2D AUC, and 2D distance in the multi-stage regime, improving over the GFIE baseline **0.311 m, 17.7°, 0.965, 0.065** on both 3D metrics while slightly worsening 2D distance. The paper reports that the full privacy configuration outperformed variants using full 17-joint skeletons or unblurred faces without head crops on the 3D metrics [2409.17886].

A different 3D strategy appears in "GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation." GazeD jointly denoises the 3D skeleton and gaze by representing gaze as an additional body joint at a fixed distance from the eye midpoint:
$$
y^{Gaze} = e + d\cdot \hat v,\qquad d=0.30\ \text{meters},
$$
with
$$
\hat v = \frac{y^{Gaze}-e}{\|y^{Gaze}-e\|}.
$$
Conditioning is provided by 2D pose and HRNet features, surroundings via deformable attention, and scene objects via an RT-DETR-like detector. The diffusion process uses a **DDIM scheduler**, **N=20** denoising steps, and **H=20** hypotheses, with a coordinate-space MSE loss over pose and gaze joints [2601.12948].

Across three benchmarks, GazeD reports strong 3D accuracy. On GAFA, the default **H=20, AVG** aggregation gives **19.5 / 20.5** for 3D/2D mean angular error; on GFIE it gives **13.6** 3D MAE; and on Ego-Gaze it gives **17.5 / 17.4** for 3D/2D MAE. Oracle aggregation is substantially lower, for example **9.9** on GFIE, which indicates large residual diversity across plausible hypotheses [2601.12948].

The common direction of these 3D systems is not merely geometric reconstruction. They combine geometry with context, and they do so under different design pressures: privacy preservation in one case, and uncertainty-aware multimodal conditioning in the other.

## 5. Domain-specific datasets and applications

A major feature of GAZED research is its proliferation into domain-specific benchmarks in which the gaze target is semantically and operationally constrained. In retail, "GOO: A Dataset for Gaze Object Prediction in Retail Environments" defines gaze object prediction as returning the bounding box of the object being looked at. GOO contains **201,552 images** across GOO-Synth and GOO-Real, with **24 product classes** and approximately **80 grocery items per image** in the real environment. GOO-Real has **9,552 images** from **100 volunteers**, while GOO-Synth contains **192,000 synthetic images** rendered in Unreal Engine from **50 virtual cameras** and **20 synthetic humans**. Synthetic pretraining transfers effectively: for Chong et al.’s baseline, GOO-Real **5-shot** performance improved from **0.796 / 0.252 / 51.4°** without pretraining to **0.889 / 0.150 / 29.1°** with pretraining for AUC, distance, and angular error [2105.10793].

In intelligent vehicles, "What Do You See in Vehicle? Comprehensive Vision Solution for In-Vehicle Gaze Estimation" introduced IVGaze, a dataset of **44,705 infrared face images** from **125 subjects**, together with the dual-stream GazeDPTR model. The dataset uses a DMS camera at **1280×800**, targets distributed across driving-relevant zones, and subject-independent three-fold cross-validation. GazeDPTR combines original and perspective-normalized face streams, fuses them with camera-pose-aware transformers, and reaches **6.71°** mean angular error, improving over the single-stream GazePTR at **7.04°** and over other baselines such as XGaze at **7.06°** and GazeTR at **7.33°** [2403.15664].

In situated language understanding, "A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions" introduced GazeVQA with **17,276 question–answer pairs** over **10,760 images**, split **13,785 / 1,811 / 1,860**, with **10 distinct answers per test question**. The task is to answer ambiguous Japanese questions using the questioner’s gaze. Adapter-based conditioning on gaze-informed regions improved the model when only the adapter was trained: **+ Adapter (GT)** achieved **Acc 40.09, Bs 82.01**, whereas the baseline ClipCap under full fine-tuning achieved **Acc 36.80, Bs 81.75** [2403.17545].

In egocentric multimodal modeling, "GazeLLM: Multimodal LLMs incorporating Human Visual Attention" uses gaze to crop first-person video around a **448×448** region centered at gaze from a **1440×1440** frame, reducing the pixels to **approximately 10.3%** of the full image. Across **137 videos** from Ego-Exo4D, human evaluation found that **Full and Gaze were significantly better than Center (p < 0.05)**, with **no significant difference between Full and Gaze**, and Gaze had the highest average score in **5 of 6 tasks** [2504.00221].

In medical imaging, "GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans" contributes **346 FDG-PET/CT scans**, **3,948 minutes of raw 60Hz eye-tracking data**, and **9,030 unique gaze-to-lesion trajectories**. The dataset supports 3D nnUNet segmentation with gaze as an additional channel, improving Dice from **0.6008** to **0.6819**, and transformer-based dynamic lesion localization, where **74.95%** of predicted gaze points were closer to the lesion than the raw last gaze point [2603.00162].

These domain-specific systems differ in sensor modality, annotation granularity, and downstream task, but all treat gaze as a privileged signal for resolving ambiguity: among densely packed products, among multiple dashboard zones, among omitted Japanese referents, among high-resolution egocentric regions, or among clinically relevant lesions.

## 6. GAZED as gaze-guided cinematic editing

The exact term GAZED is most literally instantiated in computational cinematography. "GAZED- Gaze-guided Cinematic Editing of Wide-Angle Monocular Video Recordings" turns a single static wide-angle recording into an edited video by first generating virtual shots and then selecting among them through gaze-guided energy minimization. For $n$ performers, it considers all non-empty actor combinations, yielding $2^n-1$ rushes at each time. Single-actor shots are framed as Medium Shot or Medium Close-Up, while multi-actor shots are Full Shot. Editing is formulated as
$$
E(\epsilon) = \sum_{t=1}^T -ln(G(r_t)) + \sum_{t=2}^T E_e(r_{t-1},r_t),
$$
with pairwise cost terms for transitions, overlap, and rhythm. The optimization is solved by dynamic programming after an initial **4-second** establishing shot [2010.11886].

The paper reports a psychophysical study with **12 users** and **12 performance videos**, evaluated on Narrational Effectiveness, Scene Actions, Actor Emotions, and Viewing Experience. Two-way balanced ANOVA showed a main effect of editing strategy with **p < 0.000001** for all four measures, and post-hoc tests showed that GAZED significantly outperformed competing strategies on multiple comparisons, including Viewing Experience where it was better than Speaker-based **(p < 0.01)**, Greedy Gaze **(p < 0.005)**, Wide **(p < 0.05)**, and Random **(p < 0.000001)** [2010.11886].

"Real Time GAZED" adapts the framework to streaming operation. It integrates **CineFilter**, with **CineConvex** and **CineCNN** variants for trajectory stabilization, uses **ByteTrack** for actor tracking, limits shot generation to **$n(n+1)/2$** combinations by spatial neighborhood, and replaces global offline optimization with an online dynamic program with bounded look-ahead and a continuity term. Agreement with offline GAZED is reported as **~85% with minimal look-ahead $f=32$** and **~98% with $f=128$**. CineConvex runs at **~250 fps** with **~0.5 s latency**, while CineCNN runs at **~1000 fps** and is used in the real-time system [2311.15581].

The cinematic branch of GAZED differs sharply from object detection and 3D gaze estimation, but it preserves the same principle: gaze acts as an operational proxy for interest. Here, the output is not a mask or box, but an edited sequence of shots constrained by cinematographic continuity.

## 7. Recurring themes, methodological tensions, and broader implications

Several recurring themes connect these otherwise heterogeneous uses of gaze. One is the contrast between **early integration** and **late integration**. "GazeDPM: Early Integration of Gaze Information in Deformable Part Models" formalizes gaze as an additional feature channel inside a DPM score,
$$
S(x, H, z) = \langle \beta, \Phi(x,z) \rangle + \langle \beta', \Phi'(H,z) \rangle,
$$
and reports that this early integration raised mAP on POET from **30.4%** for the DPM baseline and **30.8%** for late integration to **34.7%**, while also exposing learned gaze-attracting and gaze-repelling structures [1505.05753]. This same tension reappears in modern work as the difference between heatmap post-processing and object-aware joint modeling.

A second theme is that gaze can be a supervision signal even when it is unavailable at inference. "Towards Making a Dependency Parser See" uses gaze features as auxiliary tasks in a multitask sequence-labeling parser, with no gaze input at test time. The reported gains are described as **modest but positive**, both in parallel and disjoint-data regimes, suggesting that gaze can regularize shared representations beyond vision [1909.01053].

A third theme concerns the origin of gaze patterns themselves. "Why We Look Where We Look: Emergent Human-like Fixations of a Foveated Visual Language Model Maximizing Scene Understanding" shows that a foveated RL agent trained for scene comprehension exhibits hallmark human fixation signatures: early center bias, frequent fixations on people and text, and prioritization of objects being gazed at or grasped. The paper reports that fRL-SU matched human fixation frequency patterns better than saliency models, DeepGaze, and task-optimized controls with **p < 0.0001**, while an entropy-minimization variant **matched or slightly surpassed fRL-SU (p = 0.001)** [2605.17823].

These results do not collapse the different meanings of GAZED into a single framework, but they do delimit a coherent research program. Attention can be estimated, segmented, localized, denoised, fused with object semantics, regularized by geometry, used to edit video, or used only during training. The persistent technical question is how much structure should be imposed on gaze: as a point, as a field, as a box, as an object token, as a 3D ray, as an auxiliary loss, or as a proxy for human relevance.

Source: https://www.emergentmind.com/topics/gazed