GAZED: Gaze-Aware Systems Overview

• GAZED is a family of gaze-aware methods that integrate structured visual attention signals, such as heatmaps and gaze points, across various domains.
• Research in GAZED employs techniques ranging from early feature integration to object-aware geometric reasoning, yielding measurable performance improvements.
• Applications span gaze-guided cinematic editing, fixation-driven segmentation, and 3D gaze estimation, enhancing human-computer interaction in multiple fields.

GAZED is not a single standardized term in the arXiv literature. In some works it is the exact name of gaze-guided cinematic editing systems; in others, closely related names such as GazeD denote a specific 3D gaze-estimation model; and in many papers, “gazed” simply refers to the object, region, or point being visually attended by a person. Across these usages, the common technical core is the estimation or exploitation of visual attention as a structured signal: a heatmap, a gaze point, a segmentation mask, a bounding box, a 3D vector, a gaze target, or a shot-selection prior (Moorthy et al., 2020, Achary et al., 2023, Catalini et al., 19 Jan 2026, Hanifi et al., 2023).

1. Terminological scope and core definitions

Within the cited literature, “GAZED” has at least three distinct statuses. First, it appears as the exact name of cinematic systems, most notably "GAZED- Gaze-guided Cinematic Editing of Wide-Angle Monocular Video Recordings" and "Real Time GAZED: Online Shot Selection and Editing of Virtual Cameras from Wide-Angle Monocular Video Recordings," where gaze is used to guide virtual camera selection and editing (Moorthy et al., 2020, Achary et al., 2023). Second, "GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation" uses a near-homographic name for a diffusion-based model that jointly estimates 3D pose and 3D gaze from a single RGB image (Catalini et al., 19 Jan 2026). Third, several papers explicitly note that they do not define “GAZED” as a named dataset or algorithm; instead, they use “gazed” in the descriptive sense of the object being looked at, or of gaze-grounded prediction more generally (Hanifi et al., 2023, Tonini et al., 2023, Inadumi et al., 2024).

This terminological heterogeneity matters because the underlying research problems differ. In tabletop human–robot interaction, the target is often the gazed object on a table. In in-the-wild gaze target detection, the task expands to the gaze heatmap, the precise pixel point, an in/out-of-frame decision, and the class and location of the gazed object. In 3D settings, the problem becomes one of regressing or retrieving a gaze ray or a 3D target under geometric, privacy, or multimodal constraints (Hanifi et al., 2023, Tonini et al., 2023, Toaiari et al., 2024).

Taken together, these works suggest that GAZED is best understood not as a single canonical benchmark or architecture, but as a family of gaze-aware formulations in which attention is treated as an object-aware, context-sensitive, and often geometrically constrained variable.

2. Gazed-object perception in tabletop interaction and fixation-driven segmentation

A concrete instance of gazed-object detection appears in tabletop human–robot interaction with the iCub humanoid robot. The system in "iCub Detecting Gazed Objects: A Pipeline Estimating Human Attention" is an online, vision-only pipeline with three pathways: Human Attention Estimation, Object Detection, and Attentive Object Detection. Human attention is estimated by combining OpenPose, a face recognition/detection module from Lombardi et al. (2022), and a Visual Target Detection model based on a spatio-temporal CNN with a ConvLSTM adapted from Chong et al. (2020). The final gazed object is selected by extracting the hottest region from the attention heatmap and matching it to detected object boxes by overlap via IoU, or by nearest-center fallback when there is no overlap (Hanifi et al., 2023).

The same work introduced the ObjectAttention dataset for tabletop HRI. It contains 250 videos, 22,732 frames, and 10 participants (4F/6M), with 5 sessions per participant, object counts increasing from 1 to 5, and a “pringles” distractor placed in at least one trial per session but never gazed at. Fine-tuning the VTD model on ObjectAttention improved performance from AUC = 87.5% ± 0.9 and L2 distance = 0.131 ± 0.014 to AUC = 92.5% ± 1.9 and L2 distance = 0.089 ± 0.014. Full-pipeline top-1 accuracy decreased slightly with clutter but remained around 70% even in the most cluttered settings, while the distractor caused errors in ~3% of frames in which it was present (Hanifi et al., 2023).

A more fine-grained formulation is fixation-driven object segmentation. "Personal Fixations-Based Object Segmentation with Object Localization and Boundary Preservation" defines PFOS as mapping an RGB image $I \in \mathbb{R}^{H\times W \times 3}$ and a personal fixation map to a binary mask $M \in \{0,1\}^{H\times W}$. Sparse fixation points are converted into a fixation density map by

$$FDM = nor\_min\text{-}max(FM \circledast G_\sigma(x,y;\sigma)),$$

with $\sigma$ corresponding to 1° visual angle (24 px at 800×600). The proposed OLBP network combines an Object Localization Module, which interprets the fixation prior via multi-scale dilated convolutions and feature re-weighting, with a Boundary Preservation Module, which predicts and reinjects boundary cues to reduce background leakage (Li et al., 2021).

The PFOS dataset contains 700 images at 800×600, each with fixation maps from 15 subjects, for a total of 10,500 personal fixation maps. It is intentionally challenging: 35.1% constrained and 64.9% unconstrained fixation maps. On the test set, OLBP achieved IoU 73.7%, S-measure 81.1%, weighted F 80.0%, E-measure 88.7%, and F-measure 84.3%, outperforming 17 state-of-the-art baselines and degrading only modestly under added fixation noise (Li et al., 2021).

These two lines of work represent complementary formulations of gazed-object perception. The iCub pipeline maps gaze to a discrete tabletop object through heatmap-to-box fusion, whereas PFOS maps gaze to a dense object mask through fixation-conditioned segmentation. A plausible implication is that later systems can interpolate between these endpoints by using gaze first for object proposal selection and then for boundary refinement.

3. Object-aware gaze target detection in natural scenes

The move from pixel-level heatmap regression to object-aware reasoning is explicit in "Object-aware Gaze Target Detection." That model predicts, for each person in an image, a gaze heatmap $H$, a precise gaze point $g=(x,y)$, a watch-outside probability $p_{out}$, and the class and bounding box of the gazed object. Its architecture combines an Object Detector Transformer, a Gaze Cone Predictor, and a Gaze Object Transformer. The detector jointly finds heads and scene objects, the cone predictor estimates a per-head 2D/3D cone with a fixed binocular field of view angle $\alpha = 120^\circ$, and GOT uses cone-consistent masking and an object score matrix $\Sigma$ to model head–object and head–head associations (Tonini et al., 2023).

This formulation yields a more explicitly explainable output than heatmap-only methods. On GazeFollow, the RGB-only model achieved AUC = 0.922, average distance = 0.072, and minimum distance = 0.033. On VideoAttentionTarget, it achieved AUC = 0.923, distance = 0.102, and out-of-frame AP = 0.944. The paper also reports up to +2.91% AUC, ≈50% reduction in gaze distance, +9% out-of-frame AP, and +11–13% AP gains for gazed-object classification and localization (Tonini et al., 2023).

A related but more compact formulation appears in "Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning." This work argues that pixel-level regression is unstable because human gaze is fundamentally object-oriented. It therefore uses a two-stage framework: object-level semantic encoding with object masks from YOLO11x and SAM2-hiera-large, followed by geometry-guided localization based on a 2D gaze direction, a field-of-view prior, and multi-scale feature fusion from a frozen DINOv3 ViT-L/16 backbone. The field-of-view mechanism thresholds directional responses at a best-performing $\theta = 120^\circ$, and the training loss combines heatmap BCE, a direction loss $M \in \{0,1\}^{H\times W}$0, and an in/out loss where available (Mi et al., 28 Jun 2026).

The resulting model reports AUC of 0.961 on GazeFollow, 0.948 on VideoAttentionTarget, 0.987 on ChildPlay, and 0.977 on GOO-Real, while maintaining 7.1M trainable parameters. The ablations indicate that object-aware encoding, multi-scale fusion, and geometry are all useful, with deep features dominant and shallow features refining localization (Mi et al., 28 Jun 2026).

Both papers replace purely spatial heatmap regression with object-centric structure. In the former, the structure is learned by explicit head–object association in a DETR-like architecture; in the latter, it is induced through object masks, semantic tokens, and a field-of-view prior. This suggests a broader shift in GAZED-style systems from “where is the peak?” to “which entity is being selected, under what geometric constraints?”

4. Three-dimensional gaze, privacy preservation, and diffusion-based modeling

Three-dimensional gaze target detection introduces explicit geometry and, increasingly, privacy constraints. "Upper-Body Pose-based Gaze Estimation for Privacy-Preserving 3D Gaze Target Detection" avoids using face images at inference time. Its inputs are a scene RGB image with blurred face, an aligned depth map, a head location mask, 2D/3D eye positions, and 13 upper-body joints from hip upward. The method predicts a unit 3D gaze direction, constructs alignment heatmaps

$M \in \{0,1\}^{H\times W}$1

and retrieves the 3D target by maximizing ray–point alignment in a local neighborhood around the 2D heatmap argmax (Toaiari et al., 2024).

The benchmark is GFIE, with 71,799 RGB-D frames, 61 subjects, and non-overlapping train/validation/test splits of 59,217 / 6,281 / 6,281. On the GFIE test set, the privacy-preserving method achieved 0.284 m, 15.9°, 0.983, 0.083 for 3D distance, 3D angle error, 2D AUC, and 2D distance in the multi-stage regime, improving over the GFIE baseline 0.311 m, 17.7°, 0.965, 0.065 on both 3D metrics while slightly worsening 2D distance. The paper reports that the full privacy configuration outperformed variants using full 17-joint skeletons or unblurred faces without head crops on the 3D metrics (Toaiari et al., 2024).

A different 3D strategy appears in "GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation." GazeD jointly denoises the 3D skeleton and gaze by representing gaze as an additional body joint at a fixed distance from the eye midpoint:

$M \in \{0,1\}^{H\times W}$2

with

$M \in \{0,1\}^{H\times W}$3

Conditioning is provided by 2D pose and HRNet features, surroundings via deformable attention, and scene objects via an RT-DETR-like detector. The diffusion process uses a DDIM scheduler, N=20 denoising steps, and H=20 hypotheses, with a coordinate-space MSE loss over pose and gaze joints (Catalini et al., 19 Jan 2026).

Across three benchmarks, GazeD reports strong 3D accuracy. On GAFA, the default H=20, AVG aggregation gives 19.5 / 20.5 for 3D/2D mean angular error; on GFIE it gives 13.6 3D MAE; and on Ego-Gaze it gives 17.5 / 17.4 for 3D/2D MAE. Oracle aggregation is substantially lower, for example 9.9 on GFIE, which indicates large residual diversity across plausible hypotheses (Catalini et al., 19 Jan 2026).

The common direction of these 3D systems is not merely geometric reconstruction. They combine geometry with context, and they do so under different design pressures: privacy preservation in one case, and uncertainty-aware multimodal conditioning in the other.

5. Domain-specific datasets and applications

A major feature of GAZED research is its proliferation into domain-specific benchmarks in which the gaze target is semantically and operationally constrained. In retail, "GOO: A Dataset for Gaze Object Prediction in Retail Environments" defines gaze object prediction as returning the bounding box of the object being looked at. GOO contains 201,552 images across GOO-Synth and GOO-Real, with 24 product classes and approximately 80 grocery items per image in the real environment. GOO-Real has 9,552 images from 100 volunteers, while GOO-Synth contains 192,000 synthetic images rendered in Unreal Engine from 50 virtual cameras and 20 synthetic humans. Synthetic pretraining transfers effectively: for Chong et al.’s baseline, GOO-Real 5-shot performance improved from 0.796 / 0.252 / 51.4° without pretraining to 0.889 / 0.150 / 29.1° with pretraining for AUC, distance, and angular error (Tomas et al., 2021).

In intelligent vehicles, "What Do You See in Vehicle? Comprehensive Vision Solution for In-Vehicle Gaze Estimation" introduced IVGaze, a dataset of 44,705 infrared face images from 125 subjects, together with the dual-stream GazeDPTR model. The dataset uses a DMS camera at 1280×800, targets distributed across driving-relevant zones, and subject-independent three-fold cross-validation. GazeDPTR combines original and perspective-normalized face streams, fuses them with camera-pose-aware transformers, and reaches 6.71° mean angular error, improving over the single-stream GazePTR at 7.04° and over other baselines such as XGaze at 7.06° and GazeTR at 7.33° (Cheng et al., 2024).

In situated language understanding, "A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions" introduced GazeVQA with 17,276 question–answer pairs over 10,760 images, split 13,785 / 1,811 / 1,860, with 10 distinct answers per test question. The task is to answer ambiguous Japanese questions using the questioner’s gaze. Adapter-based conditioning on gaze-informed regions improved the model when only the adapter was trained: + Adapter (GT) achieved Acc 40.09, Bs 82.01, whereas the baseline ClipCap under full fine-tuning achieved Acc 36.80, Bs 81.75 (Inadumi et al., 2024).

In egocentric multimodal modeling, "GazeLLM: Multimodal LLMs incorporating Human Visual Attention" uses gaze to crop first-person video around a 448×448 region centered at gaze from a 1440×1440 frame, reducing the pixels to approximately 10.3% of the full image. Across 137 videos from Ego-Exo4D, human evaluation found that Full and Gaze were significantly better than Center (p < 0.05), with no significant difference between Full and Gaze, and Gaze had the highest average score in 5 of 6 tasks (Rekimoto, 31 Mar 2025).

In medical imaging, "GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans" contributes 346 FDG-PET/CT scans, 3,948 minutes of raw 60Hz eye-tracking data, and 9,030 unique gaze-to-lesion trajectories. The dataset supports 3D nnUNet segmentation with gaze as an additional channel, improving Dice from 0.6008 to 0.6819, and transformer-based dynamic lesion localization, where 74.95% of predicted gaze points were closer to the lesion than the raw last gaze point (Wu et al., 26 Feb 2026).

These domain-specific systems differ in sensor modality, annotation granularity, and downstream task, but all treat gaze as a privileged signal for resolving ambiguity: among densely packed products, among multiple dashboard zones, among omitted Japanese referents, among high-resolution egocentric regions, or among clinically relevant lesions.

6. GAZED as gaze-guided cinematic editing

The exact term GAZED is most literally instantiated in computational cinematography. "GAZED- Gaze-guided Cinematic Editing of Wide-Angle Monocular Video Recordings" turns a single static wide-angle recording into an edited video by first generating virtual shots and then selecting among them through gaze-guided energy minimization. For $M \in \{0,1\}^{H\times W}$4 performers, it considers all non-empty actor combinations, yielding $M \in \{0,1\}^{H\times W}$5 rushes at each time. Single-actor shots are framed as Medium Shot or Medium Close-Up, while multi-actor shots are Full Shot. Editing is formulated as

$M \in \{0,1\}^{H\times W}$6

with pairwise cost terms for transitions, overlap, and rhythm. The optimization is solved by dynamic programming after an initial 4-second establishing shot (Moorthy et al., 2020).

The paper reports a psychophysical study with 12 users and 12 performance videos, evaluated on Narrational Effectiveness, Scene Actions, Actor Emotions, and Viewing Experience. Two-way balanced ANOVA showed a main effect of editing strategy with p < 0.000001 for all four measures, and post-hoc tests showed that GAZED significantly outperformed competing strategies on multiple comparisons, including Viewing Experience where it was better than Speaker-based (p < 0.01), Greedy Gaze (p < 0.005), Wide (p < 0.05), and Random (p < 0.000001) (Moorthy et al., 2020).

"Real Time GAZED" adapts the framework to streaming operation. It integrates CineFilter, with CineConvex and CineCNN variants for trajectory stabilization, uses ByteTrack for actor tracking, limits shot generation to $M \in \{0,1\}^{H\times W}$7 combinations by spatial neighborhood, and replaces global offline optimization with an online dynamic program with bounded look-ahead and a continuity term. Agreement with offline GAZED is reported as ~85% with minimal look-ahead $M \in \{0,1\}^{H\times W}$8 and ~98% with $M \in \{0,1\}^{H\times W}$9. CineConvex runs at ~250 fps with ~0.5 s latency, while CineCNN runs at ~1000 fps and is used in the real-time system (Achary et al., 2023).

The cinematic branch of GAZED differs sharply from object detection and 3D gaze estimation, but it preserves the same principle: gaze acts as an operational proxy for interest. Here, the output is not a mask or box, but an edited sequence of shots constrained by cinematographic continuity.

7. Recurring themes, methodological tensions, and broader implications

Several recurring themes connect these otherwise heterogeneous uses of gaze. One is the contrast between early integration and late integration. "GazeDPM: Early Integration of Gaze Information in Deformable Part Models" formalizes gaze as an additional feature channel inside a DPM score,

$$FDM = nor\_min\text{-}max(FM \circledast G_\sigma(x,y;\sigma)),$$0

and reports that this early integration raised mAP on POET from 30.4% for the DPM baseline and 30.8% for late integration to 34.7%, while also exposing learned gaze-attracting and gaze-repelling structures (Shcherbatyi et al., 2015). This same tension reappears in modern work as the difference between heatmap post-processing and object-aware joint modeling.

A second theme is that gaze can be a supervision signal even when it is unavailable at inference. "Towards Making a Dependency Parser See" uses gaze features as auxiliary tasks in a multitask sequence-labeling parser, with no gaze input at test time. The reported gains are described as modest but positive, both in parallel and disjoint-data regimes, suggesting that gaze can regularize shared representations beyond vision (Strzyz et al., 2019).

A third theme concerns the origin of gaze patterns themselves. "Why We Look Where We Look: Emergent Human-like Fixations of a Foveated Visual LLM Maximizing Scene Understanding" shows that a foveated RL agent trained for scene comprehension exhibits hallmark human fixation signatures: early center bias, frequent fixations on people and text, and prioritization of objects being gazed at or grasped. The paper reports that fRL-SU matched human fixation frequency patterns better than saliency models, DeepGaze, and task-optimized controls with p < 0.0001, while an entropy-minimization variant matched or slightly surpassed fRL-SU (p = 0.001) (Murlidaran et al., 18 May 2026).

These results do not collapse the different meanings of GAZED into a single framework, but they do delimit a coherent research program. Attention can be estimated, segmented, localized, denoised, fused with object semantics, regularized by geometry, used to edit video, or used only during training. The persistent technical question is how much structure should be imposed on gaze: as a point, as a field, as a box, as an object token, as a 3D ray, as an auxiliary loss, or as a proxy for human relevance.