GazeDETR: End-to-End Gaze Detection

• The paper introduces a DETR-based system that disentangles head localization and gaze prediction using two specialized decoders.
• It leverages a shared ResNet-50 backbone with separate transformer branches to predict human head boxes, gaze targets, and in/out interactions from one image.
• Empirical results on benchmarks such as GazeFollow and VideoAttentionTarget demonstrate state-of-the-art performance with improved mAP and robust gaze detection.

GazeDETR is an end-to-end transformer architecture for multi-person gaze target detection in images, introduced in "GazeDETR: Gaze Detection using Disentangled Head and Gaze Representations" (Belen et al., 18 Aug 2025). Given one RGB image, it predicts a set of triples $$\langle \text{Human}, \text{Interaction}, \text{Gaze Target} \rangle$$, comprising a head bounding box, a gaze target prediction, and an interaction label, especially whether the gaze target is inside or outside the frame. Its defining architectural claim is that head localization and gaze prediction should not share a single entangled decoder representation: the model instead uses two disentangled decoders, with a human head predictor specialized for local information and a gaze decoder that incorporates both local and global information. The name should also be distinguished from "Gaze-DETR: Using Expert Gaze to Reduce False Positives in Vulvovaginal Candidiasis Screening" (Kong et al., 2024), which is a gaze-guided object detection training framework for cytology images rather than a scene gaze target detector.

1. Problem setting and conceptual motivation

GazeDETR studies gaze communication in natural scenes. The task matters for human-robot interaction, social interaction analysis, and digital phenotyping and behavioral assessment. In contrast to older multi-branch gaze-following methods, the model does not require head boxes as input at test time. It is explicitly framed as an end-to-end multi-person system that jointly solves head localization, gaze target prediction, and in/out interaction classification from a single image (Belen et al., 18 Aug 2025).

The paper’s central motivation is that prior end-to-end transformer-based gaze detection methods use a single decoder and therefore learn a shared multitask representation for localizing heads, associating them with gaze targets, and classifying in/out interaction. In the paper’s terminology, this yields an entangled representation. The critique is task-specific rather than generic: head localization wants predominantly local evidence, whereas gaze prediction requires both local cues from the head, face, and eyes and global scene context. A single shared decoder is therefore described as producing inconsistent attentive fields.

This formulation also clarifies what GazeDETR is not. It is not merely a detector with an auxiliary gaze head, nor a two-stream method with externally supplied head crops. It is a DETR-style set predictor in which person-conditioned gaze inference is retained, but the representational burden is distributed across two specialized decoders rather than a single unified one. A plausible implication is that the architecture treats person detection and scene-conditioned gaze reasoning as related but non-isomorphic subproblems.

2. Architectural organization

The model is built around a shared CNN backbone, a dedicated head-localization branch, a separate scene-context branch, and a second decoder for gaze. In the main implementation, the backbone is ResNet-50. The backbone output is

$f_1 \in \mathbb{R}^{C \times H \times W},$

after which two separate $1 \times 1$ convolutions produce

$$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$

The first stream is reserved for head localization, and the second for scene-context encoding (Belen et al., 18 Aug 2025).

The human head predictor applies its own transformer encoder to $z_1$, producing

$e_1 \in \mathbb{R}^{d \times H \times W},$

and a DETR-like transformer decoder then predicts, for each of 100 learned queries, a binary human/not-human classification and a head bounding box. The decoder’s spatial queries $p_q$ are learned positional encodings containing spatial priors such as box center, width, and height. Qualitative analysis in the paper reports that this decoder’s cross-attention is localized around head regions and often within the predicted box, which the authors interpret as evidence that the head branch uses local information.

The Global Scene Feature Encoder (GSFE) processes $z_2$ with its own transformer encoder to produce

$e_2 \in \mathbb{R}^{d \times H \times W}.$

Cross-branch linear projections and additions then yield $m_1$ and $f_1 \in \mathbb{R}^{C \times H \times W},$0, and these are concatenated feature-wise to form the global memory

$f_1 \in \mathbb{R}^{C \times H \times W},$1

The use of concatenation rather than addition is deliberate: the paper states that addition does not ensure separate effective interactions between gaze queries and the two feature sources.

The gaze decoder constitutes the second disentangled decoder. Rather than using a new randomly initialized gaze query, it reuses the head predictor’s learned spatial queries $f_1 \in \mathbb{R}^{C \times H \times W},$2. This design is the key coupling mechanism between the two branches: the head predictor establishes who and where the person is, and the gaze decoder then predicts where that person is looking using the same spatial prior together with the fused global memory. The architecture is therefore disentangled at the decoder level but not disconnected.

The reported hyperparameters are likewise fixed and explicit: the head predictor uses a $f_1 \in \mathbb{R}^{C \times H \times W},$3 convolution, a 6-layer encoder, and a 6-layer decoder; the GSFE uses a $f_1 \in \mathbb{R}^{C \times H \times W},$4 convolution and a 6-layer encoder; the gaze decoder uses a 6-layer decoder and a 3-layer FFN for gaze location; the number of head queries is 100; attention heads are 8; dropout is 0.1; and transformer initialization is Xavier.

3. Decoder interaction, attention structure, and output representation

The most characteristic technical element of GazeDETR is the reuse of head-conditioned spatial queries inside the gaze decoder. In the gaze decoder’s self-attention, with $f_1 \in \mathbb{R}^{C \times H \times W},$5, the key and query are defined as

$f_1 \in \mathbb{R}^{C \times H \times W},$6

so the model replaces a standard randomly learned gaze query with the head predictor’s spatial query (Belen et al., 18 Aug 2025).

Its cross-attention then operates over the fused global memory $f_1 \in \mathbb{R}^{C \times H \times W},$7. The paper gives

$f_1 \in \mathbb{R}^{C \times H \times W},$8

and

$f_1 \in \mathbb{R}^{C \times H \times W},$9

With weight matrices omitted for brevity, the cross-attention contains the terms

$1 \times 1$0

The paper explicitly interprets these three interactions as access to human-head-related features, scene/global features, and positional context, respectively.

The outputs of the second decoder are transformed by a 3-layer FFN into gaze location prediction, and the model also includes in/out classification and optionally heatmap prediction in training. This establishes a representational division of labor. The head decoder predicts binary head presence and head boxes; the gaze decoder predicts gaze location and gaze interaction classification. The paper does not report an explicit disentanglement regularizer or orthogonality loss. Disentanglement is therefore architectural rather than penalty-based.

A recurring misconception addressed by the model design is that scene gaze estimation can be treated as a straightforward extension of object detection attention. The paper argues otherwise: standard DETR-like detection attention often highlights object extremities, which is appropriate for localization but not necessarily for gaze prediction. GazeDETR’s two-decoder structure is meant to correct exactly this mismatch.

4. Optimization, matching, and training protocol

The human head predictor uses a box regression loss

$1 \times 1$1

which combines $1 \times 1$2 and generalized IoU, while the head class loss is described as focal loss. For gaze prediction, the paper uses $1 \times 1$3 loss for 2D gaze coordinate prediction, $1 \times 1$4 loss for heatmap prediction, and binary cross-entropy for in/out classification. The total training loss is written as

$1 \times 1$5

The paper does not provide an explicit disentanglement loss (Belen et al., 18 Aug 2025).

Hungarian bipartite matching is used, but a crucial design distinction is emphasized: the matching cost and the optimization loss are not the same. The manuscript’s printed matching expression is incomplete, but the ablation study states that in the best setup, gaze-related terms are not used in matching; matching relies only on head localization terms. The reason is empirical rather than aesthetic: including gaze terms in matching caused the same head to be matched or predicted multiple times with different gaze outputs, harming mAP badly. This is one of the paper’s clearest technical conclusions.

The training pipeline is fully end-to-end. Images are resized to at least 480 px and at most 800 px in height, with width capped at 1333 px; random horizontal flip is applied with probability 0.5, random crop with probability 0.5, and the shorter side is zero-padded so batch samples have equal size. For video datasets, frames are sampled every 5 frames during training to reduce overfitting. Optimization uses AdamW with weight decay $1 \times 1$6, learning rates $1 \times 1$7 for the backbone and $1 \times 1$8 for the transformers, training from scratch on GazeFollow and then fine-tuning on VideoAttentionTarget and ChildPlay. Supplementary details specify 50 epochs on GazeFollow and 20 epochs of fine-tuning on VideoAttentionTarget or ChildPlay.

5. Datasets, evaluation protocol, and empirical results

GazeDETR is evaluated on GazeFollow, VideoAttentionTarget, and ChildPlay. GazeFollow contains 122k training images, 130k gaze instances, and 4782 test instances. VideoAttentionTarget contains 1,331 videos from 50 TV shows, 71k frames, and 164k gaze instances. ChildPlay contains 401 video clips from 95 YouTube videos, 120k frames, and 257k gaze instances. Across these datasets, the annotations include head bounding boxes, 2D gaze points, and binary in/out labels (Belen et al., 18 Aug 2025).

The evaluation protocol is joint rather than box-only. For end-to-end detection, a prediction is a true positive only if the predicted head box has IoU $1 \times 1$9 with ground truth and the associated gaze point is also correct, defined as $$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$0 distance $$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$1. Reported metrics include AUC, Average Distance or Minimum Distance where appropriate, AP for in/out classification, and mAP for end-to-end head-plus-gaze detection.

The headline results establish state-of-the-art performance on all three benchmarks. On GazeFollow, GazeDETR reports AUC $$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$2, Avg. Dist $$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$3, Min. Dist $$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$4, and mAP $$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$5, compared with HGTTR’s $$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$6, $$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$7, $$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$8, and $$z_1 \in \mathbb{R}^{d \times H \times W}, \qquad z_2 \in \mathbb{R}^{d \times H \times W}.$$9. On VideoAttentionTarget, it reports AUC $z_1$0, Dist $z_1$1, AP $z_1$2, and mAP $z_1$3, compared with HGTTR’s $z_1$4, $z_1$5, $z_1$6, and $z_1$7. On ChildPlay, it reports AUC $z_1$8, Dist $z_1$9, AP $e_1 \in \mathbb{R}^{d \times H \times W},$0, and mAP $e_1 \in \mathbb{R}^{d \times H \times W},$1. The paper further notes that GazeDETR surpasses or matches several methods that are given head boxes as input.

The ablation study on GazeFollow isolates the contributions of the GSFE, head-query reuse, matching strategy, and model scale. Removing the GSFE lowers performance substantially: variant A, which uses no GSFE and the head query, achieves AUC $e_1 \in \mathbb{R}^{d \times H \times W},$2, AvgDist $e_1 \in \mathbb{R}^{d \times H \times W},$3, MinDist $e_1 \in \mathbb{R}^{d \times H \times W},$4, and mAP $e_1 \in \mathbb{R}^{d \times H \times W},$5, while the full model reaches $e_1 \in \mathbb{R}^{d \times H \times W},$6, $e_1 \in \mathbb{R}^{d \times H \times W},$7, $e_1 \in \mathbb{R}^{d \times H \times W},$8, and $e_1 \in \mathbb{R}^{d \times H \times W},$9. Replacing the reused head query with a new gaze query also degrades performance: with GSFE present, variant C reaches mAP $p_q$0 and the full model reaches $p_q$1. The matching ablation is especially consequential: the best model, BM1, uses head localization only for matching and gaze coordinate output during training; models predicting only heatmaps perform very poorly; and adding heatmap supervision can improve AUC while collapsing mAP, as in BM3 with AUC $p_q$2 but mAP $p_q$3. The query-depth study also supports the final configuration: 20 queries with 6 decoders yield mAP $p_q$4, 50 queries with 6 decoders yield $p_q$5, and 100 queries with 6 decoders yield $p_q$6. A plausible implication is that the architecture’s gains are driven more by decoder and query design than by backbone scaling, since moving from ResNet-50 to ResNet-101 changes mAP only from $p_q$7 to $p_q$8.

6. Related methods, naming ambiguity, and limitations

GazeDETR occupies a specific place within end-to-end gaze target detection. A close conceptual neighbor is "GazeHTA: End-to-end Gaze Target Detection with Head-Target Association" (Lin et al., 2024), which also performs multi-person end-to-end prediction from the scene image alone and uses fixed-number outputs with Hungarian matching. However, GazeHTA is not a DETR-style query-decoder architecture in the classic sense: it predicts triplets of heatmaps for heads, gaze targets, and connection maps, uses a pre-trained Stable Diffusion U-Net as a feature extractor, and fixes the proposal count at $p_q$9. The contrast is instructive. GazeHTA emphasizes dense association maps and diffusion-derived scene semantics, whereas GazeDETR emphasizes disentangled decoders, head-query-conditioned gaze decoding, and direct head-box prediction.

A second and more direct source of confusion is nomenclature. "Gaze-DETR" (Kong et al., 2024) is a different method altogether: a gaze-guided object detection training strategy for vulvovaginal candidiasis screening in ThinPrep cytologic test images. It uses expert eye movements to define gaze-only hard negatives and to reduce false positives in DETR-style medical detectors. Despite the similar name, it addresses a different task, uses different supervision, and does not study multi-person social gaze detection.

The limitations reported for GazeDETR are concrete. Failure cases occur in slightly occluded eye regions, in cases with conflicting eye direction and head direction, and for very small heads. The paper interprets these cases as evidence that the model still implicitly mixes head and eye information when estimating gaze. Its clearest stated future direction is therefore explicit eye supervision to further disentangle eye direction from head direction. The paper does not report runtime numbers, FLOPs, or latency; any claim about computational efficiency beyond the stated architecture would therefore be speculative.

Within the current literature, GazeDETR’s significance lies in showing that end-to-end gaze detection does not need to accept a single-decoder bottleneck. Its results indicate that architectural separation between head localization and scene-conditioned gaze prediction can improve both attention behavior and benchmark performance. This suggests that, for scene gaze modeling, the decisive design variable is not merely end-to-end training itself, but the internal allocation of local and global evidence across the model’s representational pathways.