Uncertainty-based Gaze Selection (UGS)

• The paper demonstrates that UGS improves multi-view gaze target estimation by selecting the gaze vector with the lowest predicted uncertainty via deterministic geometric transformation.
• It integrates an uncertainty-aware angular loss with a hard selection rule to replace less reliable predictions before generating field-of-view heatmaps.
• Experimental ablations show that UGS reduces angular error by up to 3–4% and boosts accuracy metrics, especially when head visibility varies between views.

Uncertainty-based Gaze Selection (UGS) is the multi-view fusion mechanism introduced in "Multi-view Gaze Target Estimation" for gaze target estimation (GTE) from a pair of synchronized cameras (Miao et al., 7 Aug 2025). In that formulation, each view predicts a 3D gaze vector $\mathbf{g}_i$ and a scalar uncertainty $\sigma_i$; UGS selects the gaze vector from the view with lower predicted uncertainty, transforms it into the other camera coordinate system using the relative rotations, and replaces the less reliable prediction. The refined vectors are then used to generate field-of-view (FoV) heatmaps and downstream gaze target heatmaps. UGS is therefore a prediction-level, geometry-aware reliability selector rather than a feature-level fusion block or an averaging scheme.

1. Problem setting and operational role

The underlying task is multi-view GTE. For each view $i \in \{1,2\}$, the inputs are an RGB scene image $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$, a subject head bounding box $\mathbf{x}^{box}_i \in \mathbb{R}^4$, camera intrinsics $\mathbf{K}_i \in \mathbb{R}^{3\times 3}$, camera extrinsics given by rotation $\mathbf{R}_i \in \mathbb{R}^{3\times 3}$ and translation $\mathbf{t}_i \in \mathbb{R}^{1\times 3}$, and a monocular depth map $\mathbf{D}_i$. One view is designated primary and the other reference. The final output per view consists of a 2D gaze target heatmap $\mathbf{H}_i \in \mathbb{R}^{1\times 64 \times 64}$ and a scalar in-frame probability $\sigma_i$0 (Miao et al., 7 Aug 2025).

Within the full pipeline, UGS appears after per-view 3D gaze prediction and before FoV heatmap generation and scene encoding. The sequence is: head extraction; Head Information Aggregation (HIA); per-view gaze estimation with uncertainty; UGS; FoV heatmap generation; multi-view scene encoding with Epipolar-based Scene Attention (ESA); and transformer decoding into $\sigma_i$1 and $\sigma_i$2. Its explicit role is to ensure that both views use the most reliable 3D gaze vector available before downstream scene reasoning.

This placement is central to the paper’s multi-view design. The broader method is presented as addressing limitations of single-view GTE, including face occlusion, target ambiguity, and out-of-view targets. It also reports that the full multi-view system can estimate the gaze target in the first view using the image of the person in the second view only, a capability not possessed by single-view GTE methods (Miao et al., 7 Aug 2025). A plausible implication is that UGS is most valuable when the two cameras observe markedly different head visibility conditions.

2. Formal definition and uncertainty model

For each view $\sigma_i$3, the gaze head predicts a 3D gaze vector $\sigma_i$4 and a scalar uncertainty $\sigma_i$5. The uncertainty is aleatoric uncertainty, parameterized per view by a single scalar. The gaze and uncertainty are trained jointly with an uncertainty-aware angular loss:

$\sigma_i$6

The first term is an angular loss based on cosine similarity, weighted by inverse variance, and the second term regularizes $\sigma_i$7. The paper identifies this as the classic heteroscedastic aleatoric regression loss of Kendall and Gal. It further states that no Monte Carlo sampling, ensembles, or dropout-based epistemic estimation are used; uncertainty comes directly from a deterministic prediction branch (Miao et al., 7 Aug 2025).

UGS then applies a hard selection rule:

$\sigma_i$8

If $\sigma_i$9 and $i \in \{1,2\}$0, the selected gaze vector is transferred into the other camera frame by

$i \in \{1,2\}$1

The output of the module is the pair $i \in \{1,2\}$2, in which both refined vectors come from the lower-uncertainty view up to rigid rotation. The paper explicitly characterizes this as hard view selection with deterministic geometric fusion, not averaging or probabilistic combination (Miao et al., 7 Aug 2025).

A common misconception is that UGS fuses uncertainty at the feature level. It does not. The inputs to UGS are only $i \in \{1,2\}$3, and the module itself has no trainable parameters. Another common misconception is that $i \in \{1,2\}$4 expresses model uncertainty. The paper is explicit that it is aleatoric uncertainty, intended to reflect view quality factors such as occlusion, poor view, and motion blur.

3. Architectural placement and interaction with HIA and ESA

The upstream head pathway begins by cropping the head image $i \in \{1,2\}$5 and forming a head mask $i \in \{1,2\}$6. A ResNet-18 backbone extracts head features, which are then processed by HIA. HIA flattens head features into tokens and performs cross-attention across views using the relative rotation $i \in \{1,2\}$7. Average pooling over the attended tokens yields the enhanced head embedding $i \in \{1,2\}$8. From $i \in \{1,2\}$9, a small MLP or FC heads predict $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$0 and $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$1 (Miao et al., 7 Aug 2025).

The structural distinction among the three main modules can be summarized as follows.

Module	Operating level	Function
HIA	Head-level features	Cross-view head information aggregation
UGS	3D gaze vectors and uncertainty scalars	Reliability-based hard selection and transfer
ESA	Scene feature tokens	Cross-view epipolar scene attention

UGS is therefore a rule-based layer inserted between head reasoning and scene reasoning. After UGS, the refined gaze vectors $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$2, the estimated eye 3D location, and the depth map $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$3 are used to generate FoV heatmaps $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$4. These heatmaps are concatenated with the RGB image and head mask as $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$5 and passed through a shared ViT-based scene encoder with ESA. ESA uses epipolar geometry through the fundamental matrix $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$6 to sample scene tokens along corresponding epipolar lines and perform cross-attention (Miao et al., 7 Aug 2025).

The directional dependencies are explicit. HIA can improve UGS indirectly by improving the quality of $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$7, and hence the quality of $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$8 and $\mathbf{I}_i \in \mathbb{R}^{3\times H \times W}$9. UGS, in turn, affects the scene branch through the FoV priors supplied to the encoder. ESA does not feed back into UGS. This suggests that UGS is the point where head-derived reliability is converted into a geometric prior for scene-based target localization.

4. Supervision, optimization, and implicit calibration

The full model is trained end-to-end with

$\mathbf{x}^{box}_i \in \mathbb{R}^4$0

where $\mathbf{x}^{box}_i \in \mathbb{R}^4$1 is the gaze target heatmap loss, $\mathbf{x}^{box}_i \in \mathbb{R}^4$2 is the in/out classification loss, and $\mathbf{x}^{box}_i \in \mathbb{R}^4$3 is the uncertainty-aware gaze loss. The supplementary reports typical values $\mathbf{x}^{box}_i \in \mathbb{R}^4$4, $\mathbf{x}^{box}_i \in \mathbb{R}^4$5, and $\mathbf{x}^{box}_i \in \mathbb{R}^4$6 between 0.05 and 0.3 depending on scene. Occluded targets are labeled as “inside” for $\mathbf{x}^{box}_i \in \mathbb{R}^4$7 (Miao et al., 7 Aug 2025).

The pseudo ground-truth 3D gaze vector $\mathbf{x}^{box}_i \in \mathbb{R}^4$8 is constructed from camera intrinsics and a monocular depth map. For a pixel $\mathbf{x}^{box}_i \in \mathbb{R}^4$9 and depth $\mathbf{K}_i \in \mathbb{R}^{3\times 3}$0, the 3D point in camera coordinates is

$\mathbf{K}_i \in \mathbb{R}^{3\times 3}$1

Using the eye location $\mathbf{K}_i \in \mathbb{R}^{3\times 3}$2 and ground-truth target pixel $\mathbf{K}_i \in \mathbb{R}^{3\times 3}$3, the paper defines the pseudo ground-truth gaze direction as the vector from the eye point to the target point. It further notes that because the depth estimator is monocular and up to scale, directions remain valid up to normalization even if absolute depth is uncalibrated (Miao et al., 7 Aug 2025).

UGS itself is not trained directly, because it has no parameters. Its effectiveness depends on two learned or given quantities: the learned relationship between gaze error and $\mathbf{K}_i \in \mathbb{R}^{3\times 3}$4 induced by $\mathbf{K}_i \in \mathbb{R}^{3\times 3}$5, and the calibrated camera extrinsics used for the rigid transformation. The training schedule is likewise staged: a single-view version without HIA, UGS, and ESA is first trained on GazeFollow; then on MVGT, the gaze estimator including $\mathbf{K}_i \in \mathbb{R}^{3\times 3}$6 is trained first, and finally the full multi-view model with HIA, UGS, and ESA is fine-tuned (Miao et al., 7 Aug 2025). This suggests that UGS relies on implicit supervision of uncertainty rather than explicit uncertainty labels.

5. Empirical behavior and ablation evidence

In the main comparison between the single-view and full multi-view systems, the largest gains occur when the reference view provides usable head information. For the “Head visible, target visible in reference” case, Dist improves from 0.151 to 0.129 and AP from 0.877 to 0.909 when moving from “Ours-Single” to “Ours”. For “Head visible, target not visible in reference,” Dist improves from 0.148 to 0.122 and AP from 0.878 to 0.912 (Miao et al., 7 Aug 2025).

The UGS-specific ablation isolates an additional benefit beyond uncertainty prediction and HIA. In the “Head visible / Target visible in reference” regime, the progressive sequence is: “$\mathbf{K}_i \in \mathbb{R}^{3\times 3}$7 only” with Dist 0.145, “$\mathbf{K}_i \in \mathbb{R}^{3\times 3}$8” with Dist 0.135, and “$\mathbf{K}_i \in \mathbb{R}^{3\times 3}$9” with Dist 0.130. The paper summarizes this as a further $\mathbf{R}_i \in \mathbb{R}^{3\times 3}$0–$\mathbf{R}_i \in \mathbb{R}^{3\times 3}$1 relative error reduction on top of HIA in that regime (Miao et al., 7 Aug 2025).

The supplementary further reports a challenging subset where the primary view has large initial gaze error, defined as angular error $\mathbf{R}_i \in \mathbb{R}^{3\times 3}$2. Without UGS, this subset yields Dist 0.228, AP 0.856, and angular error $\mathbf{R}_i \in \mathbb{R}^{3\times 3}$3. With UGS, the corresponding numbers are Dist 0.200, AP 0.883, and angular error $\mathbf{R}_i \in \mathbb{R}^{3\times 3}$4. This is the clearest direct evidence that UGS is especially effective when one view’s gaze estimate is poor.

The uncertainty signal used by UGS is also empirically validated. The supplementary shows that bins of larger $\mathbf{R}_i \in \mathbb{R}^{3\times 3}$5 correspond to larger angular error, and that about 93% of samples have $\mathbf{R}_i \in \mathbb{R}^{3\times 3}$6, within which the slope remains clear: higher $\mathbf{R}_i \in \mathbb{R}^{3\times 3}$7 corresponds to larger error. Qualitative examples show the primary view having a badly misaligned gaze vector with large $\mathbf{R}_i \in \mathbb{R}^{3\times 3}$8, while the reference view has a better vector with smaller $\mathbf{R}_i \in \mathbb{R}^{3\times 3}$9; after replacement through UGS, the FoV heatmap and final prediction align better with the ground truth (Miao et al., 7 Aug 2025).

These results constrain the interpretation of UGS. It is not merely a convenience layer attached to a multi-view model; its main contribution is to correct high-error per-view gaze estimates before they contaminate the FoV prior that drives downstream target localization.

6. Assumptions, limitations, extensions, and broader context

UGS rests on several explicit assumptions. It assumes that the learned $\mathbf{t}_i \in \mathbb{R}^{1\times 3}$0 is monotonic with gaze error, that the Gaussian noise model underlying $\mathbf{t}_i \in \mathbb{R}^{1\times 3}$1 is adequate for the angular loss, that per-view gaze predictions can be compared independently before selection, and that the camera extrinsics used in $\mathbf{t}_i \in \mathbb{R}^{1\times 3}$2 are accurate (Miao et al., 7 Aug 2025). The paper notes robustness to small jitter, but the transformation still presumes calibrated rotations.

Its limitations are equally explicit. Because UGS uses hard selection, only one view’s gaze vector is used per pair and potentially useful information from the other view is discarded at the vector level. Because there is no threshold heuristic, if both views are poor the module still selects the lesser of two evils. Because uncertainty is represented by a single scalar $\mathbf{t}_i \in \mathbb{R}^{1\times 3}$3 per view, directional or anisotropic uncertainty cannot be expressed. And because the supervision depends on pseudo-3D gaze vectors derived from monocular depth, uncertainty calibration can be affected by noisy gaze supervision (Miao et al., 7 Aug 2025).

The paper sketches several extensions. For more than two views, it proposes forming multiple primary-reference pairs, computing the selected uncertainty $\mathbf{t}_i \in \mathbb{R}^{1\times 3}$4 for each pair, and choosing the pair with minimal $\mathbf{t}_i \in \mathbb{R}^{1\times 3}$5. It also suggests soft fusion through uncertainty-weighted averaging,

$\mathbf{t}_i \in \mathbb{R}^{1\times 3}$6

as well as richer Bayesian models with covariance matrices and combinations of aleatoric and epistemic uncertainty, for example via dropout or ensembles. A temporal extension over video is also suggested (Miao et al., 7 Aug 2025).

In broader uncertainty-aware gaze research, several adjacent formulations illuminate what is distinctive about UGS. "EyeSeg: An Uncertainty-Aware Eye Segmentation Framework for AR/VR" outputs an uncertainty score and segmentation result for weighting and fusing multiple gaze estimates for robustness under motion blur, eyelid occlusion, and cross-domain challenges (Peng et al., 13 Jul 2025). "Bayesian Eye Tracking" estimates a full gaze distribution and gaze covariance by Monte Carlo integration over landmark and weight posteriors (Ji et al., 2021). "Enhancing accuracy of uncertainty estimation in appearance-based gaze tracking with probabilistic evaluation and calibration" argues that uncalibrated 95% confidence intervals in appearance-based gaze tracking can capture only 30–60% of ground-truth gaze angles, and proposes isotonic probability calibration to correct that behavior (Zheng et al., 24 Jan 2025). These works indicate that uncertainty can enter gaze systems at different levels—segmentation, landmark inference, gaze regression, or post-hoc calibration—whereas UGS is specifically a non-parametric, geometry-aware selector operating on per-view 3D gaze vectors.

A separate but related line of work uses uncertainty to decide where to look next rather than which gaze prediction to trust. "Active Gaze Control for Foveal Scene Exploration" selects the next fixation point by minimizing expected uncertainty in a semantic map (Dias et al., 2022), and "Uncertainty-Driven Active Vision for Implicit Scene Reconstruction" chooses the next view by maximizing perspective-conditioned occupancy uncertainty (Smith et al., 2022). By contrast, UGS in multi-view GTE does not control sensor motion or scanpaths; it resolves disagreement among already available viewpoints. That distinction is methodologically important: UGS is a reliability selector for synchronized multi-camera inference, not an active perception policy.