GLARIFY: Resolving Gaze Ambiguity in VLMs

• GLARIFY is a method that integrates noisy, spatiotemporal gaze data with video and language inputs to resolve ambiguous user queries in smart-glasses applications.
• It employs an automatically synthesized training corpus (GLARIFY-Ambi) and a heatmap-based mechanism to inject and align gaze signals within pretrained vision-language models.
• Empirical results demonstrate that GLARIFY boosts GPT-Accuracy and human evaluation scores by effectively reasoning about when to trust gaze signals.

GLARIFY is a method for resolving ambiguity in gaze-facilitated visual assistant interaction, motivated by the integration of users’ attention signals into vision-LLMs for smart-glasses and related multimodal interfaces. It addresses two coupled failure modes: verbal underspecification, in which users rely on pronouns or omitted context, and gaze unreliability, in which human attention is noisy and exhibits complex spatiotemporal relationships with speech. The method combines an automatically synthesized training corpus, GLARIFY-Ambi, with a heatmap-based mechanism for injecting gaze into a pretrained video-capable VLM while preserving the model’s pretrained knowledge (Wang et al., 26 Sep 2025).

1. Problem setting and conceptual scope

GLARIFY is situated in the setting of gaze-facilitated querying, where a user asks a multimodal question while wearing a device that can provide visual input and gaze traces. In this setting, ambiguity does not arise only from language. It also arises because gaze can be delayed, irrelevant, or distributed across multiple objects and times, so the attentional signal is not a stable proxy for referential intent. The paper identifies two principal challenges: ambiguous user queries and noisy, spatiotemporally complex gaze patterns (Wang et al., 26 Sep 2025).

The method is framed against a limitation of earlier gaze-grounded approaches: previous works only consider single image as visual modality input, which fails to capture the dynamic nature of the user’s attention in video and real-life scenarios. The paper further notes that simply mapping all gaze traces to one keyframe is ineffective for dynamic contexts such as AR/VR smart glasses. In this formulation, gaze is treated not as a hard pointer but as a temporally distributed, error-prone signal that must be interpreted jointly with video and language. This suggests that GLARIFY’s main contribution is not merely gaze conditioning, but explicit modeling of ambiguity in the attention channel itself (Wang et al., 26 Sep 2025).

2. Characterization of gaze ambiguity and noise

A central empirical premise of GLARIFY is that human gaze is systematically noisy during natural querying. The paper reports an analysis of hundreds of querying samples with the gaze modality and formalizes irrelevant attention by the ratio

$$R(t) = \frac{N_{\text{irr}(t)}}{N_{\text{total}(t)}}$$

where $N_{\text{irr}(t)}$ denotes the number of fixations not aligned with the intended referent at time ratio $t$, and $N_{\text{total}(t)}$ denotes the total number of fixations. The reported finding is that $R(t) > 20\%$ throughout querying and can reach $50\%$ near the end (Wang et al., 26 Sep 2025).

This characterization directly counters a common misconception in gaze-grounded interaction: that eye movements transparently reveal the target of the question. In the formulation adopted by GLARIFY, gaze may contain irrelevant fixations, delayed attention shifts, and non-linear scan paths. The method is therefore designed to disambiguate queries by combining vision with noisy, temporally rich gaze data, rather than assuming that the gaze stream already identifies the answer-bearing region. The paper explicitly states that the framework analyzes and accounts for the noisy nature of users’ gaze patterns and the fact that late-query fixations may be irrelevant (Wang et al., 26 Sep 2025).

3. GLARIFY-Ambi: synthetic data generation for ambiguous gaze-grounded QA

Because labeled gaze+video+QA corpora are scarce, GLARIFY introduces an automatic data synthesis pipeline, implemented with GPT-4o, to generate GLARIFY-Ambi. The source material comes from Video Localized Narratives and the Oops! subset, providing video keyframes $V_i$, human narrations $N_i$, and mouse traces $T_i$, with mouse traces used as proxies for gaze. The dataset generation process has three stages: QA pair generation, mouse-trace modification to inject gaze noise, and chain-of-thought generation (Wang et al., 26 Sep 2025).

In the first stage, GPT-4o is prompted to create three QA pairs for each narrative, one each for detail, action, and complex reasoning. For each, it produces both direct and indirect versions of the question, with the indirect version often using pronouns, together with an answer. In the second stage, the traces are modified to simulate realistic ambiguity. Spatial noise is injected by corrupting one trace with an irrelevant trace from another background or actor. Temporal noise is injected by filling missing traces using semantically relevant actors or objects, mimicking continuous tracking. In the third stage, GPT-4o generates a natural-language CoT rationale. The paper distinguishes spatial CoT, which resolves referents and discards semantically misaligned gaze, from temporal CoT, which reasons over event order and attention shifts. Each synthesized sample stores the direct question, indirect question, answer, modified video frames, modified traces, and CoT (Wang et al., 26 Sep 2025).

The paper states that Table 1 reports over 70,000 training QA pairs, with more than 99% survival after augmentation and filtering. System prompts are described as explicitly modeling AR or assistant settings, requiring indirect ambiguous questions, and generating reasoning-rich CoTs. Within the overall method, GLARIFY-Ambi functions not only as supervision for answer prediction but also as supervision for deciding when gaze should be trusted and when it should be discounted (Wang et al., 26 Sep 2025).

4. Heatmap-based spatiotemporal integration into a pretrained VLM

GLARIFY represents the user’s gaze as a sequence of heatmaps aligned to video keyframes. If video frames are denoted by $I_t$, then the associated gaze heatmaps are $N_{\text{irr}(t)}$0, and the gaze signal is represented as a spatiotemporal tensor

$N_{\text{irr}(t)}$1

The base model is Qwen2.5-Omni-3B, chosen as a large VLM supporting unified input across video, images, text, and audio. The architectural intervention is deliberately minimal: the paper emphasizes preserving pretrained knowledge while injecting gaze information into the vision pathway (Wang et al., 26 Sep 2025).

The video frames and gaze heatmaps are both patchified:

$N_{\text{irr}(t)}$2

$N_{\text{irr}(t)}$3

Each gaze patch is then projected into the same latent space as image patches using a small learnable linear layer,

$N_{\text{irr}(t)}$4

while the image patches are encoded by the vision transformer,

$N_{\text{irr}(t)}$5

Fusion is performed additively at the patch level:

$N_{\text{irr}(t)}$6

This design choice is important. Rather than cropping or masking the video, GLARIFY adds the projected gaze embedding to the corresponding image patch embedding. The full visual context is therefore preserved, while the representation is biased toward regions that were gazed at. The fused tokens $N_{\text{irr}(t)}$7 are then passed to the downstream multimodal Thinker. The paper states that only the small gaze-projecting layer is updated in Stage 1, and that in Stage 2 only the Thinker module and projection layer are trained, with the vision backbone frozen. This is intended to minimize catastrophic forgetting and maintain the original language-vision capability of the pretrained model (Wang et al., 26 Sep 2025).

5. Training protocol and evaluation methodology

Training follows a two-stage regimen on 8 A800 GPUs with mixed precision. In Stage 1, only the heatmap projection is trained on QA pairs, without reasoning supervision. In Stage 2, the Thinker and projection layer are fine-tuned on the full GLARIFY-Ambi corpus with CoT reasoning, while the vision backbone remains frozen (Wang et al., 26 Sep 2025).

The evaluation uses a hold-out test set and includes both automatic and human-centered criteria. The objective metrics are GPT-Accuracy, defined as LLM-based grading of whether the answer matches the annotated reference, and Video-Factuality, defined as human-annotated correctness with respect to the video. The subjective metrics are Helpfulness, a human Likert rating from 1 to 10 for response utility, and Trust, a human rating of reliability and explainability. The baselines include InternVL2.5-4B, vanilla Qwen2.5-Omni-3B, Qwen2.5-Omni-3B with finetuning but no gaze, Qwen2.5-Omni-3B with gaze but without explicit noise mitigation or reasoning, GLARIFY Stage 1, GLARIFY CoT Only, and the full GLARIFY system (Wang et al., 26 Sep 2025).

6. Empirical results, ablations, and methodological implications

The principal quantitative result is the improvement in GPT-Accuracy obtained by the full system relative to gaze-naive and gaze-simple baselines. The paper reports the following main comparison (Wang et al., 26 Sep 2025):

Method	GPT-Accuracy
InternVL2.5-4B	23.15%
Qwen2.5-Omni-3B	28.3%
Qwen2.5-Omni-3B + Finetune	26.7%
Qwen2.5-Omni-3B + Gaze	29.85%
GLARIFY	38.05%

The ablation results further isolate the contribution of the two stages. “GLARIFY Stage 1” reaches 28.5%, “GLARIFY CoT Only” reaches 31.7%, and the full system reaches 38.05%. The paper interprets this as evidence that robust spatial alignment is necessary, but that structured CoT reasoning for interpreting noisy gaze is essential for peak disambiguation performance. Put differently, simply inserting gaze into a VLM is not sufficient; the model must also learn to reason about when the gaze stream is misleading (Wang et al., 26 Sep 2025).

On human-centered evaluation, GLARIFY receives the best reported scores in Video-Factuality, Helpfulness, and Trust, with 8.51/10, 7.64/10, and 7.74/10 respectively. The qualitative comparison described for Figure 1 states that when baseline models guess incorrectly because of ambiguous pronouns or misleading gaze, GLARIFY follows the chain of user attention to resolve references and events more accurately. The paper’s broader claim is therefore narrower than generic “gaze improves VLMs”: its evidence supports the position that spatiotemporal gaze becomes useful when it is modeled as noisy, temporally structured, and subject to explicit reasoning, rather than treated as a direct localization signal (Wang et al., 26 Sep 2025).