GLARIFY-Ambi: Dual Gaze & Ambiguity Dataset

• The paper [2505.21589] demonstrates that integrating concept-level gaze cues in optical illusions improves ambiguous classification accuracy by over 20%.
• GLARIFY-Ambi also introduces a synthetic video QA dataset that uses noisy gaze traces and chain-of-thought rationales to resolve ambiguous queries.
• Gaze signals are encoded as abstract, concept-level features that outperform pixel-level saliency in resolving semantic ambiguity across modalities.

Searching arXiv for the cited papers to ground the article in the current record. GLARIFY-Ambi designates a dataset lineage centered on ambiguity and gaze, but the term is used in two distinct senses in the available record. In "Do you see what I see? An Ambiguous Optical Illusion Dataset exposing limitations of Explainable AI" (Newen et al., 27 May 2025), the underlying resource is introduced as Ambivision, also referred to as Ambi and, in a broader explainability context, GLARIFY-Ambi. There, it is a compact optical-illusion dataset of intermingled animal pairs with concept-level annotations for eye position and gaze direction. In "Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm" (Wang et al., 26 Sep 2025), GLARIFY-Ambi instead denotes a synthetic, spatiotemporal, gaze-facilitated video question answering dataset built from Video Localized Narratives. Across both usages, the common theme is that ambiguity cannot be resolved reliably by pixel saliency alone; instead, the decisive signal is treated as a concept-level or attention-level cue, operationalized through gaze.

1. Nomenclature and scope

The name GLARIFY-Ambi is not monosemous in the cited literature. In the optical-illusion setting, the official dataset names are Ambivision and Ambi, while the paper states that the dataset is sometimes referenced in a broader XAI context as GLARIFY-Ambi (Newen et al., 27 May 2025). The Kaggle distribution uses the Ambivision name, and the code is hosted at the Ambivision GitHub repository. In the later vision-language setting, GLARIFY-Ambi is a separate dataset explicitly introduced under that name as part of the GLARIFY framework for gaze-facilitated visual assistant interaction (Wang et al., 26 Sep 2025).

This dual usage matters because the two resources differ in modality, scale, and intended task. One is an image classification benchmark built from optical illusions of overlapping animals; the other is a video QA corpus with noisy spatiotemporal gaze traces, ambiguous questions, and chain-of-thought rationales. The shared naming reflects a common research program around ambiguity, gaze, and alignment, but not a single homogeneous dataset.

Designation in use	Paper	Core modality
Ambivision / Ambi / sometimes GLARIFY-Ambi	(Newen et al., 27 May 2025)	Optical-illusion images
GLARIFY-Ambi	(Wang et al., 26 Sep 2025)	Video keyframes + noisy gaze/trace + QA + CoT

A plausible implication is that "GLARIFY-Ambi" functions as an umbrella label in some discussions, while the underlying benchmark identity depends on the paper and task context.

2. Optical-illusion GLARIFY-Ambi: Ambivision as concept-annotated ambiguity benchmark

In the optical-illusion formulation, the dataset consists of over 200 images, each embedding one animal hidden within the body of another, with 41 images RGB and the majority black-and-white due to the difficulty of generating convincing color illusions (Newen et al., 27 May 2025). The images are built from intermingled animal pairs, such as penguin within a horse, lion/eagle, rabbit/bear, eagle/cheetah, badger/pigeon, bear/fox, and the paper highlights a rich bird subset for visual analyses. Each illusion is designed so that shared textures, edges, and parts plausibly belong to either animal.

The dataset is positioned at the intersection of explainability and ambiguity research. Rather than centering saliency or pixel-level attributions, it foregrounds abstract visual concepts, especially gaze direction and eye cues, as the features that resolve semantic conflict (Newen et al., 27 May 2025). The motivation is explicit: existing XAI methods such as Grad-CAM, Integrated Gradients, and prototype-based techniques can highlight the same regions for both classes when an image supports multiple interpretations. In rabbit-duck-style ambiguity, the same region can explain both rival classes.

To support concept-aware analysis, the dataset provides four versions: original images with labels and annotations; direction arrows drawn directly into the images; a baseline with random markings; and a baseline with only the eye encircled (Newen et al., 27 May 2025). This structure enables controlled experiments on whether concept-level cues improve classification and explanation beyond a control condition that merely adds visible markings.

The intended use-cases include uncertainty quantification, bias and concept sensitivity in perception, model alignment, and benchmarking classification under ambiguity. The safety-critical motivation is framed around calibrated decisions in domains such as autonomous driving and medical imaging, where the ability to reason under ambiguous evidence is consequential (Newen et al., 27 May 2025).

3. Construction and annotations in the optical-illusion dataset

The optical illusions are generated via iterative prompt engineering with ChatGPT-4 and ChatGPT-4o, guided by Gestalt principles such as proximity to induce ambiguous grouping (Newen et al., 27 May 2025). Each successful image took on average about 150 prompts, totaling roughly 30,000 prompts. During generation, the authors actively vary artistic style, pose and activity, and the eye coordinates and gaze direction for both animals to reduce systematic class-specific cues.

The central annotation schema operationalizes gaze parametrically. Each image includes class labels for both animals, the right-eye position

$e = [e_x, e_y],$

the normalized head or gaze direction vector

$d = [d_x, d_y],$

and bounding boxes for both animals as pairs

$(x_1, y_1), (x_2, y_2).$

The gaze direction is constructed as

$g = e + \alpha \cdot d,$

where $\alpha \in \mathbb{R}$ scales the arrow length to be visible given the image size (Newen et al., 27 May 2025). Gaze vectors are normalized to a unified notation, and "looking straight ahead" is annotated as $(0.0, 0.0)$. When two eyes are visible, the right eye is annotated; if only one eye is visible, that eye is annotated.

The authors emphasize gaze uniqueness: although looking direction is sometimes shared, the combination of the right-eye coordinate and gaze vector is unique per animal, providing a concept-level key to disambiguation (Newen et al., 27 May 2025). The Appendix scatter plots of bird eye coordinates and all classes are reported to show diverse distributions centered around plausible anatomical regions rather than corners or edges. This is presented as a bias-mitigation measure against trivial confounds introduced by generation.

The dataset is publicly accessible via Kaggle under the Ambivision name, and source code is available in the Ambivision GitHub repository. The paper states open-source distribution and includes evaluation scripts, but does not specify a formal license text (Newen et al., 27 May 2025).

4. Experimental findings on ambiguous image classification and XAI

The main task is image classification under ambiguity, with one evaluation regime counting either animal as correct to reflect the illusion’s nature; the authors also report qualitatively similar trends when only one animal is allowed as correct (Newen et al., 27 May 2025). Pretrained ImageNet classifiers—ResNet18, ResNet34, ResNet50, VGG13, and VGG16—were fine-tuned with ADAM, using learning rates $1\times10^{-4}$, $1\times10^{-5}$, and $5\times10^{-6}$, for up to 1000 epochs, on an NVIDIA A100-SXM4-80GB GPU. Images were resized to $3\times224\times224$ and normalized by mean.

Two concept-injection strategies were examined. One concatenated gaze coordinates late in the network and learned via an MLP on top of the task loss:

$d = [d_x, d_y],$0

with

$d = [d_x, d_y],$1

The paper states that this approach did not yield a detectable increase in accuracy (Newen et al., 27 May 2025). The alternative encoded the gaze vector directly into the image by drawing the arrow and then fine-tuned the vision model; this consistently improved performance across architectures and learning rates.

The primary reported metric is accuracy vs. epochs. Across settings, gaze arrows or eye annotations improved accuracy by large margins relative to both "no direction" and "random markings" baselines (Newen et al., 27 May 2025). At LR = $d = [d_x, d_y],$2, ResNet34 with "Only Eyes" peaked around 62.71% accuracy at 180 epochs, and the "Direction" traces were consistently above "No Direction", often by 15–25+ percentage points through mid-to-late training. ResNet50 and VGG16 showed the same pattern, with "Only Eyes" and "Direction" outperforming "No Direction" and "Random", and "Only Eyes" typically the top curve. Similar gains persisted at the lower learning rates across all tested architectures. The authors summarize that eye and gaze cues "raise accuracy rates by over 20%" in ambiguous classification (Newen et al., 27 May 2025).

The explainability analysis is equally central. On unannotated intermingled animal pairs, Grad-CAM and Integrated Gradients highlight overlapping areas that do not belong to one animal specifically and can attribute the wrong animal’s features. PipNet prototypes extracted for a target class often include features from the other animal because textures are intentionally similar. ACE fails to recover abstract concepts such as gaze direction and eyes, instead detecting low-level patterns that do not generalize as disambiguating concepts across the dataset (Newen et al., 27 May 2025). The paper’s qualitative conclusion is that, in these settings, pixel-importance explanations are insufficient when decisive evidence is conceptual rather than regional.

5. GLARIFY-Ambi as a spatiotemporal video QA dataset

In the later work, GLARIFY-Ambi is a synthetic, spatiotemporal, gaze-facilitated video QA dataset constructed to train and evaluate VLMs that must resolve ambiguous user queries using noisy human attention (Wang et al., 26 Sep 2025). It is built on top of Video Localized Narratives (VideoLN, Oops subset), which provides multi-frame video keyframes, actor/background narrations, and mouse-trace alignments per narration segment.

Each sample contains visual frames

$d = [d_x, d_y],$3

a per-frame gaze or trace signal

$d = [d_x, d_y],$4

two questions per target—$d = [d_x, d_y],$5 (direct) and $d = [d_x, d_y],$6 (indirect, ambiguous via pronouns or ellipsis)—a reference answer $d = [d_x, d_y],$7, and a chain-of-thought rationale $d = [d_x, d_y],$8 (Wang et al., 26 Sep 2025). Sequence length is variable, inherited from the VideoLN keyframe sequence associated with each narration segment. Frames may have missing or perturbed trace data by design.

The motivation is grounded in an analysis of a real user study with smart glasses and 21 participants. Spoken queries commonly relied on pronouns and ellipses: pronouns appeared in 68%, 48%, and 43% of queries across three daily scenarios, while omission of constraints occurred in 20%, 40%, and 39% of queries (Wang et al., 26 Sep 2025). The paper also quantifies irrelevant fixations over normalized query time $d = [d_x, d_y],$9 as

$(x_1, y_1), (x_2, y_2).$0

where $(x_1, y_1), (x_2, y_2).$1 is the count of fixations not aligned with the query’s referent and $(x_1, y_1), (x_2, y_2).$2 is the total fixations in the temporal bin. Empirically, $(x_1, y_1), (x_2, y_2).$3 remains above 20% throughout and rises toward the end to approximately 50%, indicating substantial noise.

The synthesis pipeline uses GPT-4o to generate QA pairs and CoT, and to transform VideoLN entries into GLARIFY-Ambi samples (Wang et al., 26 Sep 2025). For each actor narration, GPT-4o generates three QA pairs containing a direct question, an indirect question, an answer, keyframes, and traces. Trace data are then modified in two ways. In the spatial correlation reasoning case, one keyframe’s true trace is replaced with an irrelevant trace from another narration in the same video. In the temporal correlation reasoning case, the subject’s trace is propagated into frames lacking trace to simulate attention continuity. Finally, a spatial or temporal CoT rationale is generated for each QA pair.

The resulting dataset is substantially larger than the optical-illusion benchmark. In the final train split, it contains 72,260 QA pairs, comprising 51,096 spatial and 21,164 temporal pairs with CoT. The final test split contains 14,860 QA pairs, comprising 10,420 spatial and 4,440 temporal pairs with CoT (Wang et al., 26 Sep 2025). During pipeline design, 200 samples were held out, and 2,000 QA pairs were randomly sampled from the test split for evaluation.

6. Gaze integration, evaluation, and broader significance

GLARIFY encodes gaze as per-frame heatmaps and injects them into a pretrained video VLM, Qwen2.5-Omni-3B, through a lightweight adapter intended to preserve base knowledge (Wang et al., 26 Sep 2025). With notation

$(x_1, y_1), (x_2, y_2).$4

the image and heatmap tensors are patchified:

$(x_1, y_1), (x_2, y_2).$5

A linear projection maps gaze patches to the model dimension:

$(x_1, y_1), (x_2, y_2).$6

while a ViT encodes visual patches:

$(x_1, y_1), (x_2, y_2).$7

Fusion is performed elementwise,

$(x_1, y_1), (x_2, y_2).$8

so that tokens at fixated patches are softly elevated without discarding context (Wang et al., 26 Sep 2025). The adapter adds only about 0.0341% new parameters.

Training proceeds in two stages. Stage 1 trains only $(x_1, y_1), (x_2, y_2).$9 on QA pairs without CoT for one epoch, with all base Qwen2.5-Omni modules frozen. Stage 2 jointly fine-tunes the Thinker and $g = e + \alpha \cdot d,$0 on the full GLARIFY-Ambi with CoT for two epochs. The Appendix reports 8× A800 GPUs, bfloat16, and Flash Attention; Stage 1 uses batch size 2/GPU, grad accumulation 2, and LR = $g = e + \alpha \cdot d,$1, while Stage 2 resumes with batch size 4/GPU and the same accumulation (Wang et al., 26 Sep 2025).

The evaluation task is to answer the indirect question $g = e + \alpha \cdot d,$2 for a given $g = e + \alpha \cdot d,$3 sample. Metrics include GPT-Accuracy, Video-Factuality, and human-rated Helpfulness and Trust (Wang et al., 26 Sep 2025). On the hold-out test set, InternVL2.5-4B reaches 23.15% GPT-Accuracy, Qwen2.5-Omni-3B reaches 28.3%, Qwen2.5-Omni-3B + standard finetune reaches 26.7%, and Qwen2.5-Omni-3B + naive gaze injection (no CoT) reaches 29.85%. GLARIFY reaches 38.05% GPT-Accuracy, with Video-Factuality 8.51, Helpfulness 7.64, and Trust 7.74 (Wang et al., 26 Sep 2025). Ablations show Stage 1 only at 28.5% GPT-Accuracy, CoT only at 31.7%, and the full two-stage regime at 38.05%, supporting the paper’s interpretation that a novel modality benefits from simple alignment before multi-step CoT reasoning.

Across both the optical-illusion and video-QA lines of work, the recurring claim is that ambiguity is not resolved by localizing pixels alone. In Ambivision, minimal concept-level supervision—an arrow for $g = e + \alpha \cdot d,$4 or a rough circle around the eye—substantially improves ambiguous image classification and changes explanation behavior (Newen et al., 27 May 2025). In GLARIFY-Ambi for video QA, noisy human attention becomes usable when encoded spatiotemporally and paired with structured reasoning over pronouns, omitted context, and temporal event structure (Wang et al., 26 Sep 2025). This suggests a broader research direction in which explainability and alignment are studied through abstract perceptual cues, rather than through saliency maps alone.

Both formulations also state limitations. The optical-illusion dataset is modest in scale, machine-generated, and focused primarily on two-animal illusions where the combination of eye coordinate and gaze vector distinguishes classes (Newen et al., 27 May 2025). The video dataset uses mouse traces as a proxy for gaze, does not address real-time synchronization, currently focuses on English, and does not employ explicit attention-alignment losses (Wang et al., 26 Sep 2025). Future work is explicitly directed toward larger datasets, formal human perception studies, multilinguality, and concept-based XAI methods that move beyond pixel segmentation to abstract cues such as gaze, pose, and intent.