---
title: 'GLARIFY-Ambi: Dual Gaze & Ambiguity Dataset'
url: https://www.emergentmind.com/topics/glarify-ambi-dataset
type: topic
---

# GLARIFY-Ambi: Dual Gaze & Ambiguity Dataset

Searching arXiv for the cited papers to ground the article in the current record.
GLARIFY-Ambi designates a dataset lineage centered on ambiguity and gaze, but the term is used in two distinct senses in the available record. In "Do you see what I see? An Ambiguous Optical Illusion Dataset exposing limitations of Explainable AI" [2505.21589], the underlying resource is introduced as **Ambivision**, also referred to as **Ambi** and, in a broader explainability context, **GLARIFY-Ambi**. There, it is a compact optical-illusion dataset of intermingled animal pairs with concept-level annotations for eye position and gaze direction. In "Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm" [2509.21980], **GLARIFY-Ambi** instead denotes a synthetic, spatiotemporal, gaze-facilitated video question answering dataset built from Video Localized Narratives. Across both usages, the common theme is that ambiguity cannot be resolved reliably by pixel saliency alone; instead, the decisive signal is treated as a concept-level or attention-level cue, operationalized through gaze.

## 1. Nomenclature and scope

The name **GLARIFY-Ambi** is not monosemous in the cited literature. In the optical-illusion setting, the official dataset names are **Ambivision** and **Ambi**, while the paper states that the dataset is sometimes referenced in a broader XAI context as **GLARIFY-Ambi** [2505.21589]. The Kaggle distribution uses the Ambivision name, and the code is hosted at the Ambivision GitHub repository. In the later vision-language setting, GLARIFY-Ambi is a separate dataset explicitly introduced under that name as part of the GLARIFY framework for gaze-facilitated visual assistant interaction [2509.21980].

This dual usage matters because the two resources differ in modality, scale, and intended task. One is an image classification benchmark built from optical illusions of overlapping animals; the other is a video QA corpus with noisy spatiotemporal gaze traces, ambiguous questions, and chain-of-thought rationales. The shared naming reflects a common research program around ambiguity, gaze, and alignment, but not a single homogeneous dataset.

| Designation in use | Paper | Core modality |
|---|---|---|
| Ambivision / Ambi / sometimes GLARIFY-Ambi | [2505.21589] | Optical-illusion images |
| GLARIFY-Ambi | [2509.21980] | Video keyframes + noisy gaze/trace + QA + CoT |

A plausible implication is that "GLARIFY-Ambi" functions as an umbrella label in some discussions, while the underlying benchmark identity depends on the paper and task context.

## 2. Optical-illusion GLARIFY-Ambi: Ambivision as concept-annotated ambiguity benchmark

In the optical-illusion formulation, the dataset consists of **over 200 images**, each embedding **one animal hidden within the body of another**, with **41 images RGB** and the majority black-and-white due to the difficulty of generating convincing color illusions [2505.21589]. The images are built from **intermingled animal pairs**, such as **penguin within a horse, lion/eagle, rabbit/bear, eagle/cheetah, badger/pigeon, bear/fox**, and the paper highlights a rich **bird** subset for visual analyses. Each illusion is designed so that shared textures, edges, and parts plausibly belong to either animal.

The dataset is positioned at the intersection of explainability and ambiguity research. Rather than centering saliency or pixel-level attributions, it foregrounds abstract visual concepts, especially **gaze direction** and **eye cues**, as the features that resolve semantic conflict [2505.21589]. The motivation is explicit: existing XAI methods such as **Grad-CAM**, **Integrated Gradients**, and prototype-based techniques can highlight the same regions for both classes when an image supports multiple interpretations. In rabbit-duck-style ambiguity, the same region can explain both rival classes.

To support concept-aware analysis, the dataset provides four versions: **original images with labels and annotations**; **direction arrows drawn directly into the images**; **a baseline with random markings**; and **a baseline with only the eye encircled** [2505.21589]. This structure enables controlled experiments on whether concept-level cues improve classification and explanation beyond a control condition that merely adds visible markings.

The intended use-cases include uncertainty quantification, bias and concept sensitivity in perception, model alignment, and benchmarking classification under ambiguity. The safety-critical motivation is framed around calibrated decisions in domains such as autonomous driving and medical imaging, where the ability to reason under ambiguous evidence is consequential [2505.21589].

## 3. Construction and annotations in the optical-illusion dataset

The optical illusions are generated via **iterative prompt engineering with ChatGPT-4 and ChatGPT-4o**, guided by **Gestalt principles** such as **proximity** to induce ambiguous grouping [2505.21589]. Each successful image took **on average about 150 prompts**, totaling **roughly 30,000 prompts**. During generation, the authors actively vary **artistic style**, **pose and activity**, and the **eye coordinates and gaze direction for both animals** to reduce systematic class-specific cues.

The central annotation schema operationalizes gaze parametrically. Each image includes class labels for both animals, the right-eye position
$$
e = [e_x, e_y],
$$
the normalized head or gaze direction vector
$$
d = [d_x, d_y],
$$
and bounding boxes for both animals as pairs
$$
(x_1, y_1), (x_2, y_2).
$$
The gaze direction is constructed as
$$
g = e + \alpha \cdot d,
$$
where $\alpha \in \mathbb{R}$ scales the arrow length to be visible given the image size [2505.21589]. Gaze vectors are normalized to a unified notation, and **"looking straight ahead"** is annotated as **$(0.0, 0.0)$**. When two eyes are visible, the **right eye** is annotated; if only one eye is visible, that eye is annotated.

The authors emphasize **gaze uniqueness**: although looking direction is sometimes shared, the combination of the right-eye coordinate and gaze vector is unique per animal, providing a concept-level key to disambiguation [2505.21589]. The Appendix scatter plots of bird eye coordinates and all classes are reported to show diverse distributions centered around plausible anatomical regions rather than corners or edges. This is presented as a bias-mitigation measure against trivial confounds introduced by generation.

The dataset is publicly accessible via Kaggle under the Ambivision name, and source code is available in the Ambivision GitHub repository. The paper states open-source distribution and includes evaluation scripts, but does not specify a formal license text [2505.21589].

## 4. Experimental findings on ambiguous image classification and XAI

The main task is **image classification under ambiguity**, with one evaluation regime counting **either animal as correct** to reflect the illusion’s nature; the authors also report qualitatively similar trends when only one animal is allowed as correct [2505.21589]. Pretrained ImageNet classifiers—**ResNet18, ResNet34, ResNet50, VGG13, and VGG16**—were fine-tuned with **ADAM**, using learning rates **$1\times10^{-4}$, $1\times10^{-5}$, and $5\times10^{-6}$**, for up to **1000 epochs**, on an **NVIDIA A100-SXM4-80GB GPU**. Images were resized to **$3\times224\times224$** and normalized by mean.

Two concept-injection strategies were examined. One concatenated gaze coordinates late in the network and learned via an MLP on top of the task loss:
$$
\argmin_{g \in G} [ L(f, g, \pi_{x'}, e, d) + \Omega(g) ]
$$
with
$$
L(f, g, \pi_{x'}, e, d) = L\big( L_{task}(f,g, \pi_{x'}) \ \vert \ (e,d) \big).
$$
The paper states that this approach **did not yield a detectable increase in accuracy** [2505.21589]. The alternative encoded the gaze vector directly into the image by drawing the arrow and then fine-tuned the vision model; this **consistently improved performance across architectures and learning rates**.

The primary reported metric is **accuracy vs. epochs**. Across settings, **gaze arrows** or **eye annotations** improved accuracy by large margins relative to both **"no direction"** and **"random markings"** baselines [2505.21589]. At **LR = $1\times10^{-4}$**, **ResNet34 with "Only Eyes" peaked around 62.71% accuracy at 180 epochs**, and the **"Direction"** traces were consistently above **"No Direction"**, often by **15–25+ percentage points** through mid-to-late training. **ResNet50** and **VGG16** showed the same pattern, with **"Only Eyes"** and **"Direction"** outperforming **"No Direction"** and **"Random"**, and **"Only Eyes"** typically the top curve. Similar gains persisted at the lower learning rates across all tested architectures. The authors summarize that eye and gaze cues **"raise accuracy rates by over 20%"** in ambiguous classification [2505.21589].

The explainability analysis is equally central. On unannotated intermingled animal pairs, **Grad-CAM** and **Integrated Gradients** highlight overlapping areas that do not belong to one animal specifically and can attribute the wrong animal’s features. **PipNet** prototypes extracted for a target class often include features from the other animal because textures are intentionally similar. **ACE** fails to recover abstract concepts such as gaze direction and eyes, instead detecting low-level patterns that do not generalize as disambiguating concepts across the dataset [2505.21589]. The paper’s qualitative conclusion is that, in these settings, pixel-importance explanations are insufficient when decisive evidence is conceptual rather than regional.

## 5. GLARIFY-Ambi as a spatiotemporal video QA dataset

In the later work, GLARIFY-Ambi is a **synthetic, spatiotemporal, gaze-facilitated video QA dataset** constructed to train and evaluate VLMs that must resolve ambiguous user queries using noisy human attention [2509.21980]. It is built on top of **Video Localized Narratives (VideoLN, Oops subset)**, which provides **multi-frame video keyframes**, **actor/background narrations**, and **mouse-trace alignments per narration segment**.

Each sample contains visual frames
$$
V'_{i,j},
$$
a per-frame gaze or trace signal
$$
T'_{i,j},
$$
two questions per target—**$Q_{i,j}^D$** (direct) and **$Q_{i,j}^I$** (indirect, ambiguous via pronouns or ellipsis)—a reference answer **$A_{i,j}$**, and a chain-of-thought rationale **$CoT_{i,j}$** [2509.21980]. Sequence length is variable, inherited from the VideoLN keyframe sequence associated with each narration segment. Frames may have missing or perturbed trace data by design.

The motivation is grounded in an analysis of a real user study with smart glasses and **21 participants**. Spoken queries commonly relied on pronouns and ellipses: pronouns appeared in **68%**, **48%**, and **43%** of queries across three daily scenarios, while omission of constraints occurred in **20%**, **40%**, and **39%** of queries [2509.21980]. The paper also quantifies irrelevant fixations over normalized query time $t$ as
$$
R(t) = \frac{N_{irr}(t)}{N_{total}(t)},
$$
where $N_{irr}(t)$ is the count of fixations not aligned with the query’s referent and $N_{total}(t)$ is the total fixations in the temporal bin. Empirically, $R(t)$ remains **above 20% throughout** and rises toward the end to **approximately 50%**, indicating substantial noise.

The synthesis pipeline uses **GPT-4o** to generate QA pairs and CoT, and to transform VideoLN entries into GLARIFY-Ambi samples [2509.21980]. For each actor narration, GPT-4o generates **three QA pairs** containing a direct question, an indirect question, an answer, keyframes, and traces. Trace data are then modified in two ways. In the **spatial correlation reasoning case**, one keyframe’s true trace is replaced with an irrelevant trace from another narration in the same video. In the **temporal correlation reasoning case**, the subject’s trace is propagated into frames lacking trace to simulate attention continuity. Finally, a spatial or temporal CoT rationale is generated for each QA pair.

The resulting dataset is substantially larger than the optical-illusion benchmark. In the final train split, it contains **72,260 QA pairs**, comprising **51,096 spatial** and **21,164 temporal** pairs with CoT. The final test split contains **14,860 QA pairs**, comprising **10,420 spatial** and **4,440 temporal** pairs with CoT [2509.21980]. During pipeline design, **200 samples** were held out, and **2,000 QA pairs** were randomly sampled from the test split for evaluation.

## 6. Gaze integration, evaluation, and broader significance

GLARIFY encodes gaze as per-frame heatmaps and injects them into a pretrained video VLM, **Qwen2.5-Omni-3B**, through a lightweight adapter intended to preserve base knowledge [2509.21980]. With notation
$$
I \in \mathbb{R}^{T\times C\times H\times W}, \quad H \in \mathbb{R}^{T\times 1\times H\times W},
$$
the image and heatmap tensors are patchified:
$$
X = \mathrm{patchify}(I), \quad G = \mathrm{patchify}(H).
$$
A linear projection maps gaze patches to the model dimension:
$$
Z = f_{\mathrm{linear}}(G),
$$
while a ViT encodes visual patches:
$$
V = ViT(X).
$$
Fusion is performed elementwise,
$$
\widetilde{V_t} = V_t + Z_t,
$$
so that tokens at fixated patches are softly elevated without discarding context [2509.21980]. The adapter adds only **about 0.0341% new parameters**.

Training proceeds in two stages. **Stage 1** trains only $f_{\mathrm{linear}}$ on QA pairs without CoT for **one epoch**, with all base Qwen2.5-Omni modules frozen. **Stage 2** jointly fine-tunes the Thinker and $f_{\mathrm{linear}}$ on the full GLARIFY-Ambi with CoT for **two epochs**. The Appendix reports **8× A800 GPUs**, **bfloat16**, and **Flash Attention**; Stage 1 uses **batch size 2/GPU**, **grad accumulation 2**, and **LR = $1e^{-4}$**, while Stage 2 resumes with **batch size 4/GPU** and the same accumulation [2509.21980].

The evaluation task is to answer the indirect question **$Q^I$** for a given $(V', T')$ sample. Metrics include **GPT-Accuracy**, **Video-Factuality**, and human-rated **Helpfulness** and **Trust** [2509.21980]. On the hold-out test set, **InternVL2.5-4B** reaches **23.15% GPT-Accuracy**, **Qwen2.5-Omni-3B** reaches **28.3%**, **Qwen2.5-Omni-3B + standard finetune** reaches **26.7%**, and **Qwen2.5-Omni-3B + naive gaze injection (no CoT)** reaches **29.85%**. **GLARIFY** reaches **38.05% GPT-Accuracy**, with **Video-Factuality 8.51**, **Helpfulness 7.64**, and **Trust 7.74** [2509.21980]. Ablations show **Stage 1 only** at **28.5% GPT-Accuracy**, **CoT only** at **31.7%**, and the full two-stage regime at **38.05%**, supporting the paper’s interpretation that a novel modality benefits from simple alignment before multi-step CoT reasoning.

Across both the optical-illusion and video-QA lines of work, the recurring claim is that ambiguity is not resolved by localizing pixels alone. In Ambivision, minimal concept-level supervision—an arrow for $g$ or a rough circle around the eye—substantially improves ambiguous image classification and changes explanation behavior [2505.21589]. In GLARIFY-Ambi for video QA, noisy human attention becomes usable when encoded spatiotemporally and paired with structured reasoning over pronouns, omitted context, and temporal event structure [2509.21980]. This suggests a broader research direction in which explainability and alignment are studied through abstract perceptual cues, rather than through saliency maps alone.

Both formulations also state limitations. The optical-illusion dataset is modest in scale, machine-generated, and focused primarily on two-animal illusions where the combination of eye coordinate and gaze vector distinguishes classes [2505.21589]. The video dataset uses mouse traces as a proxy for gaze, does not address real-time synchronization, currently focuses on English, and does not employ explicit attention-alignment losses [2509.21980]. Future work is explicitly directed toward larger datasets, formal human perception studies, multilinguality, and concept-based XAI methods that move beyond pixel segmentation to abstract cues such as gaze, pose, and intent.

Source: https://www.emergentmind.com/topics/glarify-ambi-dataset