IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models

Abstract: We introduce IRIS (Intent Resolution via Inference-time Saccades), a novel training-free approach that uses eye-tracking data in real-time to resolve ambiguity in open-ended VQA. Through a comprehensive user study with 500 unique image-question pairs, we demonstrate that fixations closest to the time participants start verbally asking their questions are the most informative for disambiguation in Large VLMs, more than doubling the accuracy of responses on ambiguous questions (from 35.2% to 77.2%) while maintaining performance on unambiguous queries. We evaluate our approach across state-of-the-art VLMs, showing consistent improvements when gaze data is incorporated in ambiguous image-question pairs, regardless of architectural differences. We release a new benchmark dataset to use eye movement data for disambiguated VQA, a novel real-time interactive protocol, and an evaluation suite.

• The paper introduces IRIS, a training-free method that overlays speech-timed eye fixations on images so existing vision-language models can resolve ambiguous referents during inference.
• IRIS more than doubles accuracy on ambiguous questions from 35.2% to 77.2% while preserving performance on unambiguous questions, with improvements observed across nine of ten tested VLMs.
• The results show that fixations near speech onset are most informative, suggesting practical systems should use a narrow temporal gaze window and account for tracking noise, distractors, and limited generalizability.

Overview

IRIS (Intent Resolution via Inference-time Saccades) is a training-free method that conditions large vision-LLMs (VLMs) on a user's real-time eye-tracking fixations to resolve referential ambiguity in open-ended visual question answering (2602.16138). When a user asks an underspecified question such as "What is that?" about an image containing multiple candidate referents, IRIS overlays temporally and spatially filtered fixations onto the image and passes the annotated image, together with the transcribed question, to an off-the-shelf VLM. No model parameters are modified; the gaze signal acts purely as an inference-time prior on referent selection. The paper's central empirical claim is that fixations near speech onset are maximally informative for disambiguation: conditioning on them more than doubles accuracy on ambiguous questions (35.2% → 77.2%, $p < .001$), while leaving unambiguous-question performance statistically unchanged.

Experimental design

The study used a within-subjects protocol with 10 participants (ages 19–26), monocular EyeLink 1000 tracking at 1000 Hz (calibration error < 1 dva), and 50 photographs of everyday scenes—40 designated for ambiguous questions and 10 for unambiguous ones. Participants first passed a forced central fixation check, then freely viewed each image and asked a question aloud of the assigned type. After 1.5 s of silence following speech offset, the system prompted the VLM with the image, the transcribed question (via speech-to-text), and the same image with filtered fixation markers superimposed. Participants then registered the queried object's location via mouse click (the Location of Interest, LOI), yielding 500 unique image–question pairs with synchronized speech, gaze, and stimulus data. Sessions lasted approximately 45 minutes.

Gaze processing proceeded in two stages. Fixations were detected online using velocity/acceleration thresholds (30°/s, 8000°/s²). A temporal filter retained fixations within a window around voice-activity-detected speech onset; a spatial filter then kept only fixations within 2 dva of the coordinate-wise median location of those fixations. When spatial filtering produced zero fixations (10.8% of trials), all temporally filtered fixations were retained instead—a fallback the authors note rescued roughly 1.5–2% otherwise-failed trials.

Ground truth for open-ended responses was constructed by generating candidate answers from three frontier VLMs given the image, question, and LOI, followed by human evaluation by five independent raters who selected or wrote the most accurate answer; the shortest accurate response was standardized as ground truth to control for embedding-similarity length sensitivity. Evaluation used binary accuracy (automated Gemini-2.5-Flash judgment, verified at 88% agreement with manual ratings across all 500 pairs) plus cosine similarity between frozen sentence embeddings of model and ground-truth responses. The similarity analysis was anchored by baselines and bounds: an "all-fixations" baseline, a mouse-click LOI upper bound ("perfect gaze"), inter-rater self-similarity, an image-only lower bound, and a deliberately elicited "wrong answer" floor that controls for object-level differences within images.

Temporal structure of the disambiguation signal

The paper's most informative analysis concerns when, relative to speech onset, fixations carry disambiguating information. Expanding a symmetric window around speech onset from ±250 ms to ±3000 ms increased semantic similarity monotonically from 0.57 to a peak of 0.65 at ±2500 ms, converging with the all-fixations baseline (0.615) at ±4500 ms. The mechanism tracks fixation-to-LOI distance, which decreases from 4.0° to 2.7° as the window expands, and performance correlates strongly with that distance ($r = -0.89$, $p < .001$). A sliding-window analysis (600 ms width, 400 ms step) located peak performance (0.62 similarity) in windows spanning roughly −200 ms to +400 ms around speech onset—the interval where median fixation-to-LOI distance is minimal (3.5°)—consistent with psycholinguistic findings that speakers fixate referents just before naming them. Notably, even the unfiltered all-fixations baseline significantly exceeded the image-only baseline (0.615 vs. 0.531, $p < .001$), indicating that gaze concentration alone carries signal. Temporal filtering yielded a 17.7% accuracy gain over all-fixations.

An implication worth stating directly: because the optimal signal lies in a narrow pre- and peri-onset window, later fixations actively inject noise, so practical systems should gate on speech onset timing rather than accumulate gaze over the whole viewing period.

Ambiguous versus unambiguous questions

Using the identified ~±1 s window, gaze conditioning produced a 115% relative accuracy improvement on ambiguous questions (35.2% → 77.2%, $p < .001$), approaching but remaining below the LOI "perfect gaze" bound (89.0%). Semantic similarity rose correspondingly (0.531 → 0.650, $p < .001$), versus a LOI condition at 0.691 and a ground-truth inter-rater self-similarity ceiling of 0.82. For unambiguous questions, gaze conferred no significant benefit (accuracy 83.0% → 86.0%, $p = .52$; similarity 0.656 → 0.671, $p = .685$), which the authors attribute to a ceiling effect. All conditions exceeded the "wrong answer" lower-bound baseline, confirming the comparisons are not trivially driven by object identity.

Generality across architectures

Across ten VLMs spanning proprietary and open-source families, image+gaze uniformly outperformed image-only on ambiguous trials:

Model	Image Only (%)	Image+Gaze (%)	Δ (pp)
GPT-5 Mini	49.7	83.0	+33.2
Gemini 2.5 Flash	54.2	83.3	+29.0
Gemini 2.5 Pro	59.3	82.3	+23.0
GPT-5	53.7	76.5	+22.8
Claude Sonnet 4	54.2	74.0	+19.8
Ovis 2.5 9B	52.5	69.5	+17.0
Claude 4.1 Opus	55.2	72.0	+16.7
Qwen 2.5 VL 32B	56.2	73.0	+16.7
Qwen 2.5 VL 7B	51.7	63.7	+12.0
Qwen 2.5 VL 3B	52.0	54.5	+2.5

All differences except the smallest model (Qwen 2.5 VL 3B, $p = 0.48$) were significant at $p < .001$. Gains correlate with instruction-following capability rather than parameter count, supporting the interpretation that gaze functions as an architecture-agnostic referential prior. On the similarity metric, every model fell above the diagonal (improving with gaze); GPT-5 reached 0.71 similarity with gaze augmentation.

Ablations over gaze representations using GPT-5-Mini showed that simple cross markers outperformed Gaussian heatmaps (0.820), DBSCAN bounding boxes (0.780), raw coordinates-as-text (0.748), and fixation-centered crops (0.703). Cropping performed worst because it discards contextual information and can exclude the referent itself when fixations are offset from its center.

Qualitatively, success requires fixations concentrated on the target near speech onset, clearly separated from distractors, and aligned with question intent. Failures arise when fixations fall between candidates, land on occluding surfaces (e.g., tinted glass), reflect speech-recognition errors, or miss the referent entirely. An error taxonomy over all 1500 responses found that ~57% of image-only failures on ambiguous trials stem from referent bias—the model silently assuming one instance among several—which echoes reported overconfidence of visual LLMs under inherent ambiguity. On gaze-conditioned trials, 11.6% of failures were attributable to the eye-data pipeline rather than the model, and detection errors (~5.1%) persisted even with perfect LOI input.

Limitations and open questions

The authors frame the work explicitly as a proof-of-concept under controlled laboratory conditions, and several constraints bear on the headline results. The sample is small (10 university-affiliated participants), raising generalizability questions across demographics. Eye tracking used research-grade hardware at high sampling rates with a chinrest, whereas consumer-grade trackers introduce noise whose effect on the disambiguation signal is unmeasured. Ground truth depends partly on frontier-model generations and human raters, and the automated evaluator agreement (88%) leaves residual label noise. The spatial filter fails outright on ~10.8% of trials, and the study covers static images and single-object referent questions only. Open questions left by the paper include whether the ±1 s speech-onset window remains optimal under noisy consumer eye trackers, whether the approach extends to multi-object or sequential referring expressions, and how robustly VLMs exploit gaze overlays without the explicit prompt scaffolding used here. Reproducibility is currently partial: full stimuli, code, and benchmark release are deferred until acceptance, with only practice-trial data and anonymized supplementary material available now.

Conclusion

IRIS demonstrates that naturally occurring fixations around speech onset provide a sufficient, architecture-independent signal for resolving referential ambiguity in open-ended VQA, delivered purely at inference time through image annotation. The magnitude of the gain on ambiguous questions (+42 pp accuracy), its consistency across ten VLMs, and the absence of harm on unambiguous queries collectively establish gaze as a practical disambiguation prior compatible with existing deployed models. The evidence rests, however, on a controlled setup with research-grade tracking and a modest participant pool, leaving the durability of these gains in unconstrained, consumer-hardware settings as the principal unresolved question.