Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videos

Abstract: The ability to anticipate human-object interactions is highly desirable in an intelligent assistive system in order to guide users during daily life activities and understand their short and long-term goals. Creating systems with such capabilities requires to approach several complex challenges. This work addresses the problem of human-object interaction anticipation in Egocentric Vision using Vision LLMs (VLLMs). We tackle key limitations in existing approaches by improving visual grounding capabilities through Set-of-Mark prompting and understanding user intent via the trajectory formed by the user's most recent gaze fixations. To effectively capture the temporal dynamics immediately preceding the interaction, we further introduce a novel inverse exponential sampling strategy for input video frames. Experiments conducted on the egocentric dataset HD-EPIC demonstrate that our method surpasses state-of-the-art approaches for the considered task, showing its model-agnostic nature.

• The paper introduces a novel model-agnostic framework that integrates gaze trajectory modeling with spatial segmentation (SoM) for improved HOI anticipation.
• It employs an inverse exponential sampling strategy for efficient temporal frame selection, resulting in a significant accuracy boost on benchmark datasets.
• The combined approach enhances the predictive capability of VLLMs, offering robust implications for real-time assistive systems through proactive user intent recognition.

Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videos

Problem Formulation and Task Setting

Human-object interaction anticipation is a critical capability for intelligent assistive systems operating in egocentric vision scenarios, enabling proactive responses to user intent and goal recognition. This paper targets the task of anticipating the next human-object interaction in egocentric video streams using Vision LLMs (VLLMs). The task is framed as a Visual Question Answering (VQA) challenge, adopting protocols from the HD-EPIC Gaze Interaction Anticipation benchmark. The model must select the target object that will undergo interaction, given a set of candidates and multimodal context cues.

Figure 1: Conceptual scheme of the VQA Interaction Anticipation task, illustrating multimodal inputs and candidate selection.

Architecture Overview and Methodological Contributions

The proposed architecture comprises four principal modules: Set-of-Mark (SoM) for spatial scene encoding, Gaze trajectory modeling for intent prediction, an inverse exponential Sampling module for temporally relevant frame selection, and a final VLLM module for target prediction. This model-agnostic pipeline is designed for seamless integration with any state-of-the-art VLLM.

Figure 2: The architecture includes SoM, Gaze, Sampling, and VLLM modules, facilitating spatial grounding, intention modeling, efficient temporal selection, and multimodal reasoning.

Set-of-Mark Module: Spatial Visual Prompting

SoM prompting partitions the final input frame into semantically segmented regions using Semantic-SAM masks, enhancing visual grounding and object localization. This prompts the VLLM to reason spatially about the scene and candidate interactions. Alphanumeric tags are omitted as object identity is managed via the textual prompt.

Gaze Module: Intention Modeling

The Gaze module projects the user's last $W = 15$ gaze fixations onto each frame, rendered as color-encoded circles and connected paths. This temporally ordered gaze trajectory provides a powerful prior for interaction anticipation, guiding the VLLM toward objects of recent visual attention.

Inverse Exponential Sampling: Temporal Selection

Given the quadratic complexity of transformer-based models with respect to input length, dense video input is computationally impractical. An inverse exponential sampling strategy probabilistically concentrates frame selection toward the instants preceding the interaction, controlled by a hyperparameter $\lambda$, ensuring maximal relevance and eliminating redundant temporal context.

Figure 3: Inverse exponential sampling ($n = 10$) increases temporal bias towards the interaction, contrasting uniform ($\lambda = 0$) and concentrated ($\lambda > 0$) distributions.

VLLM Module: Multimodal Reasoning

The processed visual cues and textual question are input to the VLLM, which infers the next object of interaction. The approach operates directly on input frames, reinforcing model-agnostic applicability and compatibility with a variety of VLLM architectures.

Experimental Evaluation and Ablations

Dataset and Protocol

HD-EPIC Gaze Interaction Anticipation subset is used, featuring multimodal video, gaze, and dense object interaction annotations. Each VQA sample provides candidate objects and visual context ending 0.3 seconds after gaze priming.

VLLM Selection and Model-Agnostic Results

Experiments employ LLaVA-OneVision (7B) and Gemini 2.0 Flash, chosen for state-of-the-art video understanding and compatibility with established benchmarks. The approach significantly outperforms previous challenge winners and baselines.

Approach	Accuracy (%)
K-Net	15.7
T-CoT	15.8
LLaVA-OneVision 7B	20.4
Gemini 1.5 Pro (baseline)	21.0
Qwen2.5VL-Ricoh	22.0
Ours (LLaVA-OneVision 7B)	27.2
Ours (Gemini 2.0 Flash)	27.5

The method yields accuracy improvements exceeding 5% over previous SOTA, with consistent gains irrespective of the underlying VLLM, validating its model-agnostic design.

Qualitative Analysis

Qualitative results indicate that the inclusion of Gaze trajectories principally enhances attention alignment, while SoM alone often fails to disambiguate closely situated candidates. Joint utilization of both modules further improves prediction accuracy in complex scenarios.

(Figure 1, Figure 2, and Figure 3 referenced above; additional qualitative figures omitted for brevity.)

Ablation Studies and Module Impact

Comprehensive ablation analysis demonstrates:

• SoM and Gaze synergy: Combining spatial grounding and intention cues provides the largest accuracy boost.
• Sampling strategy optimization: Inverse exponential sampling with $\lambda = \frac{1}{10}$ and $n = 15$ yields maximal performance; oversampling reduces accuracy consistent with benchmarks on VLLMs.
• Module independence: Gains persist across different VLLMs, underscoring architectural modularity.

Practical and Theoretical Implications

The approach enables precise short-term interaction anticipation, applicable to real-time assistive systems for safety, workflow guidance, and intelligent interfaces. Multimodal input design and sampling optimization inform future research in integrating spatial and intention signals into multimodal LLMs. Long-term extension and real-time deployment could further enhance proactive assistive capabilities.

Conclusion

This work introduces a model-agnostic framework leveraging Set-of-Mark spatial prompts, gaze trajectory modeling, and optimized temporal sampling for human-object interaction anticipation in egocentric videos using VLLMs. Robust empirical results on the HD-EPIC benchmark substantiate strong generalization and accuracy gains. Future directions include temporal extension for action anticipation and real-time multimodal reasoning in assistive applications.