SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding

Abstract: Human object interaction (HOI), gaze pattern, and their anticipation are intricately linked, providing valuable insights into cognitive processes, intentions, and behavior. However, most existing models handle gaze and actions separately, missing both their interdependence and the advantages of a unified solution. This paper presents a novel unified framework, SAGE (Synchronized Action-GazE), which integrates simultaneous recognition and anticipation of both HOI and human gaze into a single unified end-to-end trainable model. Our approach leverages a transformer-based architecture and incorporates gaze data into spatiotemporal attention mechanisms to simultaneously predict current and future human actions and gaze behavior. We explore this bidirectional relationship between gaze and actions under different scenarios, whether requiring a close-up, detailed view (egocentric) or a wider, more contextual view (exocentric), making our framework versatile for various applications. Additionally, due to lack of datasets for comprehensive analysis of both HOI and gaze in exocentric videos, we establish a new benchmark Exo-Cook to facilitate further research in this domain. Experiments on three benchmark datasets: VidHOI, EGTEA Gaze+, and Exo-Cook show that jointly modeling gaze and actions across current and future frames achieves consistently strong results, often surpassing specialized state-of-the-art models tailored to individual tasks. By unifying actions and attention in a comprehensive way, our work lays the groundwork for more intuitive human-machine interaction.

• The paper introduces a unified transformer model that jointly recognizes and anticipates human gaze and actions.
• It employs gaze-conditioned spatial and temporal modules (GCSA and GCTP) to significantly improve HOI detection and anticipation accuracy.
• The introduction of the Exo-Cook dataset provides a robust multi-view benchmark for evaluating integrated action and gaze prediction in varied visual contexts.

SAGE: Unified Recognition and Anticipation of Action-Gaze in Human Behavior Understanding

Introduction

The paper "SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding" (2607.04017) presents a unified transformer-based framework for jointly recognizing and anticipating both human-object interactions (HOI) and gaze in video. Unlike prior work that treats gaze and action understanding as disjoint tasks, SAGE fuses gaze estimation, gaze anticipation, HOI detection, and HOI anticipation within a single model. The design supports both egocentric and exocentric viewpoints, leverages a purpose-built multi-view exocentric cooking dataset (Exo-Cook), and demonstrates strong empirical results across three benchmarks: VidHOI, EGTEA Gaze+, and Exo-Cook.

Motivation and Background

Simultaneously modeling action and gaze is critical for downstream tasks in intuitive human-machine interaction, as gaze offers direct information about visual attention and impending intention, whereas action recognition or anticipation decodes explicit engagement and behavioral transitions. Existing pipelines generally lack explicit integration of these cues—gaze is typically treated either as a post-hoc input to action models or as an auxiliary recognition module, with little modeling of temporal or causal couplings between future gaze and future actions. Notably, nearly all previous work is restricted to first-person (egocentric) data, with the few exocentric data sources (e.g., Ego-Exo4D) not tailored for synchronized and aligned action-gaze prediction tasks.

Figure 1: Illustration of action-gaze correlation over time. Gaze reveals both the current focus of attention and future intention, providing complementary cues for recognizing ongoing actions and anticipating subsequent human-object interactions.

Exo-Cook Dataset: Benchmarking Joint Gaze-Action Models

To address the absence of joint action-gaze exocentric datasets, Exo-Cook is introduced—derived from cooking domain data within Ego-Exo4D but processed to include aligned gaze heatmaps (via calibrated 3D-to-2D projections of gaze vectors), human and object bounding boxes, and atomic action labels clustered semantically from verb-noun pairs with spaCy and BERT. K-means clustering (with $K=10$ based on the elbow method) is employed for action categorization, enabling scalable annotation and categorization without reliance on expensive manual labeling.

Figure 2: Exo-Cook labels visualized via t-SNE, with representative instances and semantically-clustered atomic actions and verb–noun annotations.

This dataset retains multiview geometry (five cameras) and variable scene complexity, offering 32,050 annotated clips with coherent gaze, bounding box, and action information. The label alignment step further ensures the temporal link between each modality is reliable, critical for the intended joint modeling.

SAGE Framework

Architecture Overview

SAGE is based on a transformer backbone with specialized modules:

• Feature Encoder: Converts each video frame into tokens encoding human-object-pair spatial relations and a global context token, compatible with both egocentric and exocentric inputs.
• Gaze Detection Module: Predicts probabilistic gaze heatmaps per frame, employing a backbone plug-in (e.g., GLC, VideoAttn, Sharingan).
• Gaze-Conditioned Spatial Attention (GCSA): Injects object-wise gaze relevance as an additive bias into the spatial self-attention mechanism within each transformer encoder layer. Objects attended to by gaze receive elevated attention scores, directly biasing HOI modeling.
• Gaze Anticipation Module: Predicts future gaze trajectories as a sequence, using embedded gaze heatmaps and leveraging context features and cross-attention with the interaction tokens.
• Gaze-Conditioned Temporal Prediction (GCTP): Cross-modality temporal module aligning future gaze tokens with prior interaction encodings (from GCSA), enabling gaze-driven action anticipation for future frames.

  Figure 3: Overview of SAGE architecture. GCSA integrates gaze-conditioned spatial attention; GCTP enables gaze-conditioned temporal action anticipation; uncertainty estimation is realized via MC-sampled gaze perturbations.

The overall modeling factorization explicitly marginalizes over stochastic gaze predictions for current and future frames, propagating gaze uncertainty and analyzing its impact on action recognition and anticipation.

Experimental Results

Task Formulation and Metrics

SAGE evaluates four interconnected tasks:

1. Gaze Detection: Predict fixations for current stimuli (F1, recall, precision).
2. Action Recognition: Identify ongoing actions or HOIs (mAP, accuracy, F1).
3. Gaze Anticipation: Regress likely future attention points.
4. Action Anticipation: Predict future human-object interactions.

The model is benchmarked on Vid-HOI (HOI exocentric), EGTEA Gaze+ (egocentric gaze-action), and Exo-Cook.

Ablation: Importance of Explicit Gaze-Conditioned Modules

Ablation studies systematically isolate the effect of multi-task joint training from the explicit coupling introduced by GCSA (for spatial gaze bias) and GCTP (for future gaze-action dependencies). Adding GCSA yields nontrivial improvements in both gaze detection F1 and action recognition accuracy even under otherwise identical joint supervision. Analogously, anticipating future actions conditioned on anticipated gaze via GCTP substantially boosts action anticipation accuracy.

Numerical Performance and Comparison

On Exo-Cook, SAGE with Sharingan as the gaze backbone achieves highest F1 scores for both gaze detection (57.7) and HOI anticipation (67.2), outperforming ST-Gaze by +2.1 mAP in HOI detection and yielding superior recall and precision in both detection and anticipation. Direct comparison with task-specific SOTA baselines across Vid-HOI and EGTEA Gaze+ also shows SAGE either matching or surpassing prior best results in all four tasks, including on detection (F1, mAP) and anticipation metrics—without relying on large-scale language or foundation model pretraining.

Figure 4: Qualitative results: SAGE action and gaze predictions on EGTEA Gaze+ (left) and Exo-Cook (right), covering current and future frames.

Uncertainty Quantification

Propagating Monte Carlo-sampled gaze perturbations through GCSA and GCTP, the model quantifies both predictive entropy and gaze-sensitivity of action inference. Notable is the finding that uncertainty in gaze can have a pronounced impact on both current and anticipated action predictions, especially in egocentric settings, but that SAGE maintains robustness in exocentric ones.

Figure 5: Gaze-conditioned action uncertainty visualization for two scenarios: EGTEA Gaze+ (left—gaze-sensitive) and Exo-Cook (right—robust to gaze perturbation).

Implications and Future Directions

Practically, SAGE's unified approach enables single-model deployment for assistive robots, cognitive monitoring, AR assistants, and human behavioral analysis, with consistent high performance across both egocentric and exocentric visual domains. The explicit uncertainty quantification component supports application in safety-critical settings, where action predictions should be modulated by gaze confidence.

Theoretically, the work demonstrates the value of tightly coupled, bidirectional modeling of gaze and action, rather than the "information bottleneck" designs of previous pipelines. SAGE makes a bold architectural claim that incorporating future gaze as a conditioning latent variable is critical for accurate behavior forecasting, a hypothesis empirically validated with superior results to multi-task and prior joint gaze-action models.

The release of Exo-Cook fills a crucial benchmarking gap, priming further progress in exocentric and multi-view behavior understanding. Full generalization to cross-scene or cross-domain tasks, as well as integration with foundation models for more nuanced action descriptions, are promising future avenues.

Conclusion

SAGE establishes a new benchmark in unified action-gaze modeling by synergizing present and future gaze with HOI recognition and anticipation, outperforming specialized architectures in both egocentric and exocentric contexts. Its modular, viewpoint-adaptive design and strong empirical results—particularly the utility of the GCSA and GCTP modules—underscore the importance of end-to-end, coupled multi-modal learning for comprehensive human behavior understanding (2607.04017).