Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformer

Abstract: Recently, Transformer has made significant progress in various vision tasks. To balance computation and efficiency in video tasks, recent works heavily rely on factorized or window-based self-attention. However, these approaches split spatiotemporal correlations between regions of interest in videos, limiting the models' ability to capture motion and long-range dependencies. In this paper, we argue that, similar to the human visual system, the importance of temporal and spatial information varies across different time scales, and attention is allocated sparsely over time through glance and gaze behavior. Is equal consideration of time and space crucial for success in video tasks? Motivated by this understanding, we propose a dual-path network called the Overall Glance and Refined Gaze (OG-ReG) Transformer. The Glance path extracts coarse-grained overall spatiotemporal information, while the Gaze path supplements the Glance path by providing local details. Our model achieves state-of-the-art results on the Kinetics-400, Something-Something v2, and Diving-48, demonstrating its competitive performance. The code will be available at https://github.com/linuxsino/OG-ReG.

• The paper introduces OG-ReG Transformer, which uses a dual-path architecture combining coarse 'Glance' and detailed 'Gaze' mechanisms to enhance spatiotemporal modeling.
• It employs SoDA for spatial-only downsampling while preserving full temporal resolution and MDConv to dynamically fuse 2D and 3D features, reducing computational cost.
• Empirical results on Kinetics-400, SSv2, and Diving-48 V2 demonstrate significant accuracy improvements, underscoring adaptive temporal dynamics' role in action recognition.

Summary of "Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformer"

Introduction and Motivation

This work rigorously addresses the limitations of current Transformer architectures in video action recognition, especially in capturing long-range spatiotemporal dependencies and leveraging temporal information efficiently. Drawing inspiration from human visual cognition, the authors criticize the prevalent use of window-based and factorized attention in existing models, arguing that these approaches sever essential temporal and spatial correlations, impeding both motion modeling and recognition of long-range dependencies. They note the human visual system's sparse, scale-varying attention patterns—deploying rapid, coarse “glances” for overall context and focused “gazes” for fine details.

Figure 1: Window-based attention fails to track objects across frames as effectively as glance-like attention, particularly in cases of significant spatial displacement due to object/camera motion.

OG-ReG Transformer Architecture

The authors propose the Overall Glance and Refined Gaze Transformer (OG-ReG), a dual-path network expressly modeling the dual-process hypothesis: a "Glance" path that aggregates coarse-grained, global spatiotemporal content and a "Gaze" path that injects detailed local information by balancing spatial and temporal focus. This architecture systematically mimics human visual behavior, imbuing the model with adaptive selectivity for space and time across scales.

Figure 2: High-level overview of OG-ReG Transformer and the OG-ReG block, highlighting the separation and interaction between Glance and Gaze pathways.

Glance Path: Spatial-only Downsampling Attention (SoDA)

The Glance path employs the proposed SoDA mechanism, where only spatial dimensions of the token embedding are downsampled—maintaining full temporal resolution. This drastically reduces the computation cost without damaging crucial temporal ordering, in contrast to previous approaches that naively downsample both. This selective spatial downsampling enables efficient extraction of coarse action context. Tokens are linearly projected for self-attention computation over downsampled space, then upsampled to restore input size.

Figure 3: Details of the OG-ReG block, showing spatial-downsampling attention (SoDA) and the flow of information before and after down-/up-sampling operations.

Gaze Path: Masked Dynamic Convolution (MDConv)

The Gaze path integrates MDConv, dynamically balancing contributions from 2D (spatial) and 3D (spatiotemporal) convolutional filters, modulated by statistics extracted from the Glance path’s similarity matrix. This design is motivated by ablation evidence that combining 2D and 3D convolutions, rather than relying exclusively on either, yields superior representations for local action cues. The method introduces attention-driven modulation factors that adjust the influence of each convolution type for each input, reinforcing the role of fine-grained spatial or temporal detail as needed.

Figure 4: The MDConv module, detailing fusion of 2D and 3D convolutional responses based on the Glance path's guidance.

Analysis and Ablation

Empirical investigations uncover that:

• Temporal information at the clip level is critical; aggressive temporal downsampling degrades recognition accuracy on clip-based datasets.
• Spatial redundancy exceeds temporal redundancy, justifying aggressive spatial downsampling in SoDA while avoiding temporal reduction.
• Combining 2D and 3D kernels in MDConv achieves the best balance for modeling fast and slow action tempos.

Visualization of the similarity matrix $\boldsymbol{A}$ across video frames highlights how OG-ReG adaptively models the tempo of actions, adjusting receptive fields and filter modulations for fast and slow sequences.

Figure 5: Visualization of similarity matrices for three video tempos from the same action class, showing OG-ReG's capability in adapting to different action dynamics.

Numerical Results

Kinetics-400

OG-ReG-B achieves 83.0% top-1 and 95.7% top-5 accuracy (ImageNet-21K pretrain), surpassing Video-Swin-B (by +0.3% top-1) and PST-B, with fewer or comparable FLOPs and more effective attention utilization (see Table k400cmp). Notably, OG-ReG-T (79.5% top-1) outperforms Video-Swin-T and PST-T by 0.7% and 1.3%, respectively.

Something-Something V2 (SSv2)

OG-ReG-B yields 71.7% top-1 and 93.1% top-5 accuracy (ImageNet-21K+Kinetics-400 pretrain), outperforming Video-Swin-B and PST-B by 2.1% and 2.5%, respectively, and providing clear evidence of superior temporal modeling capacity.

Diving-48 V2

On fine-grained temporal tasks, OG-ReG-B reaches 88.1% top-1 and 98.9% top-5 accuracy, setting a new benchmark beyond state-of-the-art Transformer and convolutional alternatives.

These results robustly support the authors’ claim that differentiating and hierarchically fusing spatial and temporal cues, rather than treating them equally across all scales, is essential for optimal video understanding.

Theoretical and Practical Implications

The OG-ReG proposal fundamentally reframes the standard approach to video transformer design. It challenges the uniform treatment of time and space, providing strong empirical and theoretical evidence that their relevance and redundancy shift across scales and contexts. Its analogy to glance/gaze mechanism from human vision constructs a formal bridge between cognitive neuroscience and deep video models. Practically, the architecture provides a blueprint for the next generation of scalable, efficient video transformers applicable to both spatially heavy (e.g., K400) and temporally heavy (e.g., SSv2) tasks.

The Fourier spectrum analysis further reveals that SoDA predominantly captures low-frequency (global, slow-changing) signals, while MDConv enhances high-frequency (local, fast-changing) components, furnishing complementary signal bands in the representation.

Figure 6: Fourier analysis shows OG-ReG-T’s features have clearer, multi-band spectral energy, compared to Video-Swin-B.

Visualization and Interpretability

The Grad-CAM visualizations substantiate that OG-ReG’s saliency tracks both object and camera motion, in contrast to windowed-attention models whose static saliency consistently locks to central regions irrespective of true action cues.

Figure 7: OG-ReG dynamically attends to moving objects/regions, even with independently moving cameras, while Video-Swin-B’s saliency remains fixed.

Future Directions

The present architecture, while advancing selectivity and efficiency, still falls short of the anticipatory dynamic allocation of attention observed in biological vision. Future work should prioritize learnable mechanisms that dynamically predict and allocate spatial and temporal resources—not only based on fixed sampling, but potentially through event-driven or predictive paradigms, efficiently targeting salient cues.

Conclusion

This work presents a compelling, systematically-validated strategy for video action recognition via dual-path hierarchical attention, adaptive convolution, and explicit exploitation of temporal dynamics. By explicitly aligning architectural bias with cognitive principles observed in human vision, OG-ReG demonstrates substantial improvements in accuracy and computational efficiency across canonical datasets. The implications extend beyond action recognition, indicating that future video understanding models should abandon rigid, homogeneous treatments of space and time in favor of adaptive, scale-aware mechanisms that better reflect the statistics and semantics of natural actions.

Reference: "Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformer" (2604.06783)