Group-DINOmics: Incorporating People Dynamics into DINO for Self-supervised Group Activity Feature Learning

Abstract: This paper proposes Group Activity Feature (GAF) learning without group activity annotations. Unlike prior work, which uses low-level static local features to learn GAFs, we propose leveraging dynamics-aware and group-aware pretext tasks, along with local and global features provided by DINO, for group-dynamics-aware GAF learning. To adapt DINO and GAF learning to local dynamics and global group features, our pretext tasks use person flow estimation and group-relevant object location estimation, respectively. Person flow estimation is used to represent the local motion of each person, which is an important cue for understanding group activities. In contrast, group-relevant object location estimation encourages GAFs to learn scene context (e.g., spatial relations of people and objects) as global features. Comprehensive experiments on public datasets demonstrate the state-of-the-art performance of our method in group activity retrieval and recognition. Our ablation studies verify the effectiveness of each component in our method. Code: https://github.com/tezuka0001/Group-DINOmics.

• The paper introduces a self-supervised framework that integrates local motion dynamics and group-relevant object localization into DINOv3 for enhanced activity feature learning.
• It employs pretext tasks of person flow estimation and inpainting-based object localization, yielding state-of-the-art retrieval and recognition on sports datasets.
• The methodology demonstrates robust annotation efficiency and scalability, significantly improving group activity retrieval metrics.

Group-DINOmics: Dynamics- and Context-Aware Self-supervised Group Activity Feature Learning

Introduction and Problem Context

Group-level video understanding, especially in domains such as sports analysis and complex human-robot interaction, critically depends on extracting latent activity features that are both group-aware and dynamics-informed. Conventional supervised group activity recognition (GAR) has been hindered by the need for annotated action vocabularies and intensive manual labeling. Recent self-supervised methods rely predominantly on static appearance cues and are limited in capturing temporal dynamics and high-order people-object interactions crucial to robust group activity representation.

The paper "Group-DINOmics: Incorporating People Dynamics into DINO for Self-supervised Group Activity Feature Learning" (2604.04467) addresses these limitations by introducing a self-supervised GAF learning paradigm that effectively integrates local motion dynamics and global contextual cues into a discriminative latent space using DINOv3 feature extractors, augmented by two pretext tasks: person flow estimation and group-relevant object localization.

Methodological Framework

The authors propose a pipeline that advances the self-supervised learning of Group Activity Features (GAFs) via two targeted pretext tasks over DINOv3 Vision Transformers: (1) person flow estimation to embed local motion signals, and (2) global context modeling through group-relevant object localization, with the latter enforced by inpainting strategies to avoid trivial appearance-based solutions.

Figure 1: Schematic overview contrasting prior self-supervised GAF learning (top) with the proposed approach incorporating person flow estimation and group-relevant object localization (bottom).

This framework employs DINOv3 (ViT-L/16) as an image feature backbone. Each video frame is passed through an inpainting model that erases group-relevant objects, forcing feature extractors to encode non-trivial global information. Features for each frame are then temporally aggregated via a transformer encoder and MLP, and temporally pooled to produce a compact GAF.

During training, the GAF is regularized by the two pretext objectives. Person flow estimation leverages pseudo-labels from external optical flow and detection models, estimating player-centric $\mathbb{R}^2$ displacements per frame. Group-relevant object localization employs masked (inpainted) images to force the GAFs to attend to scene relationships and recover object positions (e.g., ball location in sports) robustly.

Figure 2: Overview of the complete network architecture, including the DINOv3 feature extractor, GAF aggregator, and the two pretext tasks imposed on the learned GAF.

The design of these pretext tasks leverages pseudo-labels only during training, ensuring the approach remains annotation-efficient and readily scalable.

Experiments and Results

Datasets and Evaluation Protocols

The approach is validated on two challenging multi-person group activity datasets: the Volleyball Dataset (VBD) and the NBA Dataset (NBA). Evaluation comprises (a) self-supervised group activity retrieval (GAR) using nearest-neighbor search in the learned GAF space and (b) supervised GAR by attaching a linear classifier for fine-tuning.

Quantitative Comparison

The proposed method achieves state-of-the-art performance for self-supervised GAR retrieval, significantly surpassing prior methods. On VBD, the Hit@1 improves from 61.1% (GAFL) to 82.7%, and on NBA, from 24.7% to 43.9%. These margins (21.6 and 19.2 points, respectively) constitute a strong empirical claim highlighting the effectiveness of integrating dynamics- and context-aware pretext tasks.

In ablation studies, using both flow and object localization objectives in a two-stage training protocol yields best-in-class performance, and the contribution of each loss is disentangled. Auxiliary losses (computed on the image features before temporal aggregation) are shown to be important for optimizing DINOv3 to extract temporally informative cues.

Qualitative Insights

Figure 3: Visual effect of inpainting applied to group-relevant objects, demonstrating improved global feature representation.

Inpainting is demonstrated to outperform black-pixel masking and using raw images, further highlighting the importance of preventing trivial appearance-based solutions.

Figure 4: Visual comparison of group activity retrieval on VBD, showcasing improved activity-level discriminability (e.g., R-set vs. R-spike) with Group-DINOmics.

Qualitative retrievals illustrate how the proposed method retrieves videos with accurate group activity congruence, successfully differentiating subtle dynamics and spatial configurations that elude appearance-centric methods.

Robustness and Architectural Choices

The architecture is robust to noisy pseudo-labels and achieves high performance even with automatically detected person bounding boxes. The authors also demonstrate that fine-tuning only the last two blocks of DINOv3 is optimal; larger updates or adapter-based alternatives yield inferior results.

Different image backbone architectures are benchmarked; DINOv3 outperforms both supervised ViTs, MAE, and CLIP-based models, validating its balanced local-global feature encoding under the proposed pretext optimization.

Supervised Fine-tuning

In the supervised GAR setting, the learned GAFs offer competitive or superior group activity recognition accuracy to prior models using only image-level supervision—93.9% on VBD—demonstrating transferability and robustness of the learned GAF space.

Theoretical and Practical Implications

This work provides strong evidence that pretext tasks explicitly targeting dynamics and people-object global context, when enforced over high-capacity self-supervised backbones, yield more semantically meaningful and discriminative group activity representations. The design avoids heavy reliance on manual annotations and demonstrates scalability through pseudo-labeling. The reported results on activity retrieval and recognition establish new baselines for self-supervised GAR in sports analysis.

Practical implications include annotation-light deployment in real-world multi-person activity monitoring, sports analytics, and anomaly detection. The theoretical contribution justifies the value of architectural synergy between temporal transformers and local-global image encoders when coupled with functionally relevant pretext objectives.

Future Directions

Further generalization beyond ball-centric sports to activities involving more complex objects or non-sport settings, e.g., social group interactions in surveillance or robotic domains, is a clear next step. Extending the object localization objective to handle arbitrary objects (not just balls/goals), refining temporal aggregation protocols, and integrating more complex relational reasoning could further enhance feature discriminability. Integration with large-scale multimodal models or unified video-language pretraining frameworks is also a promising avenue.

Conclusion

The Group-DINOmics method advances self-supervised group activity representation learning by enforcing local dynamics and global people-object contextual awareness over DINOv3 transformers via thoughtfully designed pretext tasks. The approach yields substantial improvements in unsupervised retrieval and supervised recognition benchmarks, robustly leverages large-scale pretrained features, and provides a scalable path toward annotation-efficient multi-person video understanding.