Group Activity Feature Learning Overview

• Group Activity Feature Learning (GAFL) is a method for deriving compact, interpretable representations of multi-person activities from individual, contextual, and relational cues.
• It leverages hierarchical LSTMs, graph neural networks, and transformer models to fuse person-level and scene-level data into a latent and discriminative group descriptor.
• Empirical results demonstrate that GAFL enhances group activity recognition, prediction, and retrieval by integrating weakly supervised, self-supervised, and human-in-the-loop adaptation techniques.

Searching arXiv for recent and foundational papers on group activity feature learning and related group activity representation learning. Group Activity Feature Learning (GAFL) denotes the learning of compact, discriminative representations of multi-person behavior from person-level, scene-level, relational, or contextual evidence for tasks such as group activity recognition, social group activity recognition, prediction, and retrieval. In recent literature the term appears explicitly as Group Activity Feature or GAF learning, where a clip is mapped to a vector such as $\mathbf{G}\in\mathbb{R}^{D}$ or $\mathbf{G}_k\in\mathbb{R}^{2C}$ without requiring group activity annotations during pretraining (Tezuka et al., 6 Apr 2026, Nakatani et al., 2024, Nakatani et al., 3 Feb 2026). Earlier work develops the same idea in substance through hierarchical person-to-group LSTMs, deep structured message passing, action codes, semantic relation graphs, partial context embeddings, and group queries that are learned rather than hand-crafted (Ibrahim et al., 2015, Deng et al., 2015, Gammulle et al., 2018, Kim et al., 2022, Tamura et al., 2022).

1. Scope, task formulations, and historical development

The reviewed literature treats group activity understanding as a multi-person video problem in which the central output is a collective label, but the exact task formulation varies. Standard group activity recognition predicts a single group label for a clip or frame sequence, such as volleyball actions or collective pedestrian activities. Social group activity recognition expands the problem to multiple sub-group activities and explicit group-member identification. Group activity prediction uses only partial observations and anticipates the group label before the activity is fully executed. Group activity retrieval discards predefined class prediction during pretraining and instead learns a feature space in which similar clips are close (Tamura et al., 2022, Chen et al., 2020, Nakatani et al., 2024, Nakatani et al., 3 Feb 2026).

The survey literature characterizes GAR as requiring efficient modeling of hierarchical relationships within a scene and accurate extraction of distinctive spatiotemporal features from groups, and organizes the field into spatial structure, descriptors, non-deep learning, HRNN, relationship models, and attention mechanisms (Wang et al., 2023). That taxonomy is mirrored in the historical trajectory of GAFL. Early systems use hierarchical temporal composition from persons to groups or structured message passing over scene, action, and pose variables (Ibrahim et al., 2015, Deng et al., 2015). Later systems emphasize explicit relation graphs, reinforcement-driven frame and relation selection, and transformer-based interaction modeling (Hu et al., 2019, Han et al., 2022). More recent work broadens the supervision regimes: detector-free weak supervision with only group labels, knowledge-augmented relation inference, attribute-prediction-based GAF learning, self-supervised DINO adaptation, and human-in-the-loop retrieval-time adaptation (Kim et al., 2022, Lang et al., 2023, Nakatani et al., 2024, Tezuka et al., 6 Apr 2026, Nakatani et al., 3 Feb 2026).

A representative notation appears in the self-supervised GAF literature: a video is written as $$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$ and mapped to a compact feature $\mathbf{G}\in\mathbb{R}^{D}$ (Tezuka et al., 6 Apr 2026). In MLS-GAN, the generator input is the union of all person sequences and the scene sequence,

$$I_G=(\{x^1_1,\ldots,x^1_T\},\ldots,\{x^N_1,\ldots,x^N_T\},\{\hat{X}_1,\ldots,\hat{X}_T\}),$$

which makes explicit that GAFL is usually built from both individual streams and global context rather than from a single monolithic scene descriptor (Gammulle et al., 2018).

2. Representational substrates: persons, scenes, keypoints, objects, and tokens

A persistent design axis in GAFL is the choice of primitive representation. Person-centric pipelines extract a feature for every tracked or detected actor, usually from RoIAlign over a CNN backbone. The hierarchical deep temporal model uses AlexNet features on person crops, followed by person-level LSTMs whose hidden states are pooled into a group representation (Ibrahim et al., 2015). Deep structured models likewise use separate AlexNet-style CNNs for scene, action, and pose, and then refine those outputs through learned factor layers that mimic message passing (Deng et al., 2015). MLS-GAN uses a shared ResNet-50, with scene-level features $\hat{\theta}_t=f(\hat{X}_t)$ and person-level features $\theta_t^n=f(x_t^n)$, then temporally encodes both via LSTMs into $\hat{Z}$ and $Z^n$ (Gammulle et al., 2018).

Graph-centered methods preserve person nodes and add relational structure. Improved Actor Relation Graph models represent each actor with appearance feature $x_i^a$ and position feature $\mathbf{G}_k\in\mathbb{R}^{2C}$0, then build an adjacency matrix from appearance similarity and spatial proximity before applying GCN reasoning (Kuang et al., 2020). Progressive Relation Learning formalizes a Semantic Relation Graph with node attributes, edge attributes, and a global attribute, and then uses a feature-distilling agent to refine temporal evidence and a relation-gating agent to refine the graph (Hu et al., 2019). Sequential Relational Anticipation Model builds two graphs per frame, one for action similarity and one for positional proximity, and uses graph auto-encoders to anticipate future features and positions (Chen et al., 2020).

Transformer-era work broadens the substrate beyond actor crops. Dual-AI still starts from actor tensors $\mathbf{G}_k\in\mathbb{R}^{2C}$1, but processes them through complementary Spatial-Temporal and Temporal-Spatial transformer paths (Han et al., 2022). Detector-Free Weakly Supervised Group Activity Recognition eliminates boxes and detectors entirely, learning a clip as a set of partial context embeddings generated by tokens attending to spatial feature maps and then aggregated by temporal 1D convolutions and token self-attention (Kim et al., 2022). Social group activity recognition with transformers introduces learnable group queries over multi-scale I3D features so that each query learns a group descriptor that contains both activity information and group-member information (Tamura et al., 2022).

A separate strand replaces RGB actor crops with more structured motion proxies. DynamicFormer is explicitly keypoint-only, combining human keypoints, person features, subgroup features, and object coordinates such as the volleyball (Zhang et al., 2023). Group-DINOmics uses DINOv3 frame features $\mathbf{G}_k\in\mathbb{R}^{2C}$2, temporal transformer encoding, and temporal pooling to obtain a video-level $\mathbf{G}_k\in\mathbb{R}^{2C}$3, then injects per-person location encodings and object pseudo-labels through pretext tasks (Tezuka et al., 6 Apr 2026). The 2024 GAF-learning framework based on person attribute prediction retains a person-centric extractor but treats the resulting group vector as the central object, to be decoded back into person attributes via location guidance (Nakatani et al., 2024).

3. Composition mechanisms: from individual evidence to group representations

The mechanisms that compose primitive features into group features differ substantially across the literature. Hierarchical temporal models perform a bottom-up composition: per-person CNN features are passed through person-level LSTMs, concatenated with the original person descriptors, max-pooled across people into a frame-level group feature, and then processed by a second LSTM for whole-activity understanding (Ibrahim et al., 2015). Deep structured models use learned factor neurons instead of pooling alone. They define scene–action–pose factors and scene–global-pose factors, and refine scene, action, and pose scores by alternating variable-to-factor and factor-to-variable updates, effectively learning group-aware compatibility features (Deng et al., 2015).

In MLS-GAN, the key composition operator is the Gated Fusion Unit. After person and scene temporal embeddings are obtained, each person stream is embedded by

$\mathbf{G}_k\in\mathbb{R}^{2C}$4

then gated by all streams,

$\mathbf{G}_k\in\mathbb{R}^{2C}$5

modulated as $\mathbf{G}_k\in\mathbb{R}^{2C}$6, and summed with the scene stream to yield

$\mathbf{G}_k\in\mathbb{R}^{2C}$7

The generator then maps this fused feature and noise to an action code

$\mathbf{G}_k\in\mathbb{R}^{2C}$8

a dense vector in $\mathbf{G}_k\in\mathbb{R}^{2C}$9 that serves as an intermediate group representation (Gammulle et al., 2018). This makes the group feature explicitly latent, dense, and adversarially shaped rather than a direct softmax head.

Graph-based composition remains central in relation models. In the improved Actor Relation Graph family, the normalized relation weight is

$$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$0

with $$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$1 instantiated by embedded dot product, NCC, or SAD, and $$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$2 given by a distance mask (Kuang et al., 2020). Progressive Relation Learning adds policy-controlled refinement on top of such relational structure, using the feature-distilling agent to retain informative frames and the relation-gating agent to impose structured sparsity over graph rows so that only a few key participants maintain strong connections (Hu et al., 2019).

Transformer-based GAFL typically replaces fixed pooling or hand-designed edges with learned attention patterns. Dual-AI computes one path that first performs spatial interaction and then temporal interaction, and another that reverses the order, thereby producing complementary actor interaction features (Han et al., 2022). Detector-free weak supervision uses tokens that attend to learned partial contexts, then aggregates these contexts temporally and relationally into a single group representation (Kim et al., 2022). Social group activity recognition treats groups as DETR-style objects: decoder queries attend to multi-scale scene features and directly predict activity, size, and member points, turning the group descriptor itself into the carrier of membership structure (Tamura et al., 2022). DynamicFormer factors the composition process into a Dynamic-composition Module for person–person relations and a Dynamic-interaction Module for human–object interaction, with a Multi-level Dynamic Integration Transformer to fuse keypoint-, person-, subgroup-, composition-, and interaction-level features (Zhang et al., 2023).

Knowledge-augmented models inject priors into the composition pipeline rather than only into the classifier. Knowledge Augmented Relation Inference defines a Class–Class Distribution Map $$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$3 and a Class–Position Distribution Map $$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$4 from training statistics, then adds them to semantic self-attention and person-to-action cross-attention so that co-occurrence structure and spatial priors explicitly shape the learned person and group features (Lang et al., 2023).

4. Supervision regimes and objective design

GAFL spans a wide range of supervision regimes. Fully supervised GAR commonly uses group labels and often person action labels. Hierarchical deep temporal models train first on person actions and then on group activities (Ibrahim et al., 2015). Deep structured models apply cross-entropy losses to scene, action, and pose outputs at each message-passing step (Deng et al., 2015). Dual-AI jointly optimizes group and individual predictions,

$$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$5

where $$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$6 is the Multi-scale Actor Contrastive Loss enforcing frame–frame, frame–video, and video–video actor consistency across the two transformer paths (Han et al., 2022).

Semi-supervised and weakly supervised formulations shift the source of supervision while preserving group-level semantics. MLS-GAN trains the generator and discriminator with a joint adversarial and classification objective,

$$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$7

with $$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$8 (Gammulle et al., 2018). Detector-Free Weakly Supervised GAR uses only video-level group labels, no bounding boxes, no individual action labels, and no detector at test time; localization of relevant regions emerges from attention under standard cross-entropy on the final group label (Kim et al., 2022).

The 2024 GAF-learning framework replaces group supervision with person-attribute supervision. Its central decoder is

$$\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$$9

where the APN predicts either person action classes or person appearance features for each person from the shared GAF and that person’s location feature (Nakatani et al., 2024). This yields two variants: GAFL-PAC, which uses person action labels, and GAFL-PAF, which uses appearance features and requires no manual labels. Masked Person Modeling further forces the GAF to encode context by zeroing out a subset of person features during training (Nakatani et al., 2024).

Self-supervised GAF learning goes further by avoiding group labels entirely and supervising the latent space through dynamics and context pretext tasks. Group-DINOmics predicts person flow from the GAF by

$\mathbf{G}\in\mathbb{R}^{D}$0

and predicts group-relevant object location by

$\mathbf{G}\in\mathbb{R}^{D}$1

with auxiliary heads from per-frame DINO features and two-stage optimization: first flow loss, then object loss (Tezuka et al., 6 Apr 2026). Human-in-the-loop adaptation starts from a self-supervised GAF space and fine-tunes it per query using binary positive/negative feedback, a triplet-style contrastive loss, and a regularization term that keeps selected GAFs close to their pretrained values (Nakatani et al., 3 Feb 2026). A plausible implication is that GAFL has shifted from being only a recognition backbone to being a reusable representation space that can be specialized after pretraining for retrieval or analyst-driven search.

5. Empirical behavior across recognition, prediction, and retrieval

Reported results are not directly interchangeable across supervision regimes, datasets, or evaluation protocols, but several consistent empirical patterns recur. Methods that enrich person features with scene context, learned relations, or explicit latent group codes almost always outperform direct pooling baselines. Detector-free and label-free feature learning can also be competitive when the representation is constrained by strong relational or self-supervised objectives (Gammulle et al., 2018, Kim et al., 2022, Nakatani et al., 2024, Tezuka et al., 6 Apr 2026).

Paradigm	Setting	Reported result
MLS-GAN (Gammulle et al., 2018)	Collective Activity / Volleyball	91.7 / 91.2 MCA/MPCA on Collective Activity; 93.0 MCA and 92.4 MPCA on Volleyball
Detector-Free WSGAR (Kim et al., 2022)	NBA / Volleyball	75.8% MCA, 71.2% MPCA on NBA; 90.5% MCA, 94.4% merged MCA on Volleyball
Dual-AI (Han et al., 2022)	Volleyball	94.4% group activity, 84.4% individual with RGB only; 95.4% with optical flow fusion
DynamicFormer (Zhang et al., 2023)	Volleyball / Collective Activity	95.3% group activity and 85.4% individual on Volleyball; 94.4% group activity on Collective Activity
KARI (Lang et al., 2023)	Volleyball / Collective Activity	94.5% MCA on Volleyball; 92.8% MCA and 98.5% MPCA on Collective Activity
GAFL via person attributes (Nakatani et al., 2024)	Retrieval	GAFL-PAC group activity Hit@1 of 84.8 on VBD and 94.9 on CAD; GAFL-PAF group activity Hit@1 of 61.1 on VBD and 88.5 on CAD
Group-DINOmics (Tezuka et al., 6 Apr 2026)	Retrieval / supervised fine-tuning	VBD retrieval Hit@1 82.7 and Hit@3 93.0; supervised VBD 93.9% MCA and 96.1% Merged MCA

Ablations provide some of the clearest evidence for what GAFL actually learns. In MLS-GAN on Collective Activity, direct supervised classification without GAN and without an action code reaches only 60.5 MPCA, whereas the conditional GAN variant reaches 87.7 MPCA and the full MLS-GAN reaches 91.2 MPCA, indicating that the intermediate action-code space and gated fusion substantially improve the learned group representation (Gammulle et al., 2018). In detector-free weak supervision on NBA, a ResNet + global average pooling baseline gives 58.4% MCA and 51.7% MPCA, adding partial context embedding raises this to 64.1% and 58.5%, the designed temporal-and-context aggregation raises it to 73.6% MCA, and motion augmentation further raises it to 75.8% (Kim et al., 2022). In attribute-prediction-based GAFL, removing location guidance drops group activity Hit@1 from 84.8 to 69.5 for GAFL-PAC on VBD and from 88.5 to 57.1 for GAFL-PAF on CAD, showing that location-guided decoding is not a minor detail but a major determinant of feature quality (Nakatani et al., 2024).

Retrieval-oriented GAFL exposes a different empirical aspect: the discriminative structure of the feature space itself. Group-DINOmics improves VBD retrieval Hit@1 from 43.0 for frozen DINOv3 without pretext losses to 75.4 with only flow loss, 74.0 with only object loss, and 82.7 with the full two-stage schedule; on NBA it reaches 43.9 Hit@1 and 72.0 Hit@3 (Tezuka et al., 6 Apr 2026). Human-in-the-loop adaptation then further reshapes that space according to analyst intent: on Volleyball, GAFL baseline Precision@10 (original) is 0.533, while human-in-the-loop adaptation reaches 0.647 using 3 query videos and 5 selected videos to label, and on NBA it improves Precision@10 (original) from 0.206 to 0.233 (Nakatani et al., 3 Feb 2026). This suggests that modern GAFL is increasingly evaluated not only by classification accuracy but also by the geometry and adaptability of the learned embedding.

6. Limitations, misconceptions, and emerging directions

Several papers explicitly motivate alternatives to plain person-feature pooling or fixed detector-centric pipelines. Region-feature summarization can be brittle because it depends on complete person localization and may ignore crucial context such as balls, nets, scoreboards, or referees; detector-free transformers, partial-context tokens, and group-query decoders were introduced precisely to address that limitation (Kim et al., 2022, Tamura et al., 2022). Likewise, several models argue that group activity features should not be reduced to majority voting over person actions or to direct scene classification; instead they benefit from learned latent group codes, semantic relation graphs, knowledge-modulated attention, or attribute-prediction bottlenecks (Gammulle et al., 2018, Hu et al., 2019, Lang et al., 2023, Nakatani et al., 2024).

At the same time, many limitations remain recurrent. A large portion of the literature still relies on accurate detections, bounding boxes, and tracklets; this dependence is explicit in hierarchical LSTMs, MLS-GAN, ARG-style models, attribute-prediction GAFL, and Group-DINOmics (Ibrahim et al., 2015, Gammulle et al., 2018, Kuang et al., 2020, Nakatani et al., 2024, Tezuka et al., 6 Apr 2026). Sequence lengths are often short—10 frames in MLS-GAN and DynamicFormer, 9 or 17 frames in several transformer models, 10 or 12 frames in Group-DINOmics—so very long-horizon group dynamics are usually outside the modeled window (Gammulle et al., 2018, Zhang et al., 2023, Tezuka et al., 6 Apr 2026). Some methods note a fixed maximum number of persons, token diversity problems, sensitivity to rare group sizes, or biases introduced by ordered member-point targets (Gammulle et al., 2018, Kim et al., 2022, Tamura et al., 2022). Knowledge-augmented approaches acknowledge that their C–C and C–P maps are static global statistics rather than context-dependent knowledge (Lang et al., 2023). Self-supervised DINO-based GAFL reports strong results on ball-centric sports but has not yet demonstrated the same breadth on arbitrary surveillance or social scenes (Tezuka et al., 6 Apr 2026).

The future directions proposed in the literature are correspondingly varied. MLS-GAN suggests graph neural networks instead of simple concatenation for gating inputs, longer sequences, hierarchical temporal scales, and end-to-end backbone learning with the GAN objective (Gammulle et al., 2018). Detector-free weak supervision suggests token-diversity regularization and hybrid detector-free plus detector-based models (Kim et al., 2022). Social group transformers point toward permutation-invariant member-set prediction, dynamic query allocation, and multimodal inputs (Tamura et al., 2022). DynamicFormer suggests richer human–object interaction modeling, unsupervised or self-supervised composition learning, and broader object vocabularies (Zhang et al., 2023). Knowledge-augmented relation inference points to richer knowledge sources, learnable knowledge maps, and context-dependent priors (Lang et al., 2023). Group-DINOmics suggests extending group-relevant objects beyond the ball or goal and replacing simple temporal pooling with more powerful temporal aggregation (Tezuka et al., 6 Apr 2026). Human-in-the-loop GAFL suggests multi-round active adaptation and more efficient online fine-tuning of the embedding space (Nakatani et al., 3 Feb 2026).

Taken together, these directions indicate that GAFL has become a unifying perspective on group activity understanding: the field increasingly treats the central problem not merely as assigning a label to a clip, but as constructing a structured latent space in which multi-person dynamics, context, interaction, and analyst intent can all be represented, compared, and adapted.