---
title: Group Activity Feature Learning Overview
url: https://www.emergentmind.com/topics/group-activity-feature-learning-gafl
type: topic
---

# Group Activity Feature Learning Overview

Searching arXiv for recent and foundational papers on group activity feature learning and related group activity representation learning.
Group Activity Feature Learning (GAFL) denotes the learning of compact, discriminative representations of multi-person behavior from person-level, scene-level, relational, or contextual evidence for tasks such as group activity recognition, social group activity recognition, prediction, and retrieval. In recent literature the term appears explicitly as **Group Activity Feature** or **GAF** learning, where a clip is mapped to a vector such as $\mathbf{G}\in\mathbb{R}^{D}$ or $\mathbf{G}_k\in\mathbb{R}^{2C}$ without requiring group activity annotations during pretraining [2604.04467, 2403.02753, 2602.03157]. Earlier work develops the same idea in substance through hierarchical person-to-group LSTMs, deep structured message passing, action codes, semantic relation graphs, partial context embeddings, and group queries that are learned rather than hand-crafted [1511.06040, 1506.04191, 1812.07124, 2204.02139, 2207.05254].

## 1. Scope, task formulations, and historical development

The reviewed literature treats group activity understanding as a multi-person video problem in which the central output is a collective label, but the exact task formulation varies. Standard **group activity recognition** predicts a single group label for a clip or frame sequence, such as volleyball actions or collective pedestrian activities. **Social group activity recognition** expands the problem to multiple sub-group activities and explicit group-member identification. **Group activity prediction** uses only partial observations and anticipates the group label before the activity is fully executed. **Group activity retrieval** discards predefined class prediction during pretraining and instead learns a feature space in which similar clips are close [2207.05254, 2008.02441, 2403.02753, 2602.03157].

The survey literature characterizes GAR as requiring efficient modeling of hierarchical relationships within a scene and accurate extraction of distinctive spatiotemporal features from groups, and organizes the field into spatial structure, descriptors, non-deep learning, HRNN, relationship models, and attention mechanisms [2307.13541]. That taxonomy is mirrored in the historical trajectory of GAFL. Early systems use hierarchical temporal composition from persons to groups or structured message passing over scene, action, and pose variables [1511.06040, 1506.04191]. Later systems emphasize explicit relation graphs, reinforcement-driven frame and relation selection, and transformer-based interaction modeling [1908.02948, 2204.02148]. More recent work broadens the supervision regimes: detector-free weak supervision with only group labels, knowledge-augmented relation inference, attribute-prediction-based GAF learning, self-supervised DINO adaptation, and human-in-the-loop retrieval-time adaptation [2204.02139, 2302.14350, 2403.02753, 2604.04467, 2602.03157].

A representative notation appears in the self-supervised GAF literature: a video is written as $\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$ and mapped to a compact feature $\mathbf{G}\in\mathbb{R}^{D}$ [2604.04467]. In MLS-GAN, the generator input is the union of all person sequences and the scene sequence,
$$
I_G=(\{x^1_1,\ldots,x^1_T\},\ldots,\{x^N_1,\ldots,x^N_T\},\{\hat{X}_1,\ldots,\hat{X}_T\}),
$$
which makes explicit that GAFL is usually built from both individual streams and global context rather than from a single monolithic scene descriptor [1812.07124].

## 2. Representational substrates: persons, scenes, keypoints, objects, and tokens

A persistent design axis in GAFL is the choice of primitive representation. Person-centric pipelines extract a feature for every tracked or detected actor, usually from RoIAlign over a CNN backbone. The hierarchical deep temporal model uses AlexNet features on person crops, followed by person-level LSTMs whose hidden states are pooled into a group representation [1511.06040]. Deep structured models likewise use separate AlexNet-style CNNs for scene, action, and pose, and then refine those outputs through learned factor layers that mimic message passing [1506.04191]. MLS-GAN uses a shared ResNet-50, with scene-level features $\hat{\theta}_t=f(\hat{X}_t)$ and person-level features $\theta_t^n=f(x_t^n)$, then temporally encodes both via LSTMs into $\hat{Z}$ and $Z^n$ [1812.07124].

Graph-centered methods preserve person nodes and add relational structure. Improved Actor Relation Graph models represent each actor with appearance feature $x_i^a$ and position feature $x_i^s$, then build an adjacency matrix from appearance similarity and spatial proximity before applying GCN reasoning [2010.12968]. Progressive Relation Learning formalizes a Semantic Relation Graph with node attributes, edge attributes, and a global attribute, and then uses a feature-distilling agent to refine temporal evidence and a relation-gating agent to refine the graph [1908.02948]. Sequential Relational Anticipation Model builds two graphs per frame, one for action similarity and one for positional proximity, and uses graph auto-encoders to anticipate future features and positions [2008.02441].

Transformer-era work broadens the substrate beyond actor crops. Dual-AI still starts from actor tensors $\mathbf{X}\in\mathbb{R}^{K\times N\times C}$, but processes them through complementary Spatial-Temporal and Temporal-Spatial transformer paths [2204.02148]. Detector-Free Weakly Supervised Group Activity Recognition eliminates boxes and detectors entirely, learning a clip as a set of **partial context embeddings** generated by tokens attending to spatial feature maps and then aggregated by temporal 1D convolutions and token self-attention [2204.02139]. Social group activity recognition with transformers introduces learnable **group queries** over multi-scale I3D features so that each query learns a group descriptor that contains both activity information and group-member information [2207.05254].

A separate strand replaces RGB actor crops with more structured motion proxies. DynamicFormer is explicitly keypoint-only, combining human keypoints, person features, subgroup features, and object coordinates such as the volleyball [2305.05583]. Group-DINOmics uses DINOv3 frame features $\mathbf{I}^t$, temporal transformer encoding, and temporal pooling to obtain a video-level $\mathbf{G}$, then injects per-person location encodings and object pseudo-labels through pretext tasks [2604.04467]. The 2024 GAF-learning framework based on person attribute prediction retains a person-centric extractor but treats the resulting group vector as the central object, to be decoded back into person attributes via location guidance [2403.02753].

## 3. Composition mechanisms: from individual evidence to group representations

The mechanisms that compose primitive features into group features differ substantially across the literature. Hierarchical temporal models perform a bottom-up composition: per-person CNN features are passed through person-level LSTMs, concatenated with the original person descriptors, max-pooled across people into a frame-level group feature, and then processed by a second LSTM for whole-activity understanding [1511.06040]. Deep structured models use learned factor neurons instead of pooling alone. They define scene–action–pose factors and scene–global-pose factors, and refine scene, action, and pose scores by alternating variable-to-factor and factor-to-variable updates, effectively learning group-aware compatibility features [1506.04191].

In MLS-GAN, the key composition operator is the **Gated Fusion Unit**. After person and scene temporal embeddings are obtained, each person stream is embedded by
$$
h^n=\tanh(\dot{W}^n Z^n),
$$
then gated by all streams,
$$
q^n=\sigma\!\left(\bar{W}^n [Z^1,Z^2,\ldots,Z^N,\hat{Z}]\right),
$$
modulated as $r^n=h^n\times q^n$, and summed with the scene stream to yield
$$
C=\sum_{j=1}^{N} r^n+\hat{r}.
$$
The generator then maps this fused feature and noise to an **action code**
$$
y=FC(C,z),
$$
a dense vector in $\mathbb{R}^{1\times k}$ that serves as an intermediate group representation [1812.07124]. This makes the group feature explicitly latent, dense, and adversarially shaped rather than a direct softmax head.

Graph-based composition remains central in relation models. In the improved Actor Relation Graph family, the normalized relation weight is
$$
\mathbf{G}_{ij}=
\frac{f_s(\mathbf{x}_i^s,\mathbf{x}_j^s)\exp(f_a(\mathbf{x}_i^a,\mathbf{x}_j^a))}
{\sum_{j=1}^{N} f_s(\mathbf{x}_i^s,\mathbf{x}_j^s)\exp(f_a(\mathbf{x}_i^a,\mathbf{x}_j^a))},
$$
with $f_a$ instantiated by embedded dot product, NCC, or SAD, and $f_s$ given by a distance mask [2010.12968]. Progressive Relation Learning adds policy-controlled refinement on top of such relational structure, using the feature-distilling agent to retain informative frames and the relation-gating agent to impose structured sparsity over graph rows so that only a few key participants maintain strong connections [1908.02948].

Transformer-based GAFL typically replaces fixed pooling or hand-designed edges with learned attention patterns. Dual-AI computes one path that first performs spatial interaction and then temporal interaction, and another that reverses the order, thereby producing complementary actor interaction features [2204.02148]. Detector-free weak supervision uses tokens that attend to learned **partial contexts**, then aggregates these contexts temporally and relationally into a single group representation [2204.02139]. Social group activity recognition treats groups as DETR-style objects: decoder queries attend to multi-scale scene features and directly predict activity, size, and member points, turning the group descriptor itself into the carrier of membership structure [2207.05254]. DynamicFormer factors the composition process into a Dynamic-composition Module for person–person relations and a Dynamic-interaction Module for human–object interaction, with a Multi-level Dynamic Integration Transformer to fuse keypoint-, person-, subgroup-, composition-, and interaction-level features [2305.05583].

Knowledge-augmented models inject priors into the composition pipeline rather than only into the classifier. Knowledge Augmented Relation Inference defines a Class–Class Distribution Map $\mathbf{P}^{cc}$ and a Class–Position Distribution Map $\mathbf{P}^{cp}$ from training statistics, then adds them to semantic self-attention and person-to-action cross-attention so that co-occurrence structure and spatial priors explicitly shape the learned person and group features [2302.14350].

## 4. Supervision regimes and objective design

GAFL spans a wide range of supervision regimes. Fully supervised GAR commonly uses group labels and often person action labels. Hierarchical deep temporal models train first on person actions and then on group activities [1511.06040]. Deep structured models apply cross-entropy losses to scene, action, and pose outputs at each message-passing step [1506.04191]. Dual-AI jointly optimizes group and individual predictions,
$$
\mathcal{L}=\mathcal{L}_{cls}+\mathcal{L}_{MAC},
$$
where $\mathcal{L}_{MAC}$ is the Multi-scale Actor Contrastive Loss enforcing frame–frame, frame–video, and video–video actor consistency across the two transformer paths [2204.02148].

Semi-supervised and weakly supervised formulations shift the source of supervision while preserving group-level semantics. MLS-GAN trains the generator and discriminator with a joint adversarial and classification objective,
$$
L_{GAN}(G,D)=\min_G \max_D\ \mathbb{E}[\log D(I_{D1},I_{D2})]+\mathbb{E}[\log(1-D(I_{D1},G(I_G,z)))]+\lambda_c\,\mathbb{E}[\log D_c(k\mid I_{D1},I_{D2})],
$$
with $\lambda_c=2.5$ [1812.07124]. Detector-Free Weakly Supervised GAR uses only video-level group labels, no bounding boxes, no individual action labels, and no detector at test time; localization of relevant regions emerges from attention under standard cross-entropy on the final group label [2204.02139].

The 2024 GAF-learning framework replaces group supervision with person-attribute supervision. Its central decoder is
$$
\bm{A}_{PRED}^{p}=APN(\bm{G},\bm{F}_{loc}^{p}),
$$
where the APN predicts either person action classes or person appearance features for each person from the shared GAF and that person’s location feature [2403.02753]. This yields two variants: GAFL-PAC, which uses person action labels, and GAFL-PAF, which uses appearance features and requires no manual labels. Masked Person Modeling further forces the GAF to encode context by zeroing out a subset of person features during training [2403.02753].

Self-supervised GAF learning goes further by avoiding group labels entirely and supervising the latent space through dynamics and context pretext tasks. Group-DINOmics predicts person flow from the GAF by
$$
\mathbf{F}_{\mathbf{G}}^{p,t}=F(\mathbf{G}+\mathbf{L}^{p,t}+\mathbf{T}^{t}),
$$
and predicts group-relevant object location by
$$
\mathbf{O}_{\mathbf{G}}^{t}=O(\mathbf{G}+\mathbf{T}^{t}),
$$
with auxiliary heads from per-frame DINO features and two-stage optimization: first flow loss, then object loss [2604.04467]. Human-in-the-loop adaptation starts from a self-supervised GAF space and fine-tunes it per query using binary positive/negative feedback, a triplet-style contrastive loss, and a regularization term that keeps selected GAFs close to their pretrained values [2602.03157]. A plausible implication is that GAFL has shifted from being only a recognition backbone to being a reusable representation space that can be specialized after pretraining for retrieval or analyst-driven search.

## 5. Empirical behavior across recognition, prediction, and retrieval

Reported results are not directly interchangeable across supervision regimes, datasets, or evaluation protocols, but several consistent empirical patterns recur. Methods that enrich person features with scene context, learned relations, or explicit latent group codes almost always outperform direct pooling baselines. Detector-free and label-free feature learning can also be competitive when the representation is constrained by strong relational or self-supervised objectives [1812.07124, 2204.02139, 2403.02753, 2604.04467].

| Paradigm | Setting | Reported result |
|---|---|---|
| MLS-GAN [1812.07124] | Collective Activity / Volleyball | 91.7 / 91.2 MCA/MPCA on Collective Activity; 93.0 MCA and 92.4 MPCA on Volleyball |
| Detector-Free WSGAR [2204.02139] | NBA / Volleyball | 75.8% MCA, 71.2% MPCA on NBA; 90.5% MCA, 94.4% merged MCA on Volleyball |
| Dual-AI [2204.02148] | Volleyball | 94.4% group activity, 84.4% individual with RGB only; 95.4% with optical flow fusion |
| DynamicFormer [2305.05583] | Volleyball / Collective Activity | 95.3% group activity and 85.4% individual on Volleyball; 94.4% group activity on Collective Activity |
| KARI [2302.14350] | Volleyball / Collective Activity | 94.5% MCA on Volleyball; 92.8% MCA and 98.5% MPCA on Collective Activity |
| GAFL via person attributes [2403.02753] | Retrieval | GAFL-PAC group activity Hit@1 of 84.8 on VBD and 94.9 on CAD; GAFL-PAF group activity Hit@1 of 61.1 on VBD and 88.5 on CAD |
| Group-DINOmics [2604.04467] | Retrieval / supervised fine-tuning | VBD retrieval Hit@1 82.7 and Hit@3 93.0; supervised VBD 93.9% MCA and 96.1% Merged MCA |

Ablations provide some of the clearest evidence for what GAFL actually learns. In MLS-GAN on Collective Activity, direct supervised classification without GAN and without an action code reaches only 60.5 MPCA, whereas the conditional GAN variant reaches 87.7 MPCA and the full MLS-GAN reaches 91.2 MPCA, indicating that the intermediate action-code space and gated fusion substantially improve the learned group representation [1812.07124]. In detector-free weak supervision on NBA, a ResNet + global average pooling baseline gives 58.4% MCA and 51.7% MPCA, adding partial context embedding raises this to 64.1% and 58.5%, the designed temporal-and-context aggregation raises it to 73.6% MCA, and motion augmentation further raises it to 75.8% [2204.02139]. In attribute-prediction-based GAFL, removing location guidance drops group activity Hit@1 from 84.8 to 69.5 for GAFL-PAC on VBD and from 88.5 to 57.1 for GAFL-PAF on CAD, showing that location-guided decoding is not a minor detail but a major determinant of feature quality [2403.02753].

Retrieval-oriented GAFL exposes a different empirical aspect: the discriminative structure of the feature space itself. Group-DINOmics improves VBD retrieval Hit@1 from 43.0 for frozen DINOv3 without pretext losses to 75.4 with only flow loss, 74.0 with only object loss, and 82.7 with the full two-stage schedule; on NBA it reaches 43.9 Hit@1 and 72.0 Hit@3 [2604.04467]. Human-in-the-loop adaptation then further reshapes that space according to analyst intent: on Volleyball, GAFL baseline Precision@10 (original) is 0.533, while human-in-the-loop adaptation reaches 0.647 using 3 query videos and 5 selected videos to label, and on NBA it improves Precision@10 (original) from 0.206 to 0.233 [2602.03157]. This suggests that modern GAFL is increasingly evaluated not only by classification accuracy but also by the geometry and adaptability of the learned embedding.

## 6. Limitations, misconceptions, and emerging directions

Several papers explicitly motivate alternatives to plain person-feature pooling or fixed detector-centric pipelines. Region-feature summarization can be brittle because it depends on complete person localization and may ignore crucial context such as balls, nets, scoreboards, or referees; detector-free transformers, partial-context tokens, and group-query decoders were introduced precisely to address that limitation [2204.02139, 2207.05254]. Likewise, several models argue that group activity features should not be reduced to majority voting over person actions or to direct scene classification; instead they benefit from learned latent group codes, semantic relation graphs, knowledge-modulated attention, or attribute-prediction bottlenecks [1812.07124, 1908.02948, 2302.14350, 2403.02753].

At the same time, many limitations remain recurrent. A large portion of the literature still relies on accurate detections, bounding boxes, and tracklets; this dependence is explicit in hierarchical LSTMs, MLS-GAN, ARG-style models, attribute-prediction GAFL, and Group-DINOmics [1511.06040, 1812.07124, 2010.12968, 2403.02753, 2604.04467]. Sequence lengths are often short—10 frames in MLS-GAN and DynamicFormer, 9 or 17 frames in several transformer models, 10 or 12 frames in Group-DINOmics—so very long-horizon group dynamics are usually outside the modeled window [1812.07124, 2305.05583, 2604.04467]. Some methods note a fixed maximum number of persons, token diversity problems, sensitivity to rare group sizes, or biases introduced by ordered member-point targets [1812.07124, 2204.02139, 2207.05254]. Knowledge-augmented approaches acknowledge that their C–C and C–P maps are static global statistics rather than context-dependent knowledge [2302.14350]. Self-supervised DINO-based GAFL reports strong results on ball-centric sports but has not yet demonstrated the same breadth on arbitrary surveillance or social scenes [2604.04467].

The future directions proposed in the literature are correspondingly varied. MLS-GAN suggests graph neural networks instead of simple concatenation for gating inputs, longer sequences, hierarchical temporal scales, and end-to-end backbone learning with the GAN objective [1812.07124]. Detector-free weak supervision suggests token-diversity regularization and hybrid detector-free plus detector-based models [2204.02139]. Social group transformers point toward permutation-invariant member-set prediction, dynamic query allocation, and multimodal inputs [2207.05254]. DynamicFormer suggests richer human–object interaction modeling, unsupervised or self-supervised composition learning, and broader object vocabularies [2305.05583]. Knowledge-augmented relation inference points to richer knowledge sources, learnable knowledge maps, and context-dependent priors [2302.14350]. Group-DINOmics suggests extending group-relevant objects beyond the ball or goal and replacing simple temporal pooling with more powerful temporal aggregation [2604.04467]. Human-in-the-loop GAFL suggests multi-round active adaptation and more efficient online fine-tuning of the embedding space [2602.03157].

Taken together, these directions indicate that GAFL has become a unifying perspective on group activity understanding: the field increasingly treats the central problem not merely as assigning a label to a clip, but as constructing a structured latent space in which multi-person dynamics, context, interaction, and analyst intent can all be represented, compared, and adapted.

Source: https://www.emergentmind.com/topics/group-activity-feature-learning-gafl