ProGraD: Prompt-driven Group Activity Detection

• The paper presents a novel framework, ProGraD, that jointly discovers social groups and classifies their activities using learnable group prompts with a DINOv2 backbone.
• It employs a lightweight two-layer GroupContext Transformer that first infers actor-to-group relations and then enriches these with scene-level context for precise group detection.
• Empirical results on Café and Social-CAD benchmarks demonstrate significant gains in group mAP and outlier detection, validating its effectiveness in multi-group scenarios.

Searching arXiv for ProGraD and closely related group activity detection work to ground the article in the cited literature. Prompt-driven Group Activity Detection (ProGraD) is a group activity detection (GAD) framework that adapts a pretrained vision foundation model to the structured problem of jointly discovering social groups and classifying their collective behaviors in video. In the formulation used by ProGraD, the system is not given a single predefined group; it must infer which people belong together, where those groups are in the scene, and what each group is doing collectively. The method instantiates “prompt-driven” adaptation through learnable group prompts inserted into a DINOv2 backbone and a lightweight two-layer GroupContext Transformer (GCT) that reasons over actor tokens, group tokens, and scene context, rather than through natural-language prompting (Ponbagavathi et al., 11 Aug 2025).

1. Task definition and conceptual scope

Group activity detection is distinct from standard group activity recognition. In GAR, the output is typically a single group activity label for a clip, often under assumptions such as one salient group per clip or manually specified actors. In GAD, the output is a set of groups, each with a predicted membership over people and a group activity label, with outliers or singletons explicitly relevant to evaluation. This practical formulation is sharpened by “Towards More Practical Group Activity Detection: A New Benchmark and Model” (Kim et al., 2023), which defines GAD as simultaneously identifying the members of each group and classifying the activity of the group, and by ProGraD, which adopts the same joint membership-and-activity perspective (Ponbagavathi et al., 11 Aug 2025).

A recurring misconception is that all “prompt-driven” methods in this area are language-based. ProGraD is not a text-prompt or vision-language method. Its prompts are learnable group prompts inserted into a visual transformer to steer attention toward socially meaningful regions and configurations. The paper explicitly distinguishes this from generic prompt tuning and from text prompting, describing one prompt per potential group and using those prompts to bias a frozen or optionally fine-tuned DINOv2 backbone toward group-aware structure (Ponbagavathi et al., 11 Aug 2025).

Another misconception is to conflate prompt-driven GAD with promptive GAR. PromptGAR, for example, is a prompt-driven GAR model that treats bounding boxes, skeletal keypoints, and areas as prompts and supports missing or sparse prompt inputs without retraining, but its primary task remains group activity recognition rather than exhaustive group detection. Its area prompt behaves as a target-group selector in multi-group scenes, not a detector head that discovers all groups and localizes them exhaustively (Jin et al., 11 Mar 2025). This distinction is central to understanding ProGraD: it addresses group discovery and group labeling jointly, not merely prompt-conditioned recognition of a predesignated group.

2. Research lineage and methodological precursors

ProGraD emerged after a sequence of increasingly structured approaches to collective behavior understanding. Earlier graph-relational GAR models such as “Improved Actor Relation Graph based Group Activity Recognition” represent actors as nodes, use pairwise appearance and position relations, and apply a graph convolutional network over an actor relation graph. That line of work is actor-centric and relational, but closed-set, non-prompt, and dependent on provided person boxes; it predicts group activity and auxiliary individual actions without open-vocabulary or prompt-conditioned inference (Kuang et al., 2020).

A second strand replaced RGB-centric pipelines with human-structure-centric inputs. “Pose is all you need: The pose only group activity recognition system (POGARS)” shows that competitive group activity recognition can be achieved from tracked 2D human pose sequences and person position tracklets using late person fusion, temporal 1D CNNs, temporal attention, spatial attention, and multi-task supervision. This establishes that group behavior can be recognized from structured person-time signals without RGB appearance, but it remains a closed-set, clip-level GAR method with no explicit relation graph and no prompting mechanism (Thilakarathne et al., 2021).

A third precursor is detector-free latent query modeling. “Detector-Free Weakly Supervised Group Activity Recognition” uses a set of $K$ learnable tokens to localize and encode partial contexts of a group activity through transformer attention, then aggregates those token-specific embeddings over time and across tokens. Although it is not a prompt-learning paper, its learnable tokens function as generic latent queries over the scene and constitute a direct conceptual precursor to prompt-driven query or slot formulations (Kim et al., 2022).

The immediate GAD substrate for ProGraD is the practical GAD formulation and benchmark introduced in (Kim et al., 2023). That work moves beyond single-group sports-centric GAR assumptions by introducing Café, a multi-person, multi-group benchmark with group memberships, outliers, and group-level metrics such as Group mAP based on Group IoU. It also introduces an end-to-end Transformer-based baseline with learnable group tokens, grouping attention, and actor–group matching, eliminating off-the-shelf clustering in favor of DETR-like set prediction. ProGraD inherits this practical GAD orientation but substitutes a vision foundation model and prompt-guided adaptation for the earlier ResNet-18-based backbone and heavier task-specific design (Kim et al., 2023, Ponbagavathi et al., 11 Aug 2025).

3. Architectural composition of ProGraD

ProGraD uses DINOv2-Base (ViT-B/14) as the visual backbone. For each frame $X_t$, the model obtains patch-level image tokens

$I_t \in \mathbb{R}^{N \times D},$

and, using person bounding boxes, extracts actor-specific features

$$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$

A notable design choice is $1\times1$ RoI pooling from frozen VFM feature maps, which produces a single compact token per actor instead of flattening a $5\times5$ grid. In the parameter-efficient setting, DINOv2 remains frozen and only the prompts, the GCT, and the prediction heads are trained, yielding about $10.68$M trainable parameters; a fully fine-tuned variant raises this to $97.28$M (Ponbagavathi et al., 11 Aug 2025).

The prompt mechanism consists of learnable group prompts

$$G = \{g_1, g_2, \dots, g_K\} \in \mathbb{R}^{K \times D},$$

with the number of prompts set to the maximum number of annotated groups in the dataset. On Café, the best setting is $K=7$. The paper evaluates no prompts, shallow prompts, and deep prompts, where deep prompting inserts prompts not only at the input layer but also through intermediate transformer layers. The ablation shows that deep prompting is essential: no prompts gives $X_t$0, shallow prompts gives $X_t$1, and deep prompts gives $X_t$2 for Group [email protected] / Group [email protected] / Outlier mIoU on Café (Ponbagavathi et al., 11 Aug 2025).

The GroupContext Transformer has exactly two layers. The first is the Grouping Attention Layer: $X_t$3 Here, group tokens query actor tokens, so each group token becomes a latent representation of one potential social group. The second is the Contextual Attention Layer. After concatenating actor and grouped tokens,

$X_t$4

the model updates them by attending to scene patch tokens: $X_t$5 The first layer infers actor-group composition; the second injects scene-level context such as spatial arrangement, shared objects, and broader interaction cues. Prediction heads then produce individual action classes, group activity classes, and actor-group affinity scores. The membership head projects actor and group tokens into a shared embedding space and computes dot-product affinities; actors are assigned to the group with the highest score, and Hungarian matching is used during training for membership prediction (Ponbagavathi et al., 11 Aug 2025).

The paper describes a multi-task objective combining individual action classification loss, group activity classification loss, membership prediction loss, and a contrastive group consistency loss. It reports $X_t$6, $X_t$7, and $X_t$8. The printed total-loss equation is described as corrupted in the supplied summary, but the component losses and their roles are stated clearly enough to place ProGraD within the broader family of end-to-end token-based GAD models (Ponbagavathi et al., 11 Aug 2025).

4. Benchmarks, evaluation, and empirical profile

ProGraD is evaluated on Café and Social-CAD. Café is a multi-camera, multi-group benchmark with 6-second clips from six café locations, $X_t$9–$I_t \in \mathbb{R}^{N \times D},$0 actors per clip, about $I_t \in \mathbb{R}^{N \times D},$1M human bounding boxes, group memberships, and six group activities, with ungrouped individuals treated as outliers or singleton labels. The cited sources report the six classes slightly differently: one source lists queueing, ordering, eating, working, fighting, taking selfie, while another lists queuing, ordering, selfie, eating, studying, fighting (Kim et al., 2023, Ponbagavathi et al., 11 Aug 2025). Social-CAD is more singleton-heavy and sparse, with social accuracy and membership accuracy used for evaluation (Ponbagavathi et al., 11 Aug 2025).

On Café, ProGraD is compared against ARG, Joint, JRDB-base, HGC, and the Practical GAD baseline. The strongest baseline in the ProGraD table is Practical GAD at $I_t \in \mathbb{R}^{N \times D},$2 for Group [email protected] / Group [email protected] / Outlier mIoU / Individual Accuracy. ProGraD frozen achieves $I_t \in \mathbb{R}^{N \times D},$3, and ProGraD full fine-tuning achieves $I_t \in \mathbb{R}^{N \times D},$4. The paper highlights that ProGraD frozen beats Practical GAD by $I_t \in \mathbb{R}^{N \times D},$5 on Group [email protected], $I_t \in \mathbb{R}^{N \times D},$6 on Group [email protected], $I_t \in \mathbb{R}^{N \times D},$7 on Outlier mIoU, and $I_t \in \mathbb{R}^{N \times D},$8 on individual accuracy while using $I_t \in \mathbb{R}^{N \times D},$9M trainable parameters versus $$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$0M (Ponbagavathi et al., 11 Aug 2025).

One of the central empirical arguments is that stronger generic features alone do not solve GAD. When Practical GAD’s ResNet-18 backbone is naively replaced with DINOv2, performance on Café drops from Group [email protected] $$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$1 and Group [email protected] $$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$2 to $$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$3 and $$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$4. ProGraD’s gains are therefore attributed not to backbone substitution alone but to prompt-guided, group-aware adaptation layered on top of the VFM (Ponbagavathi et al., 11 Aug 2025).

On Social-CAD, the gains are smaller but still state-of-the-art. Joint reports $$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$5 social accuracy and $$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$6 membership accuracy; Practical GAD reports $$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$7 social accuracy; ProGraD frozen reaches $$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$8 social accuracy and $$A = \{a_1, a_2, \dots, a_M\} \in \mathbb{R}^{M \times D}.$$9 membership accuracy; ProGraD full fine-tuning reaches $1\times1$0 and $1\times1$1. This pattern is consistent with the paper’s claim that ProGraD is especially effective in complex multi-group scenarios rather than sparse or singleton-heavy settings (Ponbagavathi et al., 11 Aug 2025).

Ablations isolate the roles of the GCT and the prompts. Removing the Grouping Attention Layer drops Café performance to $1\times1$2, while removing the Contextual Attention Layer drops it further to $1\times1$3. The full model returns $1\times1$4. Prompt count also matters: $1\times1$5 is best on Café, outperforming $1\times1$6, $1\times1$7, and $1\times1$8. The paper interprets this as a balance between coverage of potential groups and redundancy or overfitting (Ponbagavathi et al., 11 Aug 2025).

5. Prompting paradigms across adjacent work

ProGraD occupies one point in a broader design space of prompt-conditioned collective behavior models. Its prompts are learnable group prompts embedded in a vision transformer. PromptGAR, by contrast, reformulates boxes, keypoints, and areas as point prompts with attributes $1\times1$9, pools them into prompt tokens, and uses a two-way recognition decoder inspired by Segment Anything. PromptGAR’s contribution is input flexibility across prompts, frames, and instances without retraining, and its area prompts provide selective recognition of a target group in multi-group scenes. Yet its task is still standard GAR: it predicts a class label for the group activity in the clip rather than a set of detected groups with memberships (Jin et al., 11 Mar 2025).

LIR-GAD extends prompt-conditioned reasoning toward multimodal language instruction. It augments an MLLM with one $5\times5$0 token and $5\times5$1 $5\times5$2 tokens, embeds box coordinates into textual question instructions, and aligns MLLM hidden states with actor and group features through a Multimodal Dual-Alignment Fusion module. Its overall loss includes

$5\times5$3

with $5\times5$4, $5\times5$5, $5\times5$6, and $5\times5$7. On Café split-by-view with $5\times5$8, it reports $5\times5$9, improving strongly on Group mAP$10.68$0 and Outlier mIoU relative to GADFormer, though not on Group mAP$10.68$1 (Peng et al., 19 Sep 2025).

These variants clarify that “prompt-driven” is not a single technique. In the current literature it includes at least three forms. First, learnable visual group prompts, as in ProGraD, which bias VFM attention toward social configurations (Ponbagavathi et al., 11 Aug 2025). Second, structured geometric prompts such as boxes, keypoints, and areas, as in PromptGAR (Jin et al., 11 Mar 2025). Third, instruction-mediated special tokens coupled to an MLLM, as in LIR-GAD (Peng et al., 19 Sep 2025). A plausible implication is that prompt-driven GAD is best understood as a family of conditioning mechanisms over group hypotheses rather than as a synonym for natural-language prompting alone.

6. Interpretability, limitations, and open technical issues

ProGraD emphasizes interpretability through attention analysis. Compared with Practical GAD, its attention maps are described as sharper, more structured, better separated across groups, and better at isolating outliers. The paper also compares naive DINOv2 attention with prompt-guided DINOv2 attention prior to the GCT and reports that prompting shifts attention from broad object/background coverage toward people, groups, tables, and other interaction-relevant regions. The evidence is indirect but supported by both ablations and visualizations: deep prompts produce large quantitative gains, and prompt-guided backbone attention changes in socially meaningful ways (Ponbagavathi et al., 11 Aug 2025).

Several limitations remain explicit. ProGraD assumes access to ground-truth actor detections; robustness under noisy or automated detections is not explored. Even in the frozen setting, inference still requires a full pass through a large VFM backbone, so parameter-efficient adaptation does not imply minimal inference cost. The paper also reports failure cases in occluded scenes and mirror-heavy environments, where reflections or ambiguous visibility can attract attention and disrupt group assignment. Class-wise analysis on Café shows weaker performance on rare classes such as Queueing and Ordering, which the paper attributes to class imbalance and suggests might require reweighting or few-shot prompt adaptation (Ponbagavathi et al., 11 Aug 2025).

The broader literature highlights additional constraints. Practical GAD benchmarks formalize outlier detection and group-level evaluation but still operate on clip-level inputs with person boxes or tracklets supplied to the model (Kim et al., 2023). PromptGAR demonstrates prompt, frame, and instance flexibility, yet remains a recognizer rather than a full detector (Jin et al., 11 Mar 2025). LIR-GAD introduces language-instructed reasoning, but its prompt template is under-specified in the supplied summary and its prompting is largely fixed-template rather than open-ended (Peng et al., 19 Sep 2025). These observations suggest that prompt-driven GAD remains a developing area in which three unresolved questions persist: how to combine prompt flexibility with full group discovery, how to reduce dependence on oracle detections, and how to extend prompt-conditioned grouping beyond closed-set supervised taxonomies.

Within that landscape, ProGraD’s technical identity is precise: DINOv2-Base features are adapted by deep learnable group prompts; actor tokens are extracted by $10.68$2 RoI pooling; a two-layer GroupContext Transformer first performs actor-to-group formation and then scene-context enrichment; and group memberships, group activities, and individual actions are predicted jointly. Its core empirical claim is equally precise: a vision foundation model becomes substantially more effective for GAD only when prompt-guided, group-aware relational reasoning is added on top, especially in multi-group scenes such as Café (Ponbagavathi et al., 11 Aug 2025).