CafGa: Dual Roles in Vision & Language

• CafGa is a research term denoting distinct methodologies in video-based group activity detection and interactive language model explanation.
• In computer vision, CafGa introduces a benchmark with strict group-centric metrics and a Transformer-based model for realistic multi-group detection.
• In explainable AI, CafGa offers an interactive tool that enables customizable text segmentation and perturbation-based assessments for feature attribution.

Searching arXiv for papers on "CafGa" to ground the article and disambiguate the term. CafGa is an overloaded research term used for two distinct artifacts in recent arXiv literature. In computer vision, it denotes a benchmark and accompanying model for practical group activity detection (GAD), introduced alongside the Caf$\bar{\text{e}}$ dataset and strict group-centric evaluation metrics (Kim et al., 2023). In explainable AI for LLMs, it denotes an interactive system for generating and evaluating feature attribution explanations at customizable granularities, centered on user-defined text segmentation and perturbation-based fidelity assessment (Boyle et al., 25 Sep 2025). The shared label does not indicate a common methodology or application domain; rather, the two uses belong to separate research lineages in video understanding and LLM interpretability, respectively.

1. Term usage and disambiguation

The computer-vision usage arises in work on group activity detection, where the benchmark and model are described as addressing “more practical GAD scenarios” through a new dataset and a model that handles “an unknown number of groups and latent group members efficiently and effectively” (Kim et al., 2023). The details identify this line of work as “CafGa benchmark” and describe it as a large, complex, and richly annotated benchmark tailored specifically for practical GAD scenarios.

The explainability usage appears in the paper titled “CafGa: Customizing Feature Attributions to Explain LLMs,” where CafGa is an “interactive tool for generating and evaluating feature attribution explanations at customizable granularities” (Boyle et al., 25 Sep 2025). Here the central problem is not video understanding but the definition of interpretable components for feature attribution in long-form text. The system is positioned against word-level explanations, which are described as computationally inefficient and semantically limited for multi-word phenomena.

A plausible implication is that references to “CafGa” require domain-specific contextualization. In vision, the term is tied to group-centric video benchmarks and end-to-end grouping models; in NLP/XAI, it refers to a human-in-the-loop attribution interface and evaluation workflow.

2. CafGa in group activity detection

In the GAD literature, CafGa is introduced to remedy practical limitations of prior datasets and methods. Existing datasets are described as focused on group activity recognition rather than full GAD, often assuming a single group per clip and requiring manual pre-identification of group members. They are further characterized as extensions of other datasets rather than being designed specifically for GAD, with many groups being singletons rather than true groups (Kim et al., 2023).

The CafGa benchmark is presented as being constructed “primarily for GAD” and as covering “more practical scenarios and metrics,” while also being “large-scale and providing rich annotations” (Kim et al., 2023). Its data acquisition procedure includes filming at six cafes, with up to four different cameras per cafe, and participants performing one of six predefined group activities: Queueing, Ordering, Eating, Working, Fighting, and Taking Selfie. Outliers are explicitly included, and individuals not belonging to any group are often spatially close to groups for realistic complexity.

The annotation protocol combines manual and automated stages. Human annotators select a key frame per 6-second clip, assign actor bounding boxes, group memberships, and group activity labels, after which multi-object tracking using ByteTrack and a YOLOX detector extends actor boxes across frames to obtain tracklets. Manual correction is then applied to tracking errors, identities, and bounding boxes. The provided annotations include actor bounding boxes, track IDs, group configuration, group activity labels, and explicit marking of outliers or non-group members (Kim et al., 2023).

The dataset is described as containing 10,297 clips, over 4 hours of video, approximately 342,000 frames, and 3.5 million person boxes. Each clip contains multiple true non-singleton groups, with an average of 1.55 groups per clip; frames contain 3–14 actors, typically 10 or 11, and about 50% are marked as outliers. The six activity classes are imbalanced, with Queueing described as rare and Taking Selfie as most common (Kim et al., 2023).

3. Benchmark design, metrics, and evaluation philosophy

A defining contribution of the GAD-oriented CafGa is its evaluation protocol. The benchmark introduces “more stringent, group-centric metrics,” especially Group mAP and Outlier mIoU (Kim et al., 2023). Group localization is based on a set-valued IoU over actors: $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$ Correct localization requires $\text{Group IoU} \geq \theta$, with $\theta = 1.0$ or $0.5$ evaluated. AP is computed per activity class from predicted groups and their confidence scores, and mean AP is then taken across classes.

Outlier mIoU measures overlap between predicted and ground-truth outlier sets: $$\text{Outlier mIoU} = \frac{1}{|V|} \sum_{v\in V} \frac{|O_v \cap \hat{O}_v|}{|O_v \cup \hat{O}_v|}.$$ The paper explicitly contrasts these metrics with earlier actor-level measures such as Social Accuracy and Social mAP, arguing that earlier metrics tolerate partial or “soft” group matches, whereas Group mAP requires full group localization and is therefore more rigorous and practical (Kim et al., 2023).

This metric design is central to the benchmark’s research significance. Rather than treating group activity as a classification problem over pre-specified memberships, CafGa frames it as simultaneous group discovery and activity classification. This suggests a shift from loosely supervised social-scene recognition toward stricter structured prediction over actor sets.

4. Model architecture associated with the GAD benchmark

The accompanying model is described as an end-to-end Transformer-based architecture that uses learnable group tokens, analogous to DETR object queries, to represent candidate groups (Kim et al., 2023). Its design directly targets the two latent variables emphasized by the benchmark: the unknown number of groups and unknown group membership.

The model begins with a ResNet-18 backbone, pretrained on ImageNet, and uses RoIAlign to extract $5 \times 5$ actor-centric features from frames using bounding boxes. Positional embeddings encode box coordinates. A Grouping Transformer then processes actor features together with $K$ learnable group tokens, where $K$ is larger than the maximum number of groups per clip.

Three attention mechanisms are specified. Self-attention is applied separately to actors and groups to capture intra-actor and intra-group relations. Cross-attention allows actor and group features to attend to frame or global context. Grouping attention allows each group token to attend to actor features and aggregate them into a group representation. In addition, actor–actor attention is restricted by distance masking using a spatial threshold $\mu$, which the paper relates to social proximity (Kim et al., 2023).

Prediction proceeds through an activity classification head and a membership-scoring mechanism based on projected actor and group embeddings, whose dot product yields an affinity score interpreted as membership probability. Each actor is assigned to the group with highest membership affinity. Empty group slots are handled through bipartite matching rather than post hoc clustering.

Training uses Hungarian matching between predicted and ground-truth groups, with the latter padded by empty groups. The total loss combines individual action loss, group activity classification loss, group membership loss, and a group consistency loss: $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$0 The group consistency term is a modified InfoNCE objective over cosine similarities of embeddings from actors in the same ground-truth group, and optimization uses Adam with linear warmup, batch size $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$1, $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$2, $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$3, and $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$4 (Kim et al., 2023).

A notable methodological claim is that inference requires no clustering or post-processing. Empty groups are ignored, and final predictions are read directly from the learned group-token outputs. This is presented as the main reason for the method’s speed advantage over clustering-based baselines.

5. Empirical findings for the GAD benchmark and model

On the CafGa dataset using ground-truth tracklets and the split by view, the reported numbers for the proposed model with $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$5 are an inference time of $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$6 s, Group mAP of $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$7 at IoU $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$8, Group mAP of $$\text{Group IoU} (G, \hat{G}) = \frac{|G \cap \hat{G}|}{|G \cup \hat{G}|}.$$9 at IoU $\text{Group IoU} \geq \theta$0, and Outlier mIoU of $\text{Group IoU} \geq \theta$1 (Kim et al., 2023). Competing methods such as JRDB-base, Joint, and HGC are reported with lower Group mAP and Outlier mIoU, while clustering-based approaches are slower.

In the detection-based setting using predicted tracklets, the model remains best among the compared methods, with Group mAP at IoU $\text{Group IoU} \geq \theta$2 of $\text{Group IoU} \geq \theta$3 and Outlier mIoU of $\text{Group IoU} \geq \theta$4 (Kim et al., 2023). On Social-CAD, the model achieves Social Accuracy $\text{Group IoU} \geq \theta$5 with a ResNet-18 backbone and a single frame, compared with $\text{Group IoU} \geq \theta$6 for Joint using I3D over 17 frames. On JRDB-Act, the reported mAP is $\text{Group IoU} \geq \theta$7, with particularly strong performance on larger group sizes such as G4 and G5+ (Kim et al., 2023).

Ablation results summarized in the source indicate that all three attention types are essential, distance masking improves performance, and group consistency loss significantly improves group localization quality and convergence. Qualitative observations state that the model can handle crowded and ambiguous scenes while distinguishing groups from outliers. The speed advantage is attributed to the absence of expensive, non-differentiable clustering steps.

The broader significance claimed for this version of CafGa is that it “sets a new standard” for realistic multi-group, multi-person, multi-view, large-scale GAD benchmarks, while also “re-frames evaluation” toward strict group discovery rather than partial membership scoring (Kim et al., 2023). Interpreted conservatively, this marks a methodological preference for explicit structural correctness in group prediction.

6. CafGa in language-model explanation

A separate work uses the same name for an interactive system for explaining LLMs via customizable feature attributions (Boyle et al., 25 Sep 2025). The paper motivates the system by noting that methods such as SHAP and LIME typically treat individual words as atomic units. According to the paper, this is “highly computationally inefficient for long-form text” and “fails to capture semantic information that spans multiple words” (Boyle et al., 25 Sep 2025).

CafGa addresses this issue by allowing explanations to be computed over user-defined segments at arbitrary granularity. The system supports default presets at the word, sentence, or paragraph level, and it also permits interactive customization in which users select arbitrary text spans to form custom groups. Segments are non-overlapping, and interaction is described as brushing or selecting phrases in the user interface (Boyle et al., 25 Sep 2025).

Attribution is computed using KernelSHAP over the user-defined text groups. The perturbation process described in the paper generates random samples by removing selected groups, queries the model for each perturbed input, applies a user-defined evaluator to the responses, and then uses weighted linear regression to estimate group-level Shapley values. A time-budgeted sampling rule is given as

$\text{Group IoU} \geq \theta$8

This ties explanation cost to API rate and a maximum desired runtime (Boyle et al., 25 Sep 2025).

Visualization has two principal components. First, group-level attributions are shown as a heatmap over the text. Second, the system visualizes deletion and insertion curves as fidelity diagnostics. The $\text{Group IoU} \geq \theta$9-axis is defined as the percentage of words perturbed rather than the percentage of groups, specifically to avoid bias from group size. The $\theta = 1.0$0-axis is the difference between the full and perturbed model prediction: $\theta = 1.0$1 Large area under the deletion curve and small area under the insertion curve are described as indicators of more faithful explanations (Boyle et al., 25 Sep 2025).

The system is explicitly interactive rather than fully automatic. Its contribution is not a new axiomatic attribution rule but a user-centered workflow for defining the explanation units themselves and for checking whether those units align with model behavior.

7. User study, effectiveness, and conceptual relation between the two CafGa usages

The language-model CafGa is evaluated through a two-stage user study. The first stage involves 10 participants, including 6 with ML expertise and 4 novices, across five task types: SQuAD QA, Yelp sentiment, prompt engineering, HotpotQA multi-hop reasoning, and BARQA long-form comprehension. The second stage involves 4 experts who compare human-crafted CafGa explanations against those generated by PartitionSHAP and MExGen (Boyle et al., 25 Sep 2025).

Experts rate the system as easy to use with average $\theta = 1.0$2, easy to learn with $\theta = 1.0$3, and enjoyable with $\theta = 1.0$4. Non-experts rate ease of use at $\theta = 1.0$5 and learning at $\theta = 1.0$6, while both groups rate the system as helpful for understanding LLM decision logic; the values reported for “System is helpful for understanding LLM” are $\theta = 1.0$7 for non-experts and $\theta = 1.0$8 for experts (Boyle et al., 25 Sep 2025).

In comparative evaluation, participants prefer human-customized CafGa explanations $\theta = 1.0$9 of the time, compared with $0.5$0 for MExGen and $0.5$1 for PartitionSHAP. The source further reports that custom segmentation can improve fidelity scores, with one worked example moving from $0.5$2 at sentence level to $0.5$3 with custom phrase-level segmentation (Boyle et al., 25 Sep 2025). The paper interprets perturbation curves as a mechanism for guarding against confirmation bias by exposing when custom explanations do or do not match model behavior.

The two usages of “CafGa” share a high-level emphasis on practical evaluation rather than a shared technical substrate. The GAD benchmark emphasizes strict group-centric localization and multi-view realism (Kim et al., 2023), whereas the interpretability system emphasizes customizable explanation granularity and perturbation-based faithfulness checks (Boyle et al., 25 Sep 2025). This suggests that the common term indexes a concern with practical deployment conditions in two different fields: structured scene understanding in video and human-centered explanation of LLMs.