Zero-Shot Compositional Action Recognition

• Zero-Shot Compositional Action Recognition is a video task that recombines observed verbs and objects to recognize unseen actions through factorized dynamic and static learning.
• It leverages semantic embeddings, independent primitive learning, and logic-driven constraints to build robust representations for novel verb-object pairs.
• Benchmarks like Sth-com and metrics such as HM and AUC emphasize both advances and challenges in mitigating shortcut learning and domain shifts.

Zero-Shot Compositional Action Recognition (ZS-CAR) is a video understanding task demanding the recognition of unseen compositional actions made from previously observed verbs and objects, or, more generally, unseen combinations of known action primitives at test time (Li et al., 2024). In contrast to conventional zero-shot action recognition, which often treats an action label as a monolithic unseen class, ZS-CAR places compositional generalization at the center of the problem: the model must recombine learned knowledge about dynamic and static components, rather than merely transfer to a new atomic label. Earlier zero-shot action recognition work already exposed relevant ingredients—semantic word-vector embeddings for multi-word action labels, dynamic temporal signatures, and open-set verb-noun recognition in egocentric video—but the task itself was formalized explicitly only later (Xu et al., 2015, Kim et al., 2019, Scott et al., 2020).

1. Task definition and formal structure

The canonical ZS-CAR formulation assumes that test actions are novel compositions, while their constituent primitives have been observed during training in other combinations. In the formulation introduced with C2C, compositional actions consist of dynamic (verbs) and static (objects) concepts, and the task requires a model to recognize unseen actions composed of previously observed verbs and objects (Li et al., 2024). The new Sth-com benchmark operationalizes this regime by ensuring all test set verbs and objects have been seen in the training set, but their compositions may be novel.

This factorization is consistent with earlier egocentric work, where action classes are verb-noun pairs such as take cup or open door, and the central question is whether models can recognize actions constructed from unseen combinations of verbs and nouns—some or all of which may never have been seen during training (Scott et al., 2020). That paper reframed zero-shot generalization as cross-modal few-shot generalization, with semantic representations of action labels acting as the support set. In that view, zero-shot action recognition becomes a special case of few-shot learning with $k=1$, where the single support exemplar is a semantic prototype rather than a video.

The primitive-composition distinction also clarifies what counts as genuine compositionality. In ZS-CAR, the action label is not merely a string; it denotes a structured combination whose components can vary independently. This motivates models that separately encode dynamic and static evidence, or that explicitly represent conditional compatibility between primitives. In C2C, for example, an action is written as $a_i = (v_l, o_k)$, and composition inference estimates action scores through both a dynamics path and a static path, then averages them (Li et al., 2024). In LogicCAR, the same structure is formalized with first-order predicates, such as $\forall x ( c(x) \implies v(x) )$ and $\forall x ( c(x) \implies o(x) )$, making the composition itself a constrained relation between primitive labels (Ye et al., 4 Aug 2025).

2. Historical emergence from zero-shot action recognition

The earliest video-specific precursor is the semantic embedding framework of Xu, Hospedales, and Gong, which mapped videos and class labels into a semantic word vector space rather than an attribute space (Xu et al., 2015). Class labels were embedded by a pretrained word2vec model through

$g: y \mapsto z,$

and for multi-word labels the vectors were averaged: $z = \frac{1}{N} \sum_{j=1}^N g(y_j).$ For labels such as “brush hair” or “ride horse,” this provided a simple additive model of action compositionality, although the paper did not specifically address or evaluate more sophisticated compositional or relational modeling of action semantics.

A different precursor appeared in DASZL, which modeled activities as compositions of dynamic action signatures and reframed fine-grained recognition as zero-shot activity recognition (Kim et al., 2019). Instead of treating an action as a static semantic point, DASZL represented an action through temporal evolution of attributes and used finite state machines to compose detectors “on the fly” from dynamic temporal patterns and learned attribute or object detectors. This line of work is notable because it made temporal structure itself a compositional substrate.

Open-set egocentric action recognition provided another bridge. The EPIC-KITCHENS study argued that fixed-set classification is often unrealistic due to the compositionality of actions, resulting in a functionally infinite-cardinality label set (Scott et al., 2020). It introduced new splits with no overlap in verb-noun pairs between train, validation, and test, and subdivided evaluation into Held-out Verb (HoV), Held-out Noun (HoN), and All Test. This shifted the discussion from unseen labels in general to unseen compositions specifically.

The explicit formalization of ZS-CAR as a distinct task arrived with C2C in 2024, which presented what it described as the first true ZS-CAR benchmark and protocol for video (Li et al., 2024). Subsequent work then concentrated on the deficiencies of existing compositional learners and on stronger inductive biases: logic-driven structural constraints in LogicCAR (Ye et al., 4 Aug 2025), shortcut mitigation in RCORE (Ahn et al., 22 Jan 2026), and enriched semantic prompting or prompt-based alignment in CLIP-style frameworks that reported particular gains on fine-grained and compositional actions (Iqbal et al., 9 Mar 2026, Wang et al., 18 Apr 2026).

3. Principal modeling paradigms

ZS-CAR methods can be organized around how they represent primitives, compositions, and cross-modal transfer.

Paradigm	Core mechanism	Representative papers
Semantic embedding	Map videos and class labels into a shared semantic space	(Xu et al., 2015, Scott et al., 2020)
Factorized component learning	Learn verbs and objects independently, then infer compositions through compatibility	(Li et al., 2024)
Logic- or structure-driven learning	Embed explicit compositional and hierarchical constraints into the objective	(Ye et al., 4 Aug 2025, Ahn et al., 22 Jan 2026)
Prompt- and VLM-based alignment	Align video representations with enriched textual prompts, sometimes with positive and negative prompts	(Iqbal et al., 9 Mar 2026, Wang et al., 18 Apr 2026)

In semantic embedding approaches, class names or descriptions define prototypes in a shared space, and a visual encoder or regressor maps videos toward those prototypes. The 2015 semantic embedding model learned a nonlinear SVR with RBF-$\chi^2$ kernel to project Bag-of-Words dense trajectory features into the word2vec space, then classified by nearest prototype: $\hat{y}^* = \arg\min_{y^*} \| f(x^*) - g(y^*) \|.$ Its self-training step updated unseen class prototypes transductively, and its auxiliary-dataset augmentation pooled disjoint action classes to improve generalization of the mapping (Xu et al., 2015). In egocentric recognition, word embeddings of verb-noun labels similarly acted as prototypes, but the paper emphasized deep metric learning and joint embedding over direct alignment (Scott et al., 2020).

Factorized component learning is more specific to ZS-CAR. C2C separates dynamic and static evidence through a dynamic encoder for verb features $\mathbf{f}_v$ and a static encoder for object features $\mathbf{f}_o$, aligns them to learnable verb and object prototypes, and infers composition scores through conditional paths such as object-given-verb and verb-given-object (Li et al., 2024). This is coupled with an enhanced training strategy designed to address component domain variation and component compatibility variation. The same paper also introduced HSIC-based independence losses, CutMix-based imagination of unseen compositions, and consensus inference from both dynamic and static paths.

Logic-driven methods make the compositional structure explicit rather than implicit. LogicCAR argued that previous models suffer from missing compositional structure constraint and neglecting semantic hierarchy constraint, then encoded both as first-order logic relaxed by fuzzy logic (Ye et al., 4 Aug 2025). The explicit compositional logic enforces entailment from composition to primitives and exclusivity among verb and object classes; the hierarchical primitive logic adds fine-to-coarse dependencies such as $a_i = (v_l, o_k)$0 and analogous rules for objects. The total objective is

$a_i = (v_l, o_k)$1

combining classification with explicit compositional logic and hierarchical primitive logic.

Prompt- and VLM-based methods emphasize richer semantics on the text side. SP-CLIP constructs multiple textual descriptions per action class—covering intent, motion, object interaction, context, and related facets—from the Stories dataset, encodes them with BERT or RoBERTa, and averages them: $a_i = (v_l, o_k)$2 It then performs contrastive alignment and zero-shot consistency scoring against unseen classes (Iqbal et al., 9 Mar 2026). A related CLIP-based framework introduced a Motion Separation Module and Motion Aggregation Block, then aligned video embeddings with both positive and negative prompts, explicitly modeling “non-class” semantics to enlarge inter-class margins (Wang et al., 18 Apr 2026). Both works claim particular relevance to compositional or fine-grained action distinctions.

4. Benchmarks, splits, and evaluation protocols

Evaluation protocols are central to ZS-CAR because compositional generalization is highly sensitive to leakage, bias, and the definition of “unseen.” Earlier zero-shot action recognition studies on HMDB51 and UCF101 typically used 50/50 partitioning of categories into seen and unseen classes, often repeated over 30 random splits (Xu et al., 2015). These protocols measure transfer to disjoint labels, but they do not by themselves ensure that the unseen classes are novel compositions of seen primitives.

The egocentric EPIC-KITCHENS splits addressed this more directly by enforcing no overlap in action classes between train, validation, and test, and by reporting Held-out Verb and Held-out Noun conditions (Scott et al., 2020). C2C then defined the stricter Sth-com benchmark specifically for ZS-CAR, while later work constructed EK100-com to analyze shortcut behavior under even lower compositional coverage (Li et al., 2024, Ahn et al., 22 Jan 2026).

Benchmark or protocol	Key property	Reported details
HMDB51 / UCF101 zero-shot splits	Seen/unseen class partitions for classical ZSAR	HMDB51: 51 action classes, 6766 videos; UCF101: 101 action classes, 13320 videos; 50/50 partitioning with 30 random splits (Xu et al., 2015)
EPIC-KITCHENS open-set splits	No overlap in verb-noun pairs; HoV, HoN, All Test	Split 1 test: 248 HoV, 249 HoN, 536 All; Split 2 test: 257 HoV, 247 HoN, 542 All; Split 3 test: 280 HoV, 238 HoN, 543 All (Scott et al., 2020)
Sth-com	Explicit ZS-CAR benchmark with seen primitives and novel compositions	5,124 verb-object compositions, 79465 videos, 161 verbs, 248 objects; unseen actions comprise nearly half of evaluation (Li et al., 2024)

Evaluation metrics also differ. In Sth-com experiments, C2C reports Harmonic Mean (HM) and AUC, following generalized compositional zero-shot evaluation (Li et al., 2024). LogicCAR reports the same metrics to show the effect of explicit logic constraints (Ye et al., 4 Aug 2025). RCORE supplements accuracy with False Seen Prediction (FSP), False Co-occurrence Prediction (FCP), and the compositional gap

$a_i = (v_l, o_k)$3

which measures whether joint composition accuracy exceeds the product of independent verb and object accuracy (Ahn et al., 22 Jan 2026). This metric is specifically diagnostic for whether a model is genuinely composing rather than independently predicting primitives.

A parallel concern is overlap between pretraining and evaluation labels. The Semantic Video Transformer paper described a “Fair ZSL” setup that avoids overlap between training and testing classes by removing semantically overlapping categories relative to Kinetics pretraining (Doshi et al., 2022). Although that work targets zero-shot action recognition rather than strict ZS-CAR, the protocol issue is directly relevant: compositional claims are difficult to interpret if test compositions or near-equivalent semantics appear in pretraining corpora.

5. Empirical findings, gains, and failure modes

On the Sth-com benchmark, C2C reported the best Harmonic Mean with VideoSwin-T backbone at 36.7 for Enhanced C2C versus 30.2 for OADis, and the best AUC at 17.4 versus 13.2 for vanilla C2C (Li et al., 2024). LogicCAR later reported HM = 45.2 and AUC = 27.0, exceeding C2C by 0.4 (HM) and 1.1 (AUC) percentage points on Sth-com (Ye et al., 4 Aug 2025). These numbers frame a substantial part of the current ZS-CAR trajectory: factorized modeling yields strong gains over adapted CZSL baselines, and structured symbolic constraints push performance further.

On broader zero-shot action recognition benchmarks, semantic prompting and prompt-alignment methods reported high absolute accuracy while emphasizing compositional or fine-grained benefits. SP-CLIP reported 53.9 ± 2.3 on HMDB-51 and 80.4 ± 3.5 on UCF-101 with CLIP, and described gains as especially notable for compositional or fine-grained actions (Iqbal et al., 9 Mar 2026). The motion-guided alignment model reported 55.2% on HMDB-51 and 82.2% on UCF-101, and also reported base-to-novel transfer results of HM 68.5% on Kinetics-400 and HM 68.3% on HMDB-51 (Wang et al., 18 Apr 2026). These results belong to the broader ZSAR landscape, but they are increasingly discussed as evidence that cross-modal prompt design and motion-aware alignment can strengthen compositional discrimination.

Method	Benchmark	Reported result
Enhanced C2C	Sth-com	HM 36.7, AUC 17.4 (Li et al., 2024)
LogicCAR	Sth-com	HM 45.2, AUC 27.0 (Ye et al., 4 Aug 2025)
RCORE	Sth-com	Unseen composition accuracy 32.23% vs 30.08% baseline; compositional gap +0.40 vs −0.42 baseline (Ahn et al., 22 Jan 2026)

A major recent development is that stronger accuracy alone is no longer considered sufficient evidence of compositional reasoning. RCORE argued that existing ZS-CAR models fail primarily due to object-driven verb shortcuts (Ahn et al., 22 Jan 2026). According to its analysis, the root causes are severe sparsity and skewness of compositional supervision and the asymmetric learning difficulty between verbs and objects. Objects are learned faster because they can be recognized from spatial cues, while verbs require temporal reasoning; as training progresses, the existing ZS-CAR model increasingly ignores visual evidence and overfits to co-occurrence statistics. On Sth-com and EK100-com, RCORE reported that C2C had negative compositional gaps—−0.42 on Sth-com unseen pairs and −1.22 on EK100-com unseen pairs—whereas RCORE achieved +0.40 on Sth-com and reduced the EK100-com gap to −0.29 (Ahn et al., 22 Jan 2026).

RCORE addresses this failure mode with two mechanisms. VOCAMix synthesizes new verb-object combinations by mixing motion from one sample with static object content from another while preserving temporal cues, and the Temporal Order Regularization Loss combines

$a_i = (v_l, o_k)$4

with

$a_i = (v_l, o_k)$5

to penalize shortcut behavior on reversed or shuffled temporal sequences (Ahn et al., 22 Jan 2026). This line of work shifts the field from composition as a label-space operation to composition as a robustness problem in video learning.

An older but related observation is domain shift. The 2015 semantic embedding paper already identified that a regressor trained on source classes and applied to disjoint unseen classes suffers from domain shift, and showed that self-training and auxiliary data augmentation substantially improved zero-shot performance on HMDB51 and UCF101 (Xu et al., 2015). This suggests that compositional generalization in video is simultaneously a problem of semantic structure and of transfer across shifted visual-semantic alignments.

6. Relation to broader compositional zero-shot learning and open directions

ZS-CAR sits within the broader field of compositional zero-shot learning (CZSL), but video introduces temporality, motion-context entanglement, and open-world feasibility constraints that are less severe in image benchmarks. The 2025 CZSL survey organized methods by disentanglement into four families—no explicit disentanglement, textual disentanglement, visual disentanglement, and cross-modal disentanglement—and argued that, for ZS-CAR, visual and cross-modal disentanglement are especially important because action components are visually and temporally entangled (Munir et al., 13 Oct 2025). The same survey identified key open challenges: modeling primitives and contextuality, scaling to the open world, generalization to unseen primitives, leveraging large multimodal models, the data annotation bottleneck, and temporal compositionality.

Several non-video CZSL ideas are explicitly described as relevant to ZS-CAR. CAPE learns a rich dependency structure between compositions through self-attention and is described as directly applicable to ZS-CAR by replacing state-object vectors with action-context or actor representations (Khan et al., 2022). TMN emphasized that generalized evaluation is essential and that naive baselines may appear competitive if only unseen pairs are evaluated; its task-driven modular gating offers a reusable mechanism for recombining subtasks, which is directly relevant for long-tailed compositional action spaces (Purushwalkam et al., 2019). A causal view of compositional zero-shot recognition further proposed treating recognition as identifying which intervention caused the image, with disentangled representations of component factors and HSIC-based independence regularization to reduce spurious correlations (Atzmon et al., 2020). In video, LogicCAR and RCORE can be read as concrete descendants of that broader anti-correlation agenda.

Dataset design remains an unresolved axis. HOMAGE provides 75 high-level activities, 453 atomic actions, multi-view and multi-modal recordings, and dense scene composition labels, and is explicitly presented as enabling future zero-shot compositional action recognition even though its main experiments focus on few-shot generalization (Rai et al., 2021). Earlier training-free work on zero-shot action recognition from object-scene compositions also showed that compositional transfer can arise without any action-labeled video training by taking the Cartesian product of objects and scenes and semantically matching those compositions to action labels (Bretti et al., 2021). This suggests that ZS-CAR may not have a single canonical formulation: verb-object composition, object-scene composition, atomic-to-activity composition, and prompt-level semantic composition all coexist under the same general requirement of recognizing unseen combinations.

Current evidence therefore points in a convergent direction. Explicit factorization of motion and context, explicit constraints on which compositions are plausible, better control of shortcut learning, and richer textual supervision all improve compositional transfer in video (Li et al., 2024, Ye et al., 4 Aug 2025, Ahn et al., 22 Jan 2026, Iqbal et al., 9 Mar 2026). A plausible implication is that future ZS-CAR systems will combine the strongest elements of these lines: disentangled motion-static representations, structured semantic prompting, logic or hierarchy constraints, and evaluation protocols that isolate genuinely unseen compositions without semantic leakage.