RAVAR: Referring Atomic Video Action Recognition

• RAVAR is a task that uses language to select a specific person in a video by focusing on appearance and spatial cues.
• It builds on AVA by adding non-action referring expressions to guide fine-grained, multi-label atomic action recognition in multi-person scenes.
• Specialized architectures like RefAtomNet fuse video, text, and location information to enhance precise person-specific action prediction.

Referring Atomic Video Action Recognition (RAVAR) is a person-centric video understanding task in which a model receives a video clip and a natural-language referring expression that identifies one specific individual, and must recognize that individual’s atomic actions while localizing the referred person in the frame (Peng et al., 2024). In the canonical formulation introduced with RefAVA, the text is used only to specify who the person is, not what action they perform: the descriptions cover approximate age, gender, appearance, and relative position in the center frame, and explicitly exclude action description, so action prediction remains a visual inference problem (Peng et al., 2024). RAVAR therefore sits at the intersection of referring expression grounding, atomic action recognition, and person-level spatial localization, while remaining distinct from standard AVA-style action localization, video question answering, and text–video retrieval (Peng et al., 2024).

1. Task definition and conceptual boundaries

RAVAR takes as input a video clip $\mathbf{x}^{VT}$, a textual referring expression $\mathbf{x}^{RT}$, and the center frame $\mathbf{x}_k$ used to derive scene-object cues, and predicts both a multi-label atomic action vector $\hat{\mathbf{y}} \in [0,1]^{N_c}$ and a bounding box $\hat{\mathbf{b}} \in \mathbb{R}^4$ for the referred person (Peng et al., 2024). In RefAVA, $N_c = 80$, following the AVA atomic action taxonomy, with labels grouped into Object Manipulation, Person Interactions, and Person Movement (Peng et al., 2024). The optimization objective is a weighted sum of binary cross-entropy for the action labels and mean squared error for the referred-person box: $$L_{BCE}(\mathbf{y}, \hat{\mathbf{y}}) = -\frac{1}{N_c} \sum_{i=1}^{N_c} \left[\mathbf{y}_i \log(\hat{\mathbf{y}}_i) + (1 - \mathbf{y}_i)\log(1 - \hat{\mathbf{y}}_i)\right]$$

$$L_{MSE}(\mathbf{b}, \hat{\mathbf{b}}) = \sum_{j=1}^{4} (\mathbf{b}_j - \hat{\mathbf{b}}_j)^2$$

with $L = L_{BCE} + \lambda L_{MSE}$ and $\lambda = 5$ in the reported experiments (Peng et al., 2024).

The task differs from standard atomic action localization in a specific and consequential way. AVA-style systems are given person boxes for all individuals and must classify actions for each of them; the person of interest is selected only afterward (Gu et al., 2017). RAVAR instead uses language to determine which person is relevant and asks for only that person’s atomic actions (Peng et al., 2024). It also differs from video question answering and video–text retrieval: those paradigms usually operate on global video semantics, coarse answers, or retrieval scores, whereas RAVAR requires person-specific grounding and fine-grained multi-label action prediction within a multi-person scene (Peng et al., 2024).

A common misconception is that RAVAR is simply “AVA with captions.” The benchmark construction contradicts that view. Referring expressions are written to uniquely identify one person in the scene while omitting action words, so the model cannot solve the task by reading off the answer from the text (Peng et al., 2024). Another misconception is that it is merely a special case of referring video object segmentation. Existing referring video models can help with localization, but RAVAR additionally requires AVA-style atomic action recognition, including simultaneous labels such as stand, talk to (a person), and carry/hold (an object) for the same person (Gu et al., 2017).

2. Dataset foundations: from AVA to RefAVA and RefAVA++

RAVAR inherits its action vocabulary and person-centric grounding structure from AVA, which densely annotates 80 atomic visual actions in 430 fifteen-minute movie clips, with exhaustive person bounding boxes and frequent multi-label annotations per person (Gu et al., 2017). AVA was designed around atomic rather than composite actions, and each person can simultaneously receive one pose label plus multiple person–object and person–person interaction labels (Gu et al., 2017). This makes AVA an appropriate substrate for RAVAR because the labels are already tied to individuals rather than to clips.

RefAVA adds referring expressions to that substrate. It was built from an AVA v2.2 subset of 17,946 video clips from 127 movies and contains 36,630 person instances, each paired with a manually written referring description and the person’s atomic action labels (Peng et al., 2024). Seven annotators wrote descriptions using age, gender, clothing, appearance, accessories, and relative position, and no two persons in the same clip share the same reference sentence (Peng et al., 2024). The train, validation, and test splits are movie-disjoint: 22,658 instances from 67 movies for training, 10,916 instances from 34 movies for validation, and 3,056 instances from 26 different movies for testing (Peng et al., 2024).

RefAVA++ extends this benchmark substantially. It is built from 33,495 clips from 231 movies, comprises 2,950,830 frames and 75,111 annotated persons, and preserves the principle that referring expressions describe appearance and relative position but intentionally exclude action descriptions (Peng et al., 18 Oct 2025). The split is again movie-level and disjoint: 48,151 training instances, 16,142 validation instances, and 10,818 test instances; 150 movies are used for training, 50 for validation, and 31 for testing (Peng et al., 18 Oct 2025).

Corpus	Scale	Distinguishing property
AVA	430 clips; 1.58M action labels	Exhaustive person boxes with 80 atomic actions (Gu et al., 2017)
RefAVA	36,630 instances	Referring expressions for one person per instance (Peng et al., 2024)
RefAVA++	2,950,830 frames; 75,111 persons	Larger RAVAR benchmark with movie-level split (Peng et al., 18 Oct 2025)

The progression from AVA to RefAVA and RefAVA++ changes the problem from “classify every visible person” to “ground one linguistically specified person and classify only that person.” This suggests that the benchmark lineage preserves the atomic-action rigor of AVA while introducing the referential ambiguity absent from conventional action localization.

3. Architectural lineage and baseline families

Before RAVAR was formalized, the “Video Action Transformer Network” already instantiated a visual-only precursor: person detections are used as queries, and for each person box the model attends over the entire spatiotemporal feature map to aggregate context relevant to that person’s atomic actions (Girdhar et al., 2018). In that architecture, a Region Proposal Network produces human boxes on the keyframe, each box yields a person-specific query embedding $\mathbf{x}^{RT}$0, and cross-attention over keys and values from the full video feature map computes

$\mathbf{x}^{RT}$1

to classify the queried person’s actions (Girdhar et al., 2018). The model was trained and evaluated on AVA v2.1, achieved 25.0% mAP on validation with a 96-frame test-time setting, and learned to emphasize hands, faces, other people, and objects without explicit supervision beyond boxes and class labels (Girdhar et al., 2018). In RAVAR terms, this is a query-conditioned atomic action recognizer where the query is a person box rather than language.

RefAVA evaluated fifteen adapted baselines spanning five methodological families: atomic action localization, video question answering, video–text retrieval, single-frame referring models, and video object segmentation (Peng et al., 2024). The adaptation pattern was consistent: add a BERT-based text branch or prompt, regress the referred-person box, and attach a multi-label action head (Peng et al., 2024). The empirical lesson was equally consistent: models strong at action recognition were weak at text-guided grounding, while models strong at multimodal grounding were not specialized for multi-label atomic actions (Peng et al., 2024).

On RefAVA, the strongest generic baseline was BLIP-2 among video–text retrieval models, reaching 32.99 mIOU, 52.13 mAP, and 66.56 AUROC on validation, and 32.75 mIOU, 53.19 mAP, and 69.92 AUROC on test (Peng et al., 2024). AskAnything, adapted from video question answering, performed better than AVA-style localization baselines at grounding the referred person, with 20.09 mIOU, 51.42 mAP, and 66.12 AUROC on validation, and 22.35 mIOU, 52.25 mAP, and 69.35 AUROC on test (Peng et al., 2024). By contrast, atomic action localization baselines such as I3D, X3D, MViTv2-B, VideoMAE2-B, and Hiera-B exhibited very low mIOU, indicating difficulty in resolving the language-conditioned identity of the target person (Peng et al., 2024).

On RefAVA++, the same pattern persisted. BLIPv2 yielded 54.75 mAP and 70.22 AUROC on test but only 23.40 mIOU, while XCLIP produced 35.80 mIOU on test but lower action-recognition metrics at 51.37 mAP and 67.15 AUROC (Peng et al., 18 Oct 2025). This baseline landscape makes clear that RAVAR is not exhausted by either retrieval-style alignment or person-localization alone. It requires a fused representation that resolves referential identity and atomic action semantics jointly.

4. RefAtomNet: the first specialized RAVAR architecture

RefAtomNet was introduced as a task-specific architecture for RAVAR, motivated by the observation that existing methods underperform because they either miss person-level grounding or fail to capture fine-grained multi-label actions (Peng et al., 2024). Its core design is a three-stream architecture composed of a video stream, a text stream, and a location-semantic stream (Peng et al., 2024). Visual tokens $\mathbf{x}^{RT}$2 and text tokens $\mathbf{x}^{RT}$3 are extracted by a BLIP-2-style multimodal encoder with ViT, BERT, and QFormer components: $\mathbf{x}^{RT}$4 and location-semantic tokens are formed by running DETR on the center frame, encoding detected object categories with BERT, concatenating them with box coordinates, and projecting them into token space: $\mathbf{x}^{RT}$5 (Peng et al., 2024).

The model’s distinguishing mechanism is agent-based attention and cross-stream fusion. For each stream $\mathbf{x}^{RT}$6, RefAtomNet computes query, key, value, and agent tokens: $\mathbf{x}^{RT}$7 then uses the text and location-semantic streams to refine visual attention via Cross-stream Agent Attention Fusion and Cross-stream Agent Token Fusion (Peng et al., 2024). The aim is to suppress irrelevant persons and clutter by using the referring expression and scene layout as “advisors” for the visual stream. The aggregated representation

$\mathbf{x}^{RT}$8

is finally fed to an MLP action head and an MLP bounding-box regressor (Peng et al., 2024).

With BLIP-2 as backbone, RefAtomNet improved the RefAVA benchmark over all adapted baselines. On validation it achieved 38.22 mIOU, 55.98 mAP, and 69.73 AUROC; on test it achieved 36.42 mIOU, 57.52 mAP, and 73.95 AUROC (Peng et al., 2024). Relative to BLIP-2, the gains were +5.23 mIOU, +3.85 mAP, and +3.17 AUROC on validation, and +3.67 mIOU, +4.33 mAP, and +4.03 AUROC on test (Peng et al., 2024). Additional detection-style metrics also improved, with mAP at IoU 0.2 rising from 45.21 to 49.47 on test and mAP at IoU 0.5 rising from 39.27 to 41.93 (Peng et al., 2024).

Ablations showed that the location-semantic stream helps especially with localization, and that both Cross-stream Agent Attention Fusion and Cross-stream Agent Token Fusion contribute to the final performance (Peng et al., 2024). Removing the location-semantic stream reduced test performance to 31.25 mIOU, 55.73 mAP, and 72.38 AUROC, while removing Cross-stream Agent Attention Fusion reduced it to 35.38 mIOU, 55.90 mAP, and 72.60 AUROC (Peng et al., 2024). The full model remained robust under test-time rephrasing and under perturbations such as rain, fog, shot noise, and Gaussian noise (Peng et al., 2024). These results established the first strong evidence that RAVAR demands specialized cross-stream grounding rather than generic multimodal fine-tuning.

5. RefAtomNet++ and the move toward semantic retrieval

RefAtomNet++ advances RAVAR by scaling the benchmark to RefAVA++ and replacing RefAtomNet’s single-level cross-stream emphasis with multi-hierarchical semantic retrieval and Mamba-based trajectory modeling (Peng et al., 18 Oct 2025). The model again begins from BLIPv2 with QFormer: $\mathbf{x}^{RT}$9 but then constructs three semantic hierarchies: holistic-sentence, partial-keyword, and scene-attribute (Peng et al., 18 Oct 2025).

At the partial-keyword level, stop words are removed from the referring expression, keyword embeddings are retained, and for each keyword $\mathbf{x}_k$0 and timestep $\mathbf{x}_k$1 the model retrieves the nearest spatial visual token: $\mathbf{x}_k$2 forming a trajectory

$\mathbf{x}_k$3

which is then aggregated by a Mamba state-space model: $\mathbf{x}_k$4 (Peng et al., 18 Oct 2025). At the scene-attribute level, DETR detections on the key frame are encoded as object-category-plus-box tokens, aligned to visual tokens over time, and aggregated in the same manner (Peng et al., 18 Oct 2025). At the holistic level, the visual tokens themselves are refined through a Mamba block: $\mathbf{x}_k$5 (Peng et al., 18 Oct 2025).

These three semantic streams are then fused by Multi-Hierarchical Semantic-Aligned Cross-Attention in separate temporal and spatial branches. The temporal branch averages over semantic levels to produce $\mathbf{x}_k$6, the spatial branch analogously produces $\mathbf{x}_k$7, and separate heads output temporal and spatial box/action predictions: $\mathbf{x}_k$8

$\mathbf{x}_k$9

and the final outputs are the averages of temporal and spatial predictions: $\hat{\mathbf{y}} \in [0,1]^{N_c}$0 (Peng et al., 18 Oct 2025).

On RefAVA, RefAtomNet++ reached 43.71 mIOU, 56.83 mAP, and 71.27 AUROC on validation, and 42.52 mIOU, 59.81 mAP, and 75.72 AUROC on test (Peng et al., 18 Oct 2025). Relative to RefAtomNet, the test improvements were +6.10 mIOU, +2.29 mAP, and +1.77 AUROC (Peng et al., 18 Oct 2025). On RefAVA++, it achieved 39.12 mIOU, 58.24 mAP, and 72.58 AUROC on validation, and 38.58 mIOU, 58.84 mAP, and 73.28 AUROC on test, again establishing the best reported results (Peng et al., 18 Oct 2025). Notably, it reduced trainable parameters from 214M in RefAtomNet to 121M (Peng et al., 18 Oct 2025).

The ablation results isolate the main sources of gain. Removing holistic-sentence-level temporal modeling caused the largest drop, reducing test mIOU on RefAVA from 42.52 to 29.00 (Peng et al., 18 Oct 2025). Removing keyword-level retrieval reduced test mIOU to 36.93, and removing scene-attribute-level retrieval reduced it to 38.60 (Peng et al., 18 Oct 2025). Replacing Mamba with LSTM, a linear layer, or a Transformer degraded performance; on RefAVA test, LSTM yielded 37.05 mIOU and 57.32 mAP, Transformer yielded 39.06 mIOU and 58.35 mAP, while Mamba achieved 42.52 mIOU and 59.81 mAP (Peng et al., 18 Oct 2025). The model also remained strong under test-time paraphrasing, reaching 39.67 mIOU, 58.82 mAP, and 75.26 AUROC on rephrased RefAVA test queries (Peng et al., 18 Oct 2025).

6. Broader research context, methodological links, and open problems

RAVAR is part of a broader shift toward atomic, query-conditioned video understanding. One research thread comes from person-centric action localization. The Action Transformer showed that high-resolution person queries attending over global spatiotemporal context can spontaneously learn to track individuals and attend to hands, faces, and interacted objects, all from box and class supervision alone (Girdhar et al., 2018). This suggests that the query-based structure fundamental to RAVAR was already latent in AVA-style models, but without language.

A second thread comes from text-guided fine-grained action representation. “Storyboard guided Alignment for Fine-grained Video Action Recognition” treats action recognition as video–text matching, uses an LLM to generate atomic sub-texts such as “Prepare diving,” “Jump off the board,” “In mid-air,” and “Dive into the water,” and defines fine-grained frame-importance scores based on these descriptions (Liu et al., 2024). The method’s fine-grained weighting

$\hat{\mathbf{y}} \in [0,1]^{N_c}$1

highlights the frames where an atomic action is at its peak, and the paper explicitly notes that such atomic descriptions can function as referring expressions for atomic behaviors (Liu et al., 2024). This suggests a route from class-level text decomposition to open-form RAVAR queries.

A third thread concerns action-centric latent decomposition. In ROAD++ Track 3, an action-slot model coupled with X3D and SlowFast backbones was used for 64-class multi-label atomic activity recognition in traffic scenes, with slot attention intended to capture different concurrent activity patterns within a clip (Cao et al., 2024). The full ensemble reached 0.58 mAP on the TACO test set, improving the challenge baseline by 4% absolute (Cao et al., 2024). A plausible implication is that slot-based latent factors could serve as the internal action entities that a referring expression selects in a future RAVAR system.

A fourth thread concerns pose- and skeleton-centric atomic representations. HAA500 provides 500 fine-grained human-centric atomic action classes with 591K labeled frames and high pose visibility, reporting an average of 69.7% detectable joints (Chung et al., 2020). HAA4D goes further by introducing 4D skeletons for 300 atomic action classes and using global alignment so that skeletons face the negative $\hat{\mathbf{y}} \in [0,1]^{N_c}$2-direction, which improved 5-way few-shot performance over implicit 3D embeddings on both HAA4D and NTU-RGB+D (Tseng et al., 2022). “Semi-Supervised Few-Shot Atomic Action Recognition” adds loose temporal alignment, attention-based aggregation, and multi-head relation comparison for short atomic actions, using a CTC-style loss

$\hat{\mathbf{y}} \in [0,1]^{N_c}$3

to handle temporal variability (Ni et al., 2020). Together these results suggest that RAVAR need not remain exclusively RGB-and-language based: skeleton-aligned or slot-based formulations may be especially useful when language refers primarily to pose trajectories, motion phases, or concurrent action factors.

The open problems are already visible in the first two RAVAR benchmarks. RefAVA assumes one unique referring person per sentence and does not yet address multi-agent references such as “the two men in blue shirts on the right” (Peng et al., 2024). Both RefAVA and RefAVA++ remain movie-centric and operate on short windows rather than long-form temporal reasoning (Peng et al., 2024, Peng et al., 18 Oct 2025). RefAtomNet++ still depends on detector quality and remains challenged by strong occlusion, crowded scenes, and very subtle actions (Peng et al., 18 Oct 2025). More generally, current RAVAR benchmarks predict the referred person’s box in the key frame rather than explicit temporal boundaries for each atomic action. This suggests that the next stage of the field may combine person grounding, temporal grounding, multi-person reference resolution, and compositional atomic-action querying within a single formulation.

RAVAR therefore names more than a benchmark. It identifies a precise research problem: language-guided, person-specific, multi-label atomic action understanding in realistic multi-person video. Its development has proceeded from AVA’s atomic action ontology (Gu et al., 2017), through person-query architectures such as the Action Transformer (Girdhar et al., 2018), to dedicated multimodal benchmarks and models such as RefAVA, RefAtomNet, RefAVA++, and RefAtomNet++ (Peng et al., 2024, Peng et al., 18 Oct 2025). The resulting field is defined by one central requirement: a model must determine which person language refers to, and then infer which atomic actions that person is performing, without being told the answer by either pre-given person boxes or action-bearing text.