RefAVA++: Benchmark for Referring Video Actions

• The paper introduces RefAVA++ as an expanded benchmark that scales data and complexity from RefAVA, facilitating robust generalization tests for language-guided action recognition.
• RefAVA++ integrates video clips and natural-language expressions to jointly localize target persons and recognize atomic actions using a strict movie-level split.
• Baseline comparisons show that dedicated models like RefAtomNet++ outperform generic AAL, VQA, and VTR methods through multi-trajectory semantic retrieval and multi-hierarchical cross-attention.

RefAVA++ is the expanded benchmark for Referring Atomic Video Action Recognition (RAVAR) introduced in "RefAtomNet++: Advancing Referring Atomic Video Action Recognition using Semantic Retrieval based Multi-Trajectory Mamba" (Peng et al., 18 Oct 2025). In RAVAR, a model receives a video and a natural-language referring expression and must jointly localize the referred person and recognize that person’s atomic actions. RefAVA++ extends the earlier RefAVA benchmark rather than redefining the task: it is a larger and more challenging version of the same RAVAR benchmark, built to support stronger generalization testing and to expose the limitations of earlier language-guided action models in complex multi-person scenes (Peng et al., 2024).

1. Task definition and conceptual position

RefAVA++ is organized around RAVAR, a reference-driven formulation of person-centric video understanding. The input is video + referring sentence; the output is target person bounding box + atomic action labels. This differs from conventional atomic action localization (AAL), where actions are typically predicted for all people in a scene and a target person is selected only afterward. In RefAVA++, the referring text is the primary mechanism for identifying the person of interest, so grounding and action recognition are inseparable components of the benchmark (Peng et al., 18 Oct 2025).

The benchmark was created in response to two limitations identified by the authors. First, the original RefAVA was considered too limited in scale for robust training and evaluation of fine-grained, language-guided action understanding. Second, the earlier RefAtomNet model, although strong on RefAVA, still had limited semantic visual retrieval capability, especially for precise localization and fine-grained action prediction in complex multi-person scenes. RefAVA++ therefore serves both as a dataset expansion and as a harder testbed for models that must align language, space, and temporal action evidence under cluttered visual conditions.

A common misconception is to treat RefAVA++ as a new task. The benchmark does not alter the underlying problem definition. The paper explicitly states that RefAVA++ is not a different task; it is an enlarged successor to RefAVA for the same RAVAR setting (Peng et al., 18 Oct 2025).

2. Corpus scale, provenance, and split protocol

RefAVA++ is built from 33,495 video clips spanning 231 movies and contains 2,950,830 annotated frames and 75,111 annotated persons / instances (Peng et al., 18 Oct 2025). The paper also states that the dataset was created by doubling the total number of annotated persons relative to RefAVA. In concrete terms, RefAVA had 1,615,140 frames and 36,630 annotated persons, whereas RefAVA++ adds 38,481 newly annotated instances and brings the total to 75,111.

The split protocol is intentionally strict at the movie level, which is meant to enforce generalization to unseen content rather than near-duplicate scene statistics. That choice is central to the benchmark’s role as an evaluation substrate for cross-movie transfer in language-guided action understanding.

Split	Instances	Clips / Movies
Train	48,151	21,709 / 150
Validation	16,142	6,513 / 50
Test	10,818	4,566 / 31

The predecessor RefAVA, introduced in "Referring Atomic Video Action Recognition" (Peng et al., 2024), used 17,946 video clips from 127 movies and contained 36,630 instances. RefAVA++ preserves the same benchmark identity while increasing both clip coverage and annotation density. This suggests that the benchmark was designed not merely as a scale-up in sample count, but as a more demanding environment for studying generalization beyond the distributional footprint of the original corpus.

3. Annotation methodology and label semantics

RefAVA++ uses a controlled annotation process based on key-frame bounding boxes from AVA. The paper reports that 8 expert annotators were recruited and that annotation quality was enforced through cross-checking and consensus: discrepant cases were discussed, only agreed-upon instances were kept, and unresolved ambiguous samples were discarded (Peng et al., 18 Oct 2025).

The referring expressions are constrained to describe salient visual attributes of the target person rather than the target action. The attributes include:

• approximate age
• gender
• clothing style
• physical appearance
• relative position in the frame

The annotations exclude action descriptions on purpose. This design prevents trivial label leakage from text to action output and forces the model to infer actions from video evidence rather than from lexical cues embedded in the referring sentence. For a benchmark centered on joint grounding and recognition, this is a consequential design decision: text specifies who is under consideration, but not what they are doing.

The action ontology consists of 80 atomic action categories, grouped into:

• OM: Object Manipulation
• PI: Person Interactions
• PM: Person Movement

These labels preserve the AVA-style atomic granularity while embedding them in a language-conditioned selection problem. In that sense, RefAVA++ occupies an intersection of vision-language grounding, person localization, and multi-label action recognition.

4. Benchmark design and empirical behavior

The benchmark evaluates 15 baselines spanning five groups, plus the earlier RefAtomNet model (Peng et al., 18 Oct 2025). The baseline families are:

• AAL baselines: I3D, X3D, MViTv2-B, VideoMAE2-B, Hiera-B
• VQA baselines: Singularity, AskAnything13B
• VTR baselines: MeVTR, CLIP4CLIP, XCLIP, BLIPv2
• Single-frame baselines: CLIP + SAM, CLIP + DETR, CLIP + REFCLIP
• VOS baseline: Su et al.
• Prior RAVAR model: RefAtomNet

All baselines are adapted to the RAVAR setting by adding a multi-label action classification head and a bounding-box regression head. The reported metrics are:

• mIOU for referred-person localization
• mAP for action recognition
• AUROC for action recognition

The benchmark findings show a clear structural pattern. Generic AAL methods perform worst on localization, because they do not provide strong vision-language grounding. VQA and VTR methods perform better because they are text-aware, but they still fall short on precise person grounding and fine-grained atomic action prediction. RefAtomNet and especially RefAtomNet++ perform best because they explicitly model referring semantics for the joint localization-action problem.

On RefAVA++, the paper reports the following headline results for RefAtomNet++:

• Validation: 39.12 mIOU / 58.24 mAP / 72.58 AUROC
• Test: 38.58 mIOU / 58.84 mAP / 73.28 AUROC

For RefAtomNet, the reported scores are:

• Validation: 35.97 / 56.54 / 70.97
• Test: 35.73 / 57.01 / 71.62

The paper highlights representative baseline trends. Among AAL systems, MViTv2-B obtains only 0.07/0.11 mIOU on validation/test, while X3D reaches 8.91/9.03 mIOU. Among VQA models, AskAnything13B reaches about 27.43/26.30 mIOU. Among VTR models, XCLIP attains 36.26/35.80 mIOU for localization, while BLIPv2 is strongest for action recognition with 54.58/54.75 mAP and 70.16/70.22 AUROC. Even in that strong VTR regime, RefAtomNet++ is better overall because it improves the localization-action joint problem rather than only one subcomponent.

5. Modeling context: from RefAtomNet to RefAtomNet++

RefAVA++ was introduced together with RefAtomNet++, which is built on BLIPv2 and extends the earlier RefAtomNet architecture from RefAVA (Peng et al., 18 Oct 2025). The earlier RefAtomNet, introduced with the original benchmark, used a three-stream design—visual, textual reference, and location-semantic streams—with cross-stream agent attention fusion and agent token fusion to suppress irrelevant visual content and localize the referred person more effectively in crowded scenes (Peng et al., 2024).

RefAtomNet++ addresses the limitations of that earlier architecture through two additions:

1. Multi-trajectory semantic-retrieval Mamba
2. Multi-hierarchical semantic-aligned cross-attention (MHS-CA)

The model operates over three semantic granularities:

• holistic-sentence level
• partial-keyword level
• scene-attribute level

At the partial-keyword level, stop words are removed, the remaining keywords are encoded, and for each keyword at each time step the nearest visual token is selected, forming a keyword-specific trajectory across time. At the scene-attribute level, DETR on the key frame provides object detections, from which bounding box coordinates and object category semantics are fused into scene-attribute tokens; these tokens also retrieve nearest visual tokens over time. At the holistic-sentence level, full-sentence visual tokens are refined directly. In all three cases, the retrieved trajectories are aggregated with Mamba, which the paper argues is beneficial because it captures long-range temporal dependencies, is memory-efficient, supports smooth, continuous trajectory modeling, and is better suited to action dynamics than discrete pairwise attention.

After retrieval and aggregation, MHS-CA performs cross-attention separately for holistic sentence retrieval, keyword-level retrieval, and scene-attribute retrieval, using learnable query prompts and separate temporal and spatial branches. Final bounding-box and action predictions are averaged across branches. The design is explicitly intended to improve cross-modal token aggregation and to align retrieval behavior with different semantic hierarchies.

The ablations reported with RefAtomNet++ strongly support this design. Removing keyword retrieval reduces validation mIOU from 43.71 to 36.93. Removing scene-attribute retrieval reduces validation mIOU to 38.97. Removing holistic semantic alignment causes the largest drop, to 27.14 on validation. Removing multi-hierarchical cross-attention reduces test mIOU from 42.52 to 36.84. The paper also reports that Mamba outperforms LSTM, linear projection, and Transformer for trajectory aggregation, and that the best settings are 6 queries and 8 frames (Peng et al., 18 Oct 2025).

6. Scope, limitations, and research significance

RefAVA++ is significant because it consolidates a problem formulation that sits between several established areas but is not reducible to any of them. It is not simply AAL with an extra text field, nor VTR with a detection head, nor VQA with multi-label outputs. The benchmark requires models to solve referring expression understanding, person grounding, and multi-label atomic action recognition jointly, under movie-level generalization constraints (Peng et al., 18 Oct 2025).

The benchmark also clarifies why strong results in adjacent domains do not automatically transfer. Vision-only AAL methods fail primarily on subject selection. Text-aware VQA and VTR models improve alignment but remain insufficient for precise grounding of the target person together with fine-grained action prediction. This suggests that RAVAR demands explicit reasoning over the semantics of the referring expression at multiple granularities rather than coarse cross-modal alignment alone.

Several boundaries of the benchmark are also explicit. The referring text intentionally omits action labels; accordingly, performance depends on whether a model can use language for target identification without conflating that with action supervision. The benchmark’s strict movie-level split increases difficulty by emphasizing unseen content. A plausible implication is that RefAVA++ is particularly suitable for studying whether vision-LLMs truly localize and track referred entities over time, rather than exploiting narrow lexical or scene priors.

Within the trajectory from RefAVA to RefAVA++, the expanded benchmark marks a shift from proof-of-concept dataset construction to a more demanding evaluation regime. RefAVA introduced the task and established the first benchmark for RAVAR (Peng et al., 2024). RefAVA++ scales that setting to 2.95 million frames and 75,111 annotated persons, broadens the movie coverage, and pairs the dataset with a model explicitly designed for semantic retrieval, trajectory aggregation, and multi-hierarchical cross-attention (Peng et al., 18 Oct 2025). As a result, RefAVA++ functions both as a dataset and as a benchmark definition for language-guided, person-specific atomic action understanding in video.