---
title: 'RefAVA++: Benchmark for Referring Video Actions'
url: https://www.emergentmind.com/topics/refava
type: topic
---

# RefAVA++: Benchmark for Referring Video Actions

RefAVA++ is the expanded benchmark for **Referring Atomic Video Action Recognition (RAVAR)** introduced in "RefAtomNet++: Advancing Referring Atomic Video Action Recognition using Semantic Retrieval based Multi-Trajectory Mamba" [2510.16444]. In RAVAR, a model receives a **video** and a **natural-language referring expression** and must jointly **localize the referred person** and **recognize that person’s atomic actions**. RefAVA++ extends the earlier RefAVA benchmark rather than redefining the task: it is a **larger and more challenging version of the same RAVAR benchmark**, built to support stronger generalization testing and to expose the limitations of earlier language-guided action models in complex multi-person scenes [2407.01872].

## 1. Task definition and conceptual position

RefAVA++ is organized around RAVAR, a reference-driven formulation of person-centric video understanding. The input is **video + referring sentence**; the output is **target person bounding box + atomic action labels**. This differs from conventional **atomic action localization (AAL)**, where actions are typically predicted for all people in a scene and a target person is selected only afterward. In RefAVA++, the referring text is the primary mechanism for identifying the person of interest, so grounding and action recognition are inseparable components of the benchmark [2510.16444].

The benchmark was created in response to two limitations identified by the authors. First, the original RefAVA was considered too limited in scale for robust training and evaluation of fine-grained, language-guided action understanding. Second, the earlier RefAtomNet model, although strong on RefAVA, still had **limited semantic visual retrieval capability**, especially for precise localization and fine-grained action prediction in complex multi-person scenes. RefAVA++ therefore serves both as a dataset expansion and as a harder testbed for models that must align language, space, and temporal action evidence under cluttered visual conditions.

A common misconception is to treat RefAVA++ as a new task. The benchmark does not alter the underlying problem definition. The paper explicitly states that RefAVA++ is **not a different task**; it is an enlarged successor to RefAVA for the same RAVAR setting [2510.16444].

## 2. Corpus scale, provenance, and split protocol

RefAVA++ is built from **33,495 video clips** spanning **231 movies** and contains **2,950,830 annotated frames** and **75,111 annotated persons / instances** [2510.16444]. The paper also states that the dataset was created by **doubling the total number of annotated persons** relative to RefAVA. In concrete terms, RefAVA had **1,615,140 frames** and **36,630 annotated persons**, whereas RefAVA++ adds **38,481 newly annotated instances** and brings the total to **75,111**.

The split protocol is intentionally strict at the movie level, which is meant to enforce generalization to unseen content rather than near-duplicate scene statistics. That choice is central to the benchmark’s role as an evaluation substrate for cross-movie transfer in language-guided action understanding.

| Split | Instances | Clips / Movies |
|---|---:|---:|
| Train | 48,151 | 21,709 / 150 |
| Validation | 16,142 | 6,513 / 50 |
| Test | 10,818 | 4,566 / 31 |

The predecessor RefAVA, introduced in "Referring Atomic Video Action Recognition" [2407.01872], used **17,946 video clips** from **127 movies** and contained **36,630 instances**. RefAVA++ preserves the same benchmark identity while increasing both clip coverage and annotation density. This suggests that the benchmark was designed not merely as a scale-up in sample count, but as a more demanding environment for studying generalization beyond the distributional footprint of the original corpus.

## 3. Annotation methodology and label semantics

RefAVA++ uses a controlled annotation process based on **key-frame bounding boxes from AVA**. The paper reports that **8 expert annotators** were recruited and that annotation quality was enforced through **cross-checking and consensus**: discrepant cases were discussed, only agreed-upon instances were kept, and unresolved ambiguous samples were discarded [2510.16444].

The referring expressions are constrained to describe **salient visual attributes** of the target person rather than the target action. The attributes include:

- **approximate age**
- **gender**
- **clothing style**
- **physical appearance**
- **relative position in the frame**

The annotations **exclude action descriptions** on purpose. This design prevents trivial label leakage from text to action output and forces the model to infer actions from video evidence rather than from lexical cues embedded in the referring sentence. For a benchmark centered on joint grounding and recognition, this is a consequential design decision: text specifies *who* is under consideration, but not *what* they are doing.

The action ontology consists of **80 atomic action categories**, grouped into:

- **OM**: Object Manipulation
- **PI**: Person Interactions
- **PM**: Person Movement

These labels preserve the AVA-style atomic granularity while embedding them in a language-conditioned selection problem. In that sense, RefAVA++ occupies an intersection of vision-language grounding, person localization, and multi-label action recognition.

## 4. Benchmark design and empirical behavior

The benchmark evaluates **15 baselines** spanning five groups, plus the earlier **RefAtomNet** model [2510.16444]. The baseline families are:

- **AAL baselines**: I3D, X3D, MViTv2-B, VideoMAE2-B, Hiera-B
- **VQA baselines**: Singularity, AskAnything13B
- **VTR baselines**: MeVTR, CLIP4CLIP, XCLIP, BLIPv2
- **Single-frame baselines**: CLIP + SAM, CLIP + DETR, CLIP + REFCLIP
- **VOS baseline**: Su et al.
- **Prior RAVAR model**: RefAtomNet

All baselines are adapted to the RAVAR setting by adding a **multi-label action classification head** and a **bounding-box regression head**. The reported metrics are:

- **mIOU** for referred-person localization
- **mAP** for action recognition
- **AUROC** for action recognition

The benchmark findings show a clear structural pattern. **Generic AAL methods perform worst on localization**, because they do not provide strong vision-language grounding. **VQA** and **VTR** methods perform better because they are text-aware, but they still fall short on precise person grounding and fine-grained atomic action prediction. **RefAtomNet** and especially **RefAtomNet++** perform best because they explicitly model referring semantics for the joint localization-action problem.

On **RefAVA++**, the paper reports the following headline results for **RefAtomNet++**:

- **Validation**: **39.12 mIOU / 58.24 mAP / 72.58 AUROC**
- **Test**: **38.58 mIOU / 58.84 mAP / 73.28 AUROC**

For **RefAtomNet**, the reported scores are:

- **Validation**: **35.97 / 56.54 / 70.97**
- **Test**: **35.73 / 57.01 / 71.62**

The paper highlights representative baseline trends. Among AAL systems, **MViTv2-B** obtains only **0.07/0.11 mIOU** on validation/test, while **X3D** reaches **8.91/9.03 mIOU**. Among VQA models, **AskAnything13B** reaches about **27.43/26.30 mIOU**. Among VTR models, **XCLIP** attains **36.26/35.80 mIOU** for localization, while **BLIPv2** is strongest for action recognition with **54.58/54.75 mAP** and **70.16/70.22 AUROC**. Even in that strong VTR regime, RefAtomNet++ is better overall because it improves the **localization-action joint problem** rather than only one subcomponent.

## 5. Modeling context: from RefAtomNet to RefAtomNet++

RefAVA++ was introduced together with **RefAtomNet++**, which is built on **BLIPv2** and extends the earlier **RefAtomNet** architecture from RefAVA [2510.16444]. The earlier RefAtomNet, introduced with the original benchmark, used a **three-stream** design—visual, textual reference, and location-semantic streams—with **cross-stream agent attention fusion** and **agent token fusion** to suppress irrelevant visual content and localize the referred person more effectively in crowded scenes [2407.01872].

RefAtomNet++ addresses the limitations of that earlier architecture through two additions:

1. **Multi-trajectory semantic-retrieval Mamba**
2. **Multi-hierarchical semantic-aligned cross-attention (MHS-CA)**

The model operates over three semantic granularities:

- **holistic-sentence level**
- **partial-keyword level**
- **scene-attribute level**

At the **partial-keyword level**, stop words are removed, the remaining keywords are encoded, and for each keyword at each time step the nearest visual token is selected, forming a keyword-specific trajectory across time. At the **scene-attribute level**, **DETR** on the key frame provides object detections, from which **bounding box coordinates** and **object category semantics** are fused into scene-attribute tokens; these tokens also retrieve nearest visual tokens over time. At the **holistic-sentence level**, full-sentence visual tokens are refined directly. In all three cases, the retrieved trajectories are aggregated with **Mamba**, which the paper argues is beneficial because it captures **long-range temporal dependencies**, is **memory-efficient**, supports **smooth, continuous trajectory modeling**, and is better suited to action dynamics than discrete pairwise attention.

After retrieval and aggregation, **MHS-CA** performs cross-attention separately for holistic sentence retrieval, keyword-level retrieval, and scene-attribute retrieval, using **learnable query prompts** and separate **temporal** and **spatial** branches. Final bounding-box and action predictions are averaged across branches. The design is explicitly intended to improve cross-modal token aggregation and to align retrieval behavior with different semantic hierarchies.

The ablations reported with RefAtomNet++ strongly support this design. Removing **keyword retrieval** reduces validation mIOU from **43.71 to 36.93**. Removing **scene-attribute retrieval** reduces validation mIOU to **38.97**. Removing **holistic semantic alignment** causes the largest drop, to **27.14** on validation. Removing **multi-hierarchical cross-attention** reduces test mIOU from **42.52 to 36.84**. The paper also reports that **Mamba outperforms LSTM, linear projection, and Transformer for trajectory aggregation**, and that the best settings are **6** queries and **8** frames [2510.16444].

## 6. Scope, limitations, and research significance

RefAVA++ is significant because it consolidates a problem formulation that sits between several established areas but is not reducible to any of them. It is not simply AAL with an extra text field, nor VTR with a detection head, nor VQA with multi-label outputs. The benchmark requires models to solve **referring expression understanding**, **person grounding**, and **multi-label atomic action recognition** jointly, under movie-level generalization constraints [2510.16444].

The benchmark also clarifies why strong results in adjacent domains do not automatically transfer. Vision-only AAL methods fail primarily on subject selection. Text-aware VQA and VTR models improve alignment but remain insufficient for precise grounding of the target person together with fine-grained action prediction. This suggests that RAVAR demands explicit reasoning over the semantics of the referring expression at multiple granularities rather than coarse cross-modal alignment alone.

Several boundaries of the benchmark are also explicit. The referring text intentionally omits action labels; accordingly, performance depends on whether a model can use language for **target identification** without conflating that with **action supervision**. The benchmark’s strict movie-level split increases difficulty by emphasizing unseen content. A plausible implication is that RefAVA++ is particularly suitable for studying whether vision-language models truly localize and track referred entities over time, rather than exploiting narrow lexical or scene priors.

Within the trajectory from RefAVA to RefAVA++, the expanded benchmark marks a shift from proof-of-concept dataset construction to a more demanding evaluation regime. RefAVA introduced the task and established the first benchmark for RAVAR [2407.01872]. RefAVA++ scales that setting to **2.95 million frames** and **75,111 annotated persons**, broadens the movie coverage, and pairs the dataset with a model explicitly designed for **semantic retrieval**, **trajectory aggregation**, and **multi-hierarchical cross-attention** [2510.16444]. As a result, RefAVA++ functions both as a dataset and as a benchmark definition for language-guided, person-specific atomic action understanding in video.

Source: https://www.emergentmind.com/topics/refava