RefAtomNet++: Advanced RAVAR Vision-Language Model

• RefAtomNet++ is a vision-language framework addressing RAVAR by jointly localizing a target person and predicting fine-grained atomic actions in video clips.
• The model employs a multi-trajectory Mamba architecture and semantic-aligned cross-attention to improve cross-modal retrieval and disambiguation in multi-person scenes.
• Empirical results on RefAVA and RefAVA++ benchmarks demonstrate state-of-the-art performance with reduced parameters and enhanced efficiency.

Searching arXiv for the cited RefAtomNet/RefAtomNet++ papers to ground the article. I’ll look up the main RefAtomNet++ and predecessor papers on arXiv, plus the unrelated AtomNet paper to disambiguate terminology. RefAtomNet++ is a vision-language framework for Referring Atomic Video Action Recognition (RAVAR), a task in which a model receives a video clip and a natural-language referring expression and jointly predicts the spatial location of the referred person-of-interest and that person’s fine-grained atomic actions. It was introduced as a successor to RefAtomNet, the first dedicated RAVAR model proposed with the RefAVA benchmark, and is designed to address limitations in cross-modal alignment and retrieval that remained in the earlier architecture (Peng et al., 18 Oct 2025). In the 2024 RAVAR work, no actual “RefAtomNet++” system existed; the name appeared only as a hypothetical future extension, whereas the 2025 paper introduced a concrete model and an expanded dataset, RefAVA++ (Peng et al., 2024).

1. Task definition and historical placement

RAVAR differs from conventional action recognition, action detection, and AVA-style atomic action localization because it does not treat all visible people symmetrically. Instead, it conditions prediction on a textual referring expression and requires the system to identify which individual the sentence denotes, localize that person, and infer that person’s atomic actions in one pass (Peng et al., 18 Oct 2025). The task is therefore centered on cross-modal disambiguation in multi-person scenes rather than on post-hoc selection from a set of independently predicted person-level actions.

The original RefAtomNet formulation established this setup using RefAVA and emphasized an end-to-end workflow in which language suppresses irrelevant visual cues and steers attention toward the intended human instance (Peng et al., 2024). RefAtomNet++ inherits the same task definition but reworks the alignment mechanism. The earlier RefAtomNet relied on cross-stream agent attention and agent token fusion across video, text, and location-semantic streams; RefAtomNet++ replaces that agent-attention-centered design with a semantic retrieval based multi-trajectory Mamba architecture and multi-hierarchical semantic-aligned cross-attention (Peng et al., 2024).

This progression is significant because it marks a shift from stream fusion around agent tokens to explicit semantic hierarchy construction. A plausible implication is that RefAtomNet++ treats alignment not merely as token interaction but as retrieval across several linguistic and contextual granularities, which is particularly relevant when multiple people share similar appearance cues or when scene context is needed to disambiguate actions.

2. RefAVA++ dataset and benchmark setting

RefAtomNet++ is evaluated on RefAVA++ and RefAVA. RefAVA++ is built from AVA videos with strict movie-level splits and contains more than $2.95$ million frames and $75{,}111$ annotated persons in total (Peng et al., 18 Oct 2025). Its train, validation, and test partitions comprise $48{,}151$ instances from 150 movies, $16{,}142$ instances from 50 movies, and $10{,}818$ instances from 31 unseen movies, respectively. Keyframe annotations are given at the clip’s center frame.

The annotation protocol uses 8 expert annotators who produce natural-language descriptions referencing identity, appearance, relative position, and salient context, but not actions. Ambiguous cases are discarded. Each annotated person is associated with a center-frame bounding box, which serves as ground truth localization for the person-of-interest. The action space contains 80 atomic classes grouped into Object Manipulation, Person Interactions, and Person Movement (Peng et al., 18 Oct 2025).

These design choices preserve continuity with RefAVA, which introduced person-level referring descriptions for AVA clips and likewise excluded action names from the referring expressions in order to avoid label leakage (Peng et al., 2024). RefAVA++ scales that setup substantially and preserves movie-level disjointness, which supports evaluation of generalization across unseen films.

The benchmark outputs remain twofold: a bounding box for the referred person and multi-label atomic action predictions, conditioned on the referring text and without external ROI cropping or post-hoc subject selection (Peng et al., 18 Oct 2025). This evaluation protocol is central to the model’s intended use in complex multi-person video settings.

3. Architectural design: semantic hierarchies, retrieval, and dual-branch fusion

RefAtomNet++ is built on BLIPv2 with a ViT visual encoder, QFormer, and OPT text encoder; object context is obtained via DETR for scene attributes, and all BLIPv2 encoders and the detector are frozen (Peng et al., 18 Oct 2025). The model operates over three semantic levels.

The holistic-sentence level encodes the entire referring sentence with the textual encoder to obtain sentence-level semantic tokens. These are fused with visual tokens by QFormer to yield visual-language tokens, which are then refined by Mamba for temporal coherence. The partial-keyword level removes stopwords, embeds the remaining keywords, and for each keyword dynamically selects the nearest visual spatial token at each timestep to form a keyword trajectory through time. The scene-attribute level runs DETR on the keyframe, embeds category labels, concatenates them with normalized box coordinates, and similarly retrieves nearest visual tokens per frame to form trajectories (Peng et al., 18 Oct 2025).

The nearest-token trajectory construction is written as

$$\mathbf{t}^{k}_{V}(l)=\arg\min_{\mathbf{t}\in\mathcal{T}(l)}\left\|\hat{\mathbf{e}}_{k}-\mathbf{t}\right\|_{2},$$

for keyword $k$, and

$$\widetilde{\mathbf{t}^{\,j}_{V}(l)}=\arg\min_{\mathbf{t}\in\mathcal{T}(l)}\left\|\mathbf{P}_{BS}(\mathbf{b}_{j})-\mathbf{t}\right\|_{2},$$

for scene attribute $j$ (Peng et al., 18 Oct 2025). The paper uses nearest selection with $K=1$.

Temporal aggregation is carried out by Mamba modules. The generic state-space formulation is

$75{,}111$0

and hierarchy-specific Mamba instances are used for keyword trajectories, scene-attribute trajectories, and holistic visual refinement (Peng et al., 18 Oct 2025). The paper motivates this choice by contrasting Mamba’s $75{,}111$1 sequence complexity with Transformer attention’s $75{,}111$2 behavior.

After retrieval and temporal modeling, RefAtomNet++ applies multi-hierarchical semantic-aligned cross-attention in two branches, one temporal and one spatial. For hierarchy $75{,}111$3 and branch $75{,}111$4, the core operator is

$75{,}111$5

with $75{,}111$6 after projection (Peng et al., 18 Oct 2025). The hierarchy-specific outputs are averaged:

$75{,}111$7

Bounding boxes and action logits are predicted independently from the temporal and spatial branches and then averaged:

$75{,}111$8

The model therefore does not depend on external tracking and instead regresses the person-of-interest box directly from fused spatio-temporal representations (Peng et al., 18 Oct 2025).

Relative to RefAtomNet, which used cross-stream agent attention to upweight salient tokens, this architecture represents a change in inductive bias. The paper states that RefAtomNet’s cross-modal alignment and retrieval remained limited; RefAtomNet++ addresses this by explicitly structuring retrieval at partial-keyword, scene-attribute, and holistic-sentence levels (Peng et al., 18 Oct 2025).

4. Optimization, parameterization, and inference path

The training objective combines multi-label classification and box regression. The binary cross-entropy term is

$75{,}111$9

with $48{,}151$0, and the bounding-box loss is

$48{,}151$1

The total loss is

$48{,}151$2

with default weights $48{,}151$3 (Peng et al., 18 Oct 2025).

The reported optimization setup uses BertAdam with learning rate $48{,}151$4, batch size 128, decay 0.9, warmup ratio 0.1, and 40 epochs on $48{,}151$5 NVIDIA A100 GPUs (Peng et al., 18 Oct 2025). All BLIPv2 encoders, QFormer, and the detector are frozen; the trainable components are the Mamba and cross-attention modules. Visual tokens have dimension $48{,}151$6 before projection to $48{,}151$7. The best number of learnable query prompts per hierarchy is 6, and the recommended temporal length is around 8 frames (Peng et al., 18 Oct 2025).

At inference time, the system processes video and referring text by building keyword and scene trajectories, aggregating them with Mamba, running multi-hierarchy cross-attention in temporal and spatial branches, and averaging the resulting box and action predictions (Peng et al., 18 Oct 2025). The paper reports that RefAtomNet++ reduces parameter count from 214M in RefAtomNet to 121M while improving both accuracy and efficiency (Peng et al., 18 Oct 2025).

This reduction matters because the predecessor’s inference cost was dominated by ViT, QFormer, and fusion modules and involved 214M trainable parameters (Peng et al., 2024). The newer design therefore couples architectural expansion in semantic structure with a smaller parameter budget.

5. Empirical results and ablation findings

The evaluation metrics are mIOU for person-of-interest localization, mAP for multi-label atomic action recognition, and AUROC across action classes (Peng et al., 18 Oct 2025). On both RefAVA and RefAVA++, RefAtomNet++ is reported to achieve state-of-the-art performance.

Benchmark	RefAtomNet	RefAtomNet++
RefAVA val	38.22 mIOU / 55.98 mAP / 69.73 AUROC	43.71 mIOU / 56.83 mAP / 71.27 AUROC
RefAVA test	36.42 mIOU / 57.52 mAP / 73.95 AUROC	42.52 mIOU / 59.81 mAP / 75.72 AUROC
RefAVA++ val	35.97 mIOU / 56.54 mAP / 70.97 AUROC	39.12 mIOU / 58.24 mAP / 72.58 AUROC
RefAVA++ test	35.73 mIOU / 57.01 mAP / 71.62 AUROC	38.58 mIOU / 58.84 mAP / 73.28 AUROC

On RefAVA test, the improvement over RefAtomNet is reported as $48{,}151$8 mIOU, $48{,}151$9 mAP, and $16{,}142$0 AUROC, with a parameter reduction of 93M (Peng et al., 18 Oct 2025). On RefAVA++ test, the gains are $16{,}142$1 mIOU, $16{,}142$2 mAP, and $16{,}142$3 AUROC (Peng et al., 18 Oct 2025). The paper also notes that AAL baselines struggle with mIOU of approximately $16{,}142$4–$16{,}142$5 on RefAVA++, while VQA and VTR baselines perform better but remain below RefAtomNet++.

The historical baseline picture is important. In the original RefAVA benchmark, BLIPv2 was the best baseline at 32.75 mIOU, 53.19 mAP, and 69.92 AUROC on test, while RefAtomNet improved those values to 36.42, 57.52, and 73.95, respectively (Peng et al., 2024). RefAtomNet++ extends that progression rather than altering the task or the metrics.

Ablation studies attribute performance to both the semantic hierarchy and the retrieval-temporal aggregation mechanism. Removing the holistic sentence branch, denoted as w/o T-QB, causes the largest reported drop; on RefAVA validation, mIOU falls to 27.14 (Peng et al., 18 Oct 2025). Removing the keyword or scene-attribute branches also degrades performance, indicating that fine-grained entity/attribute cues and contextual object cues are complementary. Mamba outperforms Transformer, LSTM, and linear projection alternatives; for example, on RefAVA test it yields 42.52 mIOU versus 39.06 for Transformer (Peng et al., 18 Oct 2025). Removing multi-hierarchical semantic-aligned cross-attention produces substantial losses, and the full model exceeds the variant without MHS-CA by up to $16{,}142$6 mIOU on RefAVA test (Peng et al., 18 Oct 2025).

Qualitative analysis indicates that RefAtomNet++ more consistently grounds the correct person-of-interest in crowded scenes and captures subtle multi-label actions, whereas baselines frequently attend to salient but irrelevant individuals or fail to use contextual objects such as phones or chairs for disambiguation (Peng et al., 18 Oct 2025). Statistical significance, however, is not reported.

6. Limitations, failure modes, misconceptions, and naming scope

The paper identifies several limitations. Performance depends on detector quality for scene attributes, so misdetections or missing objects can impair contextual retrieval. Ambiguous or underspecified references such as “the man” can still lead to incorrect person-of-interest selection in crowded scenes. The use of $16{,}142$7 nearest selection may be brittle under occlusion or noisy tokens, and there is no explicit tracking (Peng et al., 18 Oct 2025). The paper also notes vulnerability to highly dynamic camera motion, severe occlusions, scenes in which relevant contextual objects are too small or not detected, and linguistic rephrasings that remove discriminative attributes.

The proposed future directions follow directly from these limitations: end-to-end trainable scene-attribute extraction or joint detector-encoder finetuning, top-$16{,}142$8 trajectory selection with temporal smoothing and learnable selection gates, incorporation of person tracking or pose, and expansion toward audio, long-form memory, and causal reasoning (Peng et al., 18 Oct 2025). These suggestions indicate that the current model treats semantic retrieval and temporal aggregation effectively but still within a relatively constrained grounding regime.

A recurrent misconception concerns the name. RefAtomNet++ is unrelated to the separate “AtomNet” introduced for atom probe tomography point-cloud analysis, which operates on single-atom 3D APT data and addresses nanoscale microstructure extraction rather than video-language action recognition (Yu et al., 2024). The shared suffix does not imply architectural lineage between the two systems. Within RAVAR, the actual lineage is RefAtomNet to RefAtomNet++, from the 2024 introduction of the task and RefAVA to the 2025 introduction of RefAVA++ and the semantic retrieval based multi-trajectory Mamba model (Peng et al., 2024).

The code and dataset for RefAtomNet++ are released at https://github.com/KPeng9510/refAVA2, while licensing is not explicitly stated in the paper (Peng et al., 18 Oct 2025). Taken together, the available evidence presents RefAtomNet++ as a specialized RAVAR architecture whose main contribution is the explicit organization of cross-modal grounding into holistic-sentence, partial-keyword, and scene-attribute retrieval pathways, aggregated by Mamba and fused by multi-hierarchical semantic-aligned cross-attention.