REACT++: Efficient Cross-Attention for Real-Time Scene Graph Generation

Abstract: Scene Graph Generation (SGG) is a task that encodes visual relationships between objects in images as graph structures. SGG shows significant promise as a foundational component for downstream tasks, such as reasoning for embodied agents. To enable real-time applications, SGG must address the trade-off between performance and inference speed. However, current methods tend to focus on one of the following: (1) improving relation prediction accuracy, (2) enhancing object detection accuracy, or (3) reducing latency, without aiming to balance all three objectives simultaneously. To address this limitation, we build on the powerful Real-time Efficiency and Accuracy Compromise for Tradeoffs in Scene Graph Generation (REACT) architecture and propose REACT++, a new state-of-the-art model for real-time SGG. By leveraging efficient feature extraction and subject-to-object cross-attention within the prototype space, REACT++ balances latency and representational power. REACT++ achieves the highest inference speed among existing SGG models, improving relation prediction accuracy without sacrificing object detection performance. Compared to the previous REACT version, REACT++ is 20% faster with a gain of 10% in relation prediction accuracy on average. The code is available at https://github.com/Maelic/SGG-Benchmark.

• The paper introduces a decoupled two-stage pipeline that uses a YOLO-based backbone and cross-attention modules to achieve efficient real-time scene graph generation.
• It demonstrates significant improvements with up to 54.4% higher mAP in object detection, 20% better meanRecall@K, and an 84.99% latency reduction.
• Innovative components like DAMP, AIFI, CARPE, and DCS optimize both performance and speed, enabling practical deployment on resource-constrained systems.

Authoritative Summary of "REACT++: Efficient Cross-Attention for Real-Time Scene Graph Generation" (2603.06386)

Introduction and Motivation

Scene Graph Generation (SGG) aims to parse visual scenes into structured graphs, encoding objects and their pairwise relationships. SGG is a crucial component for tasks such as VQA, image captioning, and embodied agent reasoning. Existing approaches either prioritize relation prediction accuracy, object detection performance, or inference speed, but fail to balance all three objectives for real-time deployment. REACT++ extends the REACT framework and introduces a decoupled two-stage (DTS) pipeline with efficient cross-attention-driven relation modeling, leveraging a YOLO-based backbone for object detection and replacing computational bottlenecks with specialized modules for fast, accurate SGG.

Architectural Innovations

Decoupled Two-Stage Pipeline

REACT++ fully decouples the object detector and relation predictor. The object detector is frozen post-training, and non-maximum suppression (NMS) is performed before feeding region proposals into the relation head. This enables independent optimization, permits the use of real-time detectors like YOLO, and avoids redundant class label computations seen in prior Faster-RCNN-based pipelines.

Figure 1: REACT++ Stage 1 featuring YOLO with refined feature extraction via AIFI global context block and DAMP local pooling.

DAMP: Detection-Anchored Multi-Scale Pooling

YOLO architectures lack region-wise feature extractors like RoI Align. The DAMP module leverages YOLO's grid-based detection and directly gathers spatial features from multiple FPN levels using Gaussian-weighted neighborhoods anchored at the detection peaks. This replaces high-latency interpolation/pooling operations and achieves a $5.4\times$ reduction in computation for feature extraction, delivering improved accuracy and latency.

Figure 2: Latency comparison and F1@K evolution for REACT++ with each feature extraction variant, validating gains from DAMP over RoI Align.

AIFI: Attention-Based Intra-scale Feature Interaction for Global Context

Global context features, inspired by RT-DETR's AIFI block, are pooled at low computational cost and injected to augment node and edge representations. Ablation studies show this module contributes to mR@K and F1@K, facilitating better tail predicate learning.

CARPE: Cross-Attention Rotary Prototype Embedding

Traditional relation modeling with fused subject/object representations misses relational asymmetry. CARPE introduces dedicated subject/object cross-attention layers, using visual features as queries and semantic prototypes as keys/values, modulated by geometry-adaptive RoPE bias for spatial encoding. An EMA buffer stabilizes prototype representation for rare classes.

Figure 1: REACT++ Stage 1 illustrates AIFI and DAMP integration for efficient feature extraction and global context fusion.

Dynamic Candidate Selection (DCS)

DCS dynamically chooses the optimal proposal count for relation pairing by estimating the inflection point in recall performance as a function of proposal count, reducing unnecessary pairwise computation during inference with minimal impact on F1@K.

Figure 3: Latency and F1@K for REACT++ as proposal count varies, demonstrating DCS enables sub-20ms real-time inference without significant accuracy degradation.

Empirical Results

REACT++ demonstrates on three SGG datasets (PSG, IndoorVG, VG150):

• Object Detection: DTS-YOLO achieves up to 54.4% higher mAP than TS-FasterRCNN baselines and 120% higher than OS-transformer approaches.
• Relation Prediction: REACT++ improves meanRecall@K by 20% relative to REACT, and is consistently superior in F1@K, with strong performance for tail predicates.
• Speed: REACT++ with DCS attains <20ms latency, an 84.99% reduction vs. TS baselines (Figure 2). Parameter count is reduced by 77.5% on average.
• Scaling: REACT++ integrates seamlessly with all YOLO variants, with YOLO12-M achieving the highest F1@K and REACT++ consistently outperforming its counterparts.

  Figure 4: Trade-off curves for Recall@K, meanRecall@K, mAP@50 vs. latency across YOLO12 variants and REACT++.

Ablation and Component Analysis

Ablation studies isolate the impact of:

• Feature Extractor: DAMP delivers $8\%$ F1@K improvement over direct box indexing, with negligible latency overhead.
• Global Context (AIFI): Inclusion of global context boosts F1@K and mR@K, especially for rare predicates.
• DCS: Allows adaptive proposal selection, cutting latency by $\sim$66.5% with only a 1% drop in F1@K.
• Backbone Scaling: YOLO12-M strikes the best balance between speed and accuracy, confirming detector–SGG synergy.

Implications and Future Directions

REACT++ establishes a strong compromise between relational modeling, detection accuracy, and real-time latency, substantiated by numerical results on all major SGG benchmarks. It addresses architectural inefficiencies endemic to two-stage SGG with ROI-based pooling, and supports real-time deployment on resource-constrained systems. The decoupling of detection and relation prediction unlocks plug-and-play adaptability with evolving detector architectures.

Practically, the model is well-suited for downstream applications (embodied reasoning, robotics, VQA) and ready for onboard deployment. Theoretically, the demonstrated improvements in relational asymmetry handling and low-latency context fusion highlight the importance of expressive cross-attention mechanisms in prototype-based relation modeling. Future research can explore further integration of multimodal context, open-vocabulary SGG, and continual learning paradigms with the REACT++ architecture.

Conclusion

REACT++ delivers efficient, expressive, and modular scene graph generation with advances in cross-attention prototype learning, low-latency feature extraction, and dynamic proposal selection. It achieves state-of-the-art performance in real-time SGG, setting a new baseline for both practical application and theoretical research in structured visual understanding.