---
title: 'REACT++: Real-Time Scene Graph Generation'
url: https://www.emergentmind.com/topics/react-7293fbc1-8285-44ae-8406-9aa9f689ebf7
type: topic
---

# REACT++: Real-Time Scene Graph Generation

REACT++ is a real-time scene graph generation (SGG) architecture that extends the REACT framework to balance three objectives that are usually in tension: high relation prediction accuracy, high object detection accuracy, and low inference latency. It builds on a decoupled two-stage detector, replaces heavier region-feature extraction with Detection-Anchored Multi-scale Pooling (DAMP), injects global scene context through an AIFI branch, and performs relation decoding in a compact prototype space using CARPE, a cross-attention relation head with explicit geometry encoding. On the reported PSG benchmark with a YOLOv8m-DTS backbone, REACT++ attains the highest inference speed among existing SGG models while improving relation prediction accuracy without sacrificing object detection performance; compared with REACT, it is reported as 20% faster with a gain of 10% in relation prediction accuracy on average [2603.06386].

## 1. Terminology and domain scope

The name **REACT++** most specifically denotes **“REACT++: Efficient Cross-Attention for Real-Time Scene Graph Generation”**, a computer-vision model for SGG in static images [2603.06386]. In that setting, the goal is to output graph-structured triplets of the form $\langle \text{subject}, \text{predicate}, \text{object} \rangle$ from detected objects and their relations.

The term should not be conflated with similarly named systems in other domains. In question answering with large language models, **Focused ReAct** is described as a **“ReAct++-style enhancement”** that adds reiteration and early stop to the Thought–Action–Observation loop in order to reduce focus drift and action loops, but it is not the same scene-graph model family [2410.10779]. Likewise, **REACT** in wearable biosignal modeling denotes a **conditioning framework for user-adaptive sEMG hand pose estimation**, where a compact user embedding modulates a frozen EMG-to-pose backbone through FiLM; this is a distinct architecture, task, and modality [2605.30127].

A common source of confusion is therefore nominal rather than methodological. In current usage, **REACT++** is primarily the SGG system, whereas **Focused ReAct** and the sEMG **REACT** framework belong to separate literatures with different problem formulations, data modalities, and control mechanisms.

## 2. Problem setting and design objectives

Scene graph generation combines **OD** and **RelPred**. The central design problem addressed by REACT++ is the real-time compromise among three competing objectives: **high relation prediction accuracy, high object detection accuracy, and low inference latency** [2603.06386].

The paper motivates the architecture against two established lines of SGG. First, **two-stage methods** based on **Faster R-CNN + ROI Align** achieve reasonable relation accuracy, but they are latency-heavy because of expensive feature extraction and pairwise fusion; they also often degrade OD performance because object classification is re-learned within the relation head. Second, **one-stage DETR-family models** reduce latency by avoiding dense pair enumeration, but their OD accuracy is sub-optimal in SGG benchmarks due to joint learning of objects and relations, and they still need global context and directional geometry bias to capture fine-grained predicates [2603.06386].

The original **REACT** addressed latency using a **YOLO-based DTS detector** and **prototype learning for relations**, but left three bottlenecks: **slow, parameter-heavy ROI Align for subject/object feature extraction; limited global context aggregation; and symmetric prototype interactions**, which make directional relations such as $\langle \text{person}, \text{above}, \text{dog} \rangle$ versus $\langle \text{dog}, \text{above}, \text{person} \rangle$ harder to learn [2603.06386].

REACT++ is presented as a first-principles redesign of this two-stage setting. It preserves the advantages of a decoupled detector while replacing feature alignment, context pooling, and the relation head with lighter and more expressive modules that operate in a compact prototype space. This suggests that the model’s defining claim is not a single accuracy increase in isolation, but a systems-level rebalancing of feature extraction cost, relation expressivity, and detector preservation.

## 3. Detection-anchored feature extraction and global context

The backbone is a **Decoupled Two-Stage (DTS)** design. REACT++ freezes the detector and reuses its class logits, thereby preserving OD accuracy and enabling plug-in detectors from the **YOLOv8/9/10/11/12** family. In the reported setup, the stage-1 backbone is a **YOLO family detector** such as **YOLOv8m**, with a **CSPNet-based backbone** feeding a **PAN-FPN neck** over scales $P3$, $P4$, and $P5$, followed by an anchor-free detection head and **Non-Maximum Suppression (NMS)**. Stage 1 outputs a set of $K$ detections with class probabilities, box coordinates, and indices into the corresponding FPN feature maps, together with the multi-scale feature maps $\{FPN_i\}_{i \in \{3,4,5\}}$ [2603.06386].

The first major replacement is **Detection-Anchored Multi-scale Pooling (DAMP)**, which substitutes **ROI Align** with pooled feature vectors gathered directly from detector grid indices using a small Gaussian neighborhood. For a detection with FPN indices $(r_i, c_i)$ at level $i$, REACT++ gathers a $3 \times 3$ neighborhood with weights $w(\delta_r,\delta_c) \propto \exp\!\left(-(\delta_r^2+\delta_c^2)\right)$:
$$
g_i = \sum_{\delta_r,\delta_c} \left[ w(\delta_r,\delta_c) \cdot FPN_i[r_i + \delta_r, c_i + \delta_c] \right] \in \mathbb{R}^{C_i}.
$$
Each scale is then projected to a shared dimensionality $D$ and fused:
$$
o = W_{\text{fuse}} \cdot LN( W_{P3} g_{P3} \parallel W_{P4} g_{P4} \parallel W_{P5} g_{P5} ) \in \mathbb{R}^{D}.
$$
This replaces $K \times 7 \times 7 \times 3$ ROI samples with $K \times 9 \times 3$ gathers, and the gather stage has no learned parameters [2603.06386].

The paper reports that DAMP reduces feature extraction latency by **up to 32%** while improving **F1@K** by approximately **+16 points** over ROI Align in IndoorVG ablations: **F1: 8.0 → 23.8**, **latency: 29.2 → 20.8 ms**. A simpler **DA** variant is fastest but loses **1.9 F1@K** relative to DAMP, whereas **DAP** and **DAM** recover some accuracy at negligible latency; DAMP is reported as the best accuracy–latency compromise [2603.06386].

The second major addition is **AIFI**, an **Attention-based Intra-scale Feature Interaction** block used to derive a compact global context vector $g \in \mathbb{R}^{D}$, functioning as a scene prior such as *kitchen*, *beach*, or *mountain*. Empirically, AIFI yields **+0.42 F1@K (+1.8%)**, at the cost of approximately **+2.8 ms** latency and about **2M parameters** in the relation head [2603.06386].

These design choices make feature extraction explicitly detector-anchored rather than region-resampled. A plausible implication is that REACT++ treats detector outputs not only as localization hypotheses but also as indexing primitives for low-cost relation-aware feature retrieval.

## 4. Prototype-space relation reasoning

The core relation head is **CARPE**, short for **Cross-Attention Rotary Prototype Embedding**. REACT++ operates in a compact prototype space: instead of enumerating all subject–object pair interactions in a high-dimensional feature space, it lets subject and object features query a bank of learned predicate prototypes via cross-attention. The paper describes this as reducing compute from quadratic in the number of boxes to near-linear in the number of boxes times the number of prototypes, while increasing expressivity through attention, multi-head mixing, and geometry-aware logits [2603.06386].

CARPE begins from the notation
- subject visual feature $x_s \in \mathbb{R}^{D}$ from DAMP,
- object visual feature $x_o \in \mathbb{R}^{D}$ from DAMP,
- semantic class embeddings $t_s, t_o \in \mathbb{R}^{d_e}$,
- predicate prototype bank $P = \{c_k\}_{k=1..P}$ with $c_k \in \mathbb{R}^{D}$,
- geometry encoding $b \in \mathbb{R}^{9}$ derived from union box, centers, widths, heights, and area [2603.06386].

Semantic inputs are lifted using **SwiGLU**:
$$
\text{SwiGLU}(x) = W_{\text{out}} \big( \text{SiLU}(W_{\text{gate}}x) \odot W_{\text{val}}x \big),
$$
with subject and object weights optionally tied as $W_{\text{sub}} = W_{\text{obj}}$ to halve parameters. Predicate prototype seeds are likewise lifted from text embeddings to initialize prototypes [2603.06386].

To stabilize rare predicates, REACT++ maintains an **EMA prototype memory**:
$$
\hat{c}_k \leftarrow m \cdot \hat{c}_k + (1-m) \cdot \bar{r}_k^{+},
$$
with $m \approx 0.999$. If initialized, $\hat{c}_k$ replaces the learnable $c_k$ at inference, improving tail stability [2603.06386].

The geometry-aware cross-attention uses shared subject/object attention weights but different inputs and geometry biases. With multi-head attention and head dimension $d_k = D/M$, the queries, keys, and values are
- subject branch: $Q_s = W_Q x_s$, $K_p = W_K[c_1 \dots c_P]^T$, $V_p = W_V[c_1 \dots c_P]^T$,
- object branch: $Q_o = W_Q x_o$, with shared $K_p$ and $V_p$.

The attention operator is
$$
\text{Attention}(Q,K,V;B) = \text{softmax}\!\left( \frac{QK^T + B}{\sqrt{d_k}} \right)V,
$$
where the per-head geometry bias $B_h(b) \in \mathbb{R}^{1 \times P}$ conditions on relative position and scale. The enriched subject and object representations are
$$
\hat{s} = x_s + \text{Attn}(Q_s, K_p, V_p; B_s^{\text{geo}}), \qquad
\hat{o} = x_o + \text{Attn}(Q_o, K_p, V_p; B_o^{\text{geo}}).
$$
The associated attention weights
$$
\alpha_s = \text{softmax}\!\left( \frac{Q_sK_p^T + B_s}{\sqrt{d_k}} \right), \qquad
\alpha_o = \text{softmax}\!\left( \frac{Q_oK_p^T + B_o}{\sqrt{d_k}} \right)
$$
act as mixing coefficients over the prototype bank [2603.06386].

The final relation embedding is
$$
r_{s,o} = W_r[\hat{s} \parallel \hat{o} \parallel g] \in \mathbb{R}^{D},
$$
followed by predicate logits
$$
y = W_{\text{cls}} r_{s,o} + c,
$$
where $c$ applies class-dependent logit adjustments. Triplet ranking then uses the standard two-stage product of detection and predicate priors:
$$
\theta_{\text{rel}}(s,p,o) = \theta_{\text{obj}}(o)\cdot\theta_{\text{pred}}(p \mid s,o)\cdot\theta_{\text{subj}}(s).
$$
This formulation encodes asymmetry through separate subject and object queries plus geometry-conditioned biases, directly addressing the symmetry limitation identified in REACT [2603.06386].

## 5. Optimization, inference, and empirical performance

The computational motivation of REACT++ is explicit. Dense pairwise fusion in earlier two-stage SGG incurs costs of order $O(N^2D)$ plus ROI Align interpolation $O(N \cdot S \cdot H \cdot W)$. In REACT++, prototype cross-attention scales as $O(N \cdot P \cdot D)$ per branch, or $O(2NPD)$ total, and DAMP gather cost is $O(N \times 9 \times 3)$ memory fetches rather than $O(N \times 7^2 \times 3)$ for ROI Align. With typical values $N \approx 100$, $P \approx 50$, $D \approx 256$, and $M \approx 8$, the arithmetic count is similar at the stated nominal point, but REACT++ avoids the quadratic blow-up in keys/values memory; when **Dynamic Candidate Selection (DCS)** reduces proposals to approximately **47** on PSG, the operation count drops to about **$1.2 \times 10^6$**, and ROI Align is removed entirely. The paper attributes the resulting real-time behavior to this prototype-space formulation, rectangular GPU-friendly matrix sizes, and the smaller parameter count of about **35.8M total** [2603.06386].

Training is decoupled. The detector is trained separately with YOLO recipes, while relation training freezes detector heads and optimizes a predicate loss with **focal loss** and **logit adjustment**:
$$
L_{\text{rel}} = FL(y,p^*) + LA(y).
$$
The combined objective is
$$
L = \lambda_{\text{rel}} L_{\text{rel}} + \sum_i \lambda_i L_{\text{aux},i},
$$
where optional auxiliaries include prototype EMA warm-up penalties, geometry-embedding regularization, and mild $L_2$ weight decay via AdamW. Recommended hyperparameters on PSG are **AdamW**, initial learning rate **$1e^{-4}$**, cosine annealing, and **gradient accumulation 4**; the experiments section also reports a stable **SGD** variant with learning rate **$1e^{-4}$** [2603.06386].

The inference pipeline consists of five steps. First, stage 1 runs the YOLO detector with NMS and retains class logits, confidence, and FPN indices. Second, DCS estimates the accuracy–latency curve $f(k)$ over retained proposal count $k$ on a validation set and chooses
$$
x_{\text{opt}} = \min \{ x \mid |f'(x)| < \epsilon \}, \qquad \epsilon \approx 1e^{-5}.
$$
On PSG, saturation occurs at **$k \approx 47$**, reducing latency by about **66.5% on average across DTS backbones** with at most **1% F1@K drop**. Third, DAMP extracts pooled box features and AIFI computes the global context vector. Fourth, CARPE forms all subject–object pairs among the retained detections, predicts predicates, and ranks triplets using $\theta_{\text{rel}} = \theta_{\text{subj}}\theta_{\text{obj}}\theta_{\text{pred}}$. Fifth, the top-ranked relations are assembled into the scene graph [2603.06386].

The reported experiments use **PSG**, **IndoorVG**, and **VG150**, over **PredCls**, **SGCls**, and **SGDet** variants, evaluated with **R@K**, **mR@K**, **F1@K**, **OD mAP@50**, and latency in **ms per image** at **batch size 1**. On **PSG** with **YOLOv8m-DTS**, REACT++ without DCS reports **mR@20/50/100 = 22.2 / 24.9 / 27.1**, **R@20/50/100 = 29.1 / 34.2 / 37.5**, **F1@K = 28.4**, **OD mAP@50 = 53.1**, **latency = 25.9 ms**, and **35.8M parameters**. With DCS, the paper reports **the same accuracy** and **latency = 19.4 ms**. The corresponding REACT baseline without DCS reports **mR@20/50/100 = 18.3 / 20.1 / 20.9**, **R@20/50/100 = 27.6 / 30.9 / 32.3**, **F1@K = 23.9**, **latency = 32.5 ms**, and **43.3M parameters** [2603.06386].

The headline comparison is therefore multifaceted rather than single-metric. On PSG, the summary improvement is **20% faster (32.5 → 25.9 ms)**, a reported **~10% average gain in relation accuracy**, and **−17% parameters**. The same section also notes that **F1@K improves from 23.9 to 28.4**, which is approximately **+18.8% relative**. On **IndoorVG**, REACT++ reports **mR@K = 20.7**, **R@K = 28.1**, **F1@K = 23.9**, **mAP@50 = 37.2**, compared with REACT at **18.0**, **30.9**, **22.8**, and **37.2** respectively. On **VG150**, REACT++ reports **mR@K = 13.2**, **R@K = 28.9**, **F1@K = 18.2**, **mAP@50 = 31.8**, compared with REACT at **12.9**, **27.4**, **17.6**, and **31.8** [2603.06386].

## 6. Limitations, trade-offs, and broader context

The paper identifies several limits. First, **noisy datasets such as VG150** reduce the benefits of compact relation heads, and the lower-parameter CARPE head can be more sensitive to label noise and ambiguous classes. Second, REACT++ remains **reliant on detector quality**: OD accuracy drives relational performance, so poor boxes limit the gains achievable through prototype-space attention. Third, the system is formulated for **static images only**; the authors propose extending CARPE with **temporal prototypes** and **motion-aware geometry biases** for video SGG [2603.06386].

The trade-off profile is also explicit. DAMP improves both speed and F1@K relative to ROI Align, but AIFI adds a small latency cost, and DCS introduces an operating-point decision based on the empirical accuracy–latency curve. The reported **YOLO12 family** experiments further suggest a backbone sweet spot: **YOLO12m** yields **F1@K ≈ 30.0** and **mAP@50 ≈ 53.5** on PSG at about **28 ms**, whereas **YOLO12l** increases latency without being strictly more accurate in F1@K [2603.06386]. This suggests that relation learning convergence, rather than detector scale alone, constrains the practical optimum.

In application terms, the architecture is positioned as suitable for **real-time SGG** and for **reasoning for embodied agents**. The paper states that REACT++ is well-suited to **onboard deployment** because of its low latency and parameter count, and identifies future integration with **navigation/planning** and **online adaptation of prototypes to domain-specific predicates** as directions of interest [2603.06386].

Within the broader “REACT” naming landscape, REACT++ therefore occupies a specific niche: it is not a general-purpose reasoning policy, nor a personalization module for biosignals, but a compact and decoupled SGG architecture whose central contribution is an efficiency-oriented redesign of relation decoding around detector-anchored features, global scene context, and prototype-space cross-attention.

Source: https://www.emergentmind.com/topics/react-7293fbc1-8285-44ae-8406-9aa9f689ebf7