---
title: 'REACT-Drive: Retrieval-Enhanced Planning'
url: https://www.emergentmind.com/topics/react-drive
type: topic
---

# REACT-Drive: Retrieval-Enhanced Planning

Searching arXiv for the core REACT-Drive paper and a few closely related adjacent works to ground the article in current literature.
REACT-Drive is a trajectory planning framework for autonomous driving in work zones that combines Vision-Language Models (VLMs) with Retrieval-Augmented Generation (RAG) to mitigate recurrent planning failures in irregular, temporary, and dynamically changing traffic-control environments. In the formulation presented in “Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Driving” [2510.02803], REACT-Drive is designed around a specific empirical diagnosis: mainstream VLMs often fail to generate correct trajectories in work zones, and many of these failures recur as recognizable structural patterns. REACT-Drive addresses this by converting prior failure cases into structured constraint rules and executable mitigation code, then retrieving and reusing those artifacts for similar future scenes. This positions it as a retrieval-enhanced, constraint-guided planning layer rather than a purely end-to-end VLM planner [2510.02803].

## 1. Definition and empirical motivation

REACT-Drive is introduced in the context of work-zone trajectory planning, where temporary traffic control, irregular layouts, dynamically reconfigured drivable areas, lane shifts, detours, cones, drums, barriers, and construction vehicles make direct VLM planning unreliable [2510.02803]. The motivating claim is explicit: while VLMs have shown promise in autonomous driving, their planning ability in work zones had not been systematically studied before this work, and the resulting failure rate is high.

The paper reports that mainstream VLMs fail to generate correct trajectories in **68.0%** of cases [2510.02803]. In the ROADWork evaluation, scenario-level failure rates are reported as **70.37%** for GPT4o, **75.86%** for Qwen2.5-VL, **80.00%** for Gemini2.5, **81.93%** for SimLingo, **76.63%** for RoboTron-Drive, and **77.23%** for DriveLM [2510.02803]. A case is defined as a failure if **ADE > 50 px** and **FDE > 100 px**, and a scenario is defined as a failure if **> 50% of its cases fail** [2510.02803]. The paper further highlights a model-specific comparison for Qwen2.5-VL: **FDE = 285.90** on ROADWork work-zone planning versus **106.38** on normal commonsense driving cases in nuScenes, which localizes the problem to work-zone reasoning rather than general trajectory prediction [2510.02803].

This suggests that REACT-Drive is motivated less by generic VLM weakness than by a long-tail failure mode in which work-zone semantics systematically violate the assumptions under which direct multimodal planning remains stable. The framework therefore treats prior failures as reusable planning knowledge rather than isolated errors.

## 2. Failure-pattern mining and scene-graph analysis

A central component of REACT-Drive is its failure analysis pipeline, which transforms failed VLM outputs into a structured inventory of recurrent abnormal patterns [2510.02803]. The procedure combines scene graph construction, candidate subgraph extraction, candidate merging, clustering, and human verification.

Each frame is encoded as a directed scene graph
$$
G = (V, E),
$$
where nodes correspond to work-zone-related entities and auxiliary structural nodes, and edges encode directional, proximity, and lane-membership relations [2510.02803]. The auxiliary node set is
$$
\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},
$$
while the work-zone category set includes drum, cone, work vehicle, ttc sign, fence, barricade, barrier, worker, tubular marker, and vertical panel [2510.02803]. A YOLOv12 detector fine-tuned on ROADWork produces object instances
$$
\{(v_i,\, b_i)\},
$$
with class label \(v_i\) and bounding box \(b_i=[x_i,\,y_i,\,w_i,\,h_i]\) [2510.02803].

Spatial relations are computed after converting image coordinates into an ego-centered metric plane:
$$
(x_i,\,y_i)=\left(\frac{x_i^{\text{px}}+\tfrac{w_i^{\text{px}}}{2}-c_x}{\mathrm{PPM}},\ \frac{c_y-\big(y_i^{\text{px}}+\tfrac{h_i^{\text{px}}}{2}\big)}{\mathrm{PPM}}\right),
$$
with the \(y\)-axis pointing forward [2510.02803]. Directional relations are defined through
$$
\Delta\mathbf{p}_{i\to j}=\mathbf{p}_j-\mathbf{p}_i,\qquad
\theta_{i\to j}=\mathrm{wrap}_{(-\pi,\pi]}\!\big(\mathrm{atan2}(\Delta y,\Delta x)-\psi_i\big),
$$
yielding \(inFrontOf\), \(toLeftOf\), and \(toRightOf\) relations by thresholding \(\theta_{i\to j}\) [2510.02803]. Proximity relations are estimated using MiDaS depth and quantized into `near_collision` \([0,4)\) m, `super_near` \([4,7)\) m, `very_near` \([7,10)\) m, `near` \([10,16)\) m, and `visible` \([16,25)\) m [2510.02803].

Candidate abnormal subgraphs are extracted with depth-limited BFS centered on ego:
$$
V^S=\{ego\}\ \cup\ \Big\{v\in V:\ \mathrm{dist}^{+}_{G}(ego,v)\le D,\ \text{label}(v)\in\mathcal{C}_{\mathrm{wz}}\Big\},
$$
with \(D=2\), and subgraphs with fewer than \(m=3\) nodes are discarded [2510.02803]. These candidates are then merged using a structural signature
$$
\sigma(S_i)=\big(\ L(S_i),\ R(S_i),\ |V^{S_i}|,\ |E^{S_i}|\ \big),
$$
together with threshold gating on average depth and average box-center distance, and relation-preserving directed subgraph isomorphism [2510.02803].

After clustering with K-means and choosing \(K=10\) by the elbow method on SSE, the authors obtain **10 initial abnormal clusters**, then reduce them to **8 common failure patterns** through human combination and semantic summarization [2510.02803]. The final patterns are:

- **P1: Dense drums or cones on sidewalk**
- **P2: Encounter dead end road**
- **P3: Interference from large work vehicles**
- **P4: Lane borrowing through work zone**
- **P5: Lane shift across work zones**
- **P6: Overreaction to signs**
- **P7: Accelerate through the exit in the work zone**
- **P8: Turning through work zone** [2510.02803]

Each pattern is associated with one or more rule needs such as “follow the lane center,” “turn to avoid work zone,” “detour to bypass the work zone,” “return the origin lane after bypassing the work zone,” “cross the work zone,” “return center line after crossing work zone,” “follow the sign,” and “follow the front car” [2510.02803]. A plausible implication is that REACT-Drive treats work-zone planning as a finite set of recurrent structural subproblems rather than a single undifferentiated multimodal inference task.

## 3. Rule induction and executable mitigation code

REACT-Drive is a two-stage framework. In the offline stage, prior failure cases are converted into a failure-case mitigation code database; in the online stage, similar cases are retrieved and their stored mitigation logic is executed for the new scene [2510.02803].

In the offline stage, the system uses a VLM to infer applicable work-zone constraints from a failure case and to generate executable Python code implementing those constraints [2510.02803]. The input includes a front-view image, overlapping failure trajectories, a visible failure destination point, and a ground-truth annotated image [2510.02803]. The paper states that this uses **8 predefined work-zone constraint templates** based on work-zone traffic regulations from PennDOT guidance, although the full list of all eight fields is not fully enumerated in the provided text [2510.02803].

Rules are represented in a JSON-like schema with fields such as:

```json
{
  "constraints": {
    "no_cross_workzone": "UNKNOWN",
    "detour_side": "UNKNOWN",
    "return_to_original_lane_after_workzone": "UNKNOWN"
  }
}
```

where `"no_cross_workzone"` specifies whether crossing is allowed, `"detour_side"` is `"left"`, `"right"`, or `"none"`, and `"return_to_original_lane_after_workzone"` is `"True"` or `"False"` [2510.02803].

The generated mitigation code must define exactly two functions:

```python
def segment_driveable_mask(original_road_mask, workzone_info):
    ...

def plan_destination(driveable_road_mask, workzone_info):
    ...
```

The paper also describes these functions as `segment_drivable_mask` and `plan_destination`; the spelling inconsistency between `driveable` and `drivable` appears in the source, but the intended semantics are stable [2510.02803]. The first function adjusts the drivable road mask according to work-zone constraints and blocks undrivable regions; the second chooses a destination point consistent with those constraints [2510.02803].

The prompt template directs the VLM to output both completed constraints in JSON and executable Python code with no placeholders and no extra commentary [2510.02803]. This is followed by self-verification using two explicit checks. First, the **Drivability Constraint** requires the predicted target \(\mathbf{x}_{\text{pred}}\) to satisfy
$$
D(\mathbf{x}_{\text{pred}}) \le \tau_{\text{road}},
$$
where \(D(\mathbf{x})\) is the Euclidean distance transform to the drivable set \(\Omega_{\text{drive}}\) [2510.02803]. Second, the **Destination Constraint** requires the predicted destination to remain sufficiently close to the ground-truth target in pixel space:
$$
d_{\mathrm{pix}}=\sqrt{(x_{\mathrm{pred}}-x_{\mathrm{gt}})^2+(y_{\mathrm{pred}}-y_{\mathrm{gt}})^2} \le \tau.
$$
If validation fails, the system retries with actionable feedback; once it passes or reaches maximum retries, the rules and code are stored with the validation record [2510.02803].

This suggests that REACT-Drive does not merely retrieve prior trajectories. It retrieves a verified programmatic transformation of the planning problem.

## 4. Retrieval-augmented planning and online inference

The online stage of REACT-Drive uses RAG-style retrieval over the failure-case mitigation code database [2510.02803]. For a new scenario, the system extracts multimodal scene features from images, semantic annotations, and temporal context, then retrieves similar prior cases and executes their stored mitigation code [2510.02803]. If a highly similar case is not found, the system falls back to the base VLM reasoning pipeline [2510.02803].

The paper provides an explicit triplet-based similarity formulation in the pattern-analysis component. For an abnormal prototype \(R_g\) and a retrieved candidate subgraph \(N\), structural similarity is
$$
\text{sim}_{\text{struct}} = \tfrac12\!\Big(
\mathrm{Jacc}\big(\mathrm{set}(L(R_g)),\,\mathrm{set}(L(N))\big)
+
\mathrm{Jacc}\big(\mathrm{set}(R(R_g)),\,\mathrm{set}(R(N))\big)
\Big),
$$
depth similarity is
$$
\text{sim}_{\text{depth}} = \max\!\left\{0,\ 1-\frac{\big|\bar{d}(R_g)-\bar{d}(N)\big|}{3.5}\right\},
$$
and bounding-box similarity is
$$
\text{sim}_{\text{bbox}} = \max\!\left\{0,\ 1-\frac{\big|\bar{\delta}(R_g)-\bar{\delta}(N)\big|}{\max\{960,540\}}\right\}.
$$
A priority rule labels a prototype non-independent when the minimum of the three similarities is at least \(0.8\) [2510.02803].

While the paper does not fully specify the production retrieval stack—such as ANN backend, exact online Top-\(K\), or multi-retrieval fusion policy—it is clear about what is retrieved and how it is used. The retrieved artifact is the associated **stored mitigation code**, which directly modifies the drivable mask and destination selection for the new scenario [2510.02803]. In effect, retrieval influences planning by selecting which validated constraints and executable planning transformation should govern the current scene.

This distinguishes REACT-Drive from a generic retrieval-only memory system. Its retrieved knowledge is operational, not merely descriptive.

## 5. Trajectory planning formulation and empirical performance

After retrieval, REACT-Drive performs planning in three steps: retrieve a similar prior failure case, execute its mitigation code to modify the drivable road mask and compute a destination point, and generate a smoothed trajectory of **20 discrete points** from the start to that destination [2510.02803]. The paper does not provide a full optimization objective for the final smoothing stage, nor a controller-level motion model; the planning description is procedural rather than variational.

Trajectory quality is evaluated with:
$$
\mathrm{ADE} = \tfrac{1}{T}\sum_{t=1}^{T}\|\hat{\mathbf{p}}_{t}-\mathbf{p}_{t}\|_{2},
$$
$$
\mathrm{FDE} = \|\hat{\mathbf{p}}_{T}-\mathbf{p}_{T}\|_{2},
$$
$$
\mathrm{CR}  = \tfrac{1}{N}\sum_{i=1}^{N} C(\hat{Y}_i),
$$
where \(\hat{\mathbf{p}}_t\) is the predicted position, \(\mathbf{p}_t\) is the ground-truth position, and \(C(\hat{Y}_i)\) is the collision indicator [2510.02803]. Because accurate depth is not available in ROADWork, ADE and FDE are measured in pixel space [2510.02803].

The main dataset is **ROADWork**, described as the first dataset designed for driving through work zones, containing **1186 scenarios** with multi-granularity work-zone annotations and drivable trajectory annotations from real driving videos [2510.02803]. Baselines include GPT4o, Gemini-2.5, Qwen2.5-72B-VL, DriveLM, SimLingo, and RoboTron-Drive; additional mitigation baselines are fine-tuned Qwen2.5-72B-VL using QLoRA and a self-prediction variant without REACT’s constraint rules [2510.02803].

Average results on ROADWork are:

| Method | ADE | FDE | CR |
|---|---:|---:|---:|
| fine-tune VLM | 207.97 | 384.31 | 0.11 |
| VLM (Self) w/o Constraint Rules | 201.09 | 350.75 | 0.03 |
| REACT | 54.73 | 101.64 | 0.04 |

These values ground the paper’s claim that REACT-Drive yields roughly a **3×** reduction in average displacement error relative to VLM baselines under evaluation with Qwen2.5-VL [2510.02803]. Pattern-specific results show especially strong gains on P1, P3, P6, and P7, where REACT reduces **CR to 0.00** [2510.02803]. For P2, the dead-end-road pattern, REACT reduces FDE to **110.44** versus **532.96** for the fine-tuned baseline [2510.02803].

The framework also transfers across VLM backbones. With REACT, GPT4o achieves **ADE 54.73**, **FDE 101.64**, **CR 0.04**, while Qwen2.5 achieves **ADE 86.46**, **FDE 124.67**, **CR 0.07** [2510.02803]. On efficiency, REACT reports **0.58 s** inference time, compared with **17.90 s** for the fine-tuning-based mitigation method [2510.02803]. The paper further notes approximate latencies of around **5 s** for GPT4o and **3 s** for RoboTron-Drive, making REACT the fastest reported method in its comparison [2510.02803].

This suggests that REACT-Drive’s practical advantage is not only better geometric planning accuracy but also lower online computational burden, because its most expensive reasoning step—constraint and code generation—has been shifted offline into the database-construction phase.

## 6. Real-world evaluation, scope, and relation to adjacent paradigms

REACT-Drive is additionally evaluated on **15 real-world work zone scenarios** collected from local driving environments different from ROADWork, totaling **100 images** [2510.02803]. Ground-truth trajectories are jointly annotated by two authors [2510.02803]. In this physical-world evaluation, the reported results are:

| Method | ADE | FDE | CR |
|---|---:|---:|---:|
| GPT4o | 127.68 | 225.06 | 0.03 |
| DriveLM | 134.53 | 247.28 | 0.06 |
| SimLingo | 182.61 | 271.56 | 0.03 |
| RoboTron-Drive | 143.48 | 253.19 | 0.05 |
| REACT | 64.28 | 106.47 | 0.00 |

The real-world transfer result of **FDE 106.47** and **CR 0.00** is one of the strongest empirical arguments for the method’s practicality [2510.02803]. At the same time, the appendix explicitly states a limitation: despite using physical-world data collected from a real vehicle environment, the framework was **not deployed on a real autonomous vehicle** in active work zones due to safety concerns [2510.02803]. The most precise reading is therefore that REACT-Drive has physical-world evaluation but not a closed-loop real-vehicle deployment.

The paper identifies several limitations. It does not systematically study extreme-weather or nighttime work zones, and evaluation is limited to ROADWork plus the self-collected physical data [2510.02803]. The final planning and smoothing module is described procedurally rather than through a fully specified optimization or controller formulation. Performance also depends on pattern coverage: when training with only pattern 1 coverage, the system reports **ADE 341.57**, **FDE 703.26**, **CR 0.14**, whereas with **7 out of 8 patterns** covered, FDE improves to **275.07** [2510.02803]. This indicates that retrieval memory diversity materially affects generalization.

Conceptually, REACT-Drive belongs to a broader class of systems that place language reasoning above structured tool or program interfaces rather than relying on monolithic multimodal end-to-end inference. This suggests an architectural kinship with multimodal reasoning-and-tool-use systems such as MM-REACT, which likewise combine a language model with external experts instead of relying solely on joint multimodal finetuning [2303.11381]. A plausible implication is that REACT-Drive can be understood as a work-zone-specific instantiation of that broader modular pattern: identify what prior structured knowledge is missing, retrieve or execute the corresponding external procedure, and only then synthesize the final answer or plan.

In summary, REACT-Drive is best understood as a retrieval-enhanced, constraint-verified trajectory planning framework for work zones. Its distinctive contribution is to transform prior VLM failures into validated rules and executable code, then reuse those artifacts online to guide new planning problems. Within the scope studied, it demonstrates that failure memory, when lifted into programmatic constraints rather than stored as raw trajectories alone, can materially improve both robustness and runtime in one of autonomous driving’s most difficult long-tail settings [2510.02803].

Source: https://www.emergentmind.com/topics/react-drive