REACT-Drive: Retrieval-Enhanced Planning

• REACT-Drive is a trajectory planning framework for work zones that integrates vision-language models with retrieval-augmented generation to overcome recurrent planning failures.
• The framework converts historical failure cases into structured constraint rules and executable Python code to adjust drivable masks and destination points.
• Empirical results on the ROADWork dataset and real-world evaluations demonstrate significant reductions in displacement errors and collision rates compared to baseline methods.

Searching arXiv for the core REACT-Drive paper and a few closely related adjacent works to ground the article in current literature. REACT-Drive is a trajectory planning framework for autonomous driving in work zones that combines Vision-LLMs (VLMs) with Retrieval-Augmented Generation (RAG) to mitigate recurrent planning failures in irregular, temporary, and dynamically changing traffic-control environments. In the formulation presented in “Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Driving” (Liao et al., 3 Oct 2025), REACT-Drive is designed around a specific empirical diagnosis: mainstream VLMs often fail to generate correct trajectories in work zones, and many of these failures recur as recognizable structural patterns. REACT-Drive addresses this by converting prior failure cases into structured constraint rules and executable mitigation code, then retrieving and reusing those artifacts for similar future scenes. This positions it as a retrieval-enhanced, constraint-guided planning layer rather than a purely end-to-end VLM planner (Liao et al., 3 Oct 2025).

1. Definition and empirical motivation

REACT-Drive is introduced in the context of work-zone trajectory planning, where temporary traffic control, irregular layouts, dynamically reconfigured drivable areas, lane shifts, detours, cones, drums, barriers, and construction vehicles make direct VLM planning unreliable (Liao et al., 3 Oct 2025). The motivating claim is explicit: while VLMs have shown promise in autonomous driving, their planning ability in work zones had not been systematically studied before this work, and the resulting failure rate is high.

The paper reports that mainstream VLMs fail to generate correct trajectories in 68.0% of cases (Liao et al., 3 Oct 2025). In the ROADWork evaluation, scenario-level failure rates are reported as 70.37% for GPT4o, 75.86% for Qwen2.5-VL, 80.00% for Gemini2.5, 81.93% for SimLingo, 76.63% for RoboTron-Drive, and 77.23% for DriveLM (Liao et al., 3 Oct 2025). A case is defined as a failure if ADE > 50 px and FDE > 100 px, and a scenario is defined as a failure if > 50% of its cases fail (Liao et al., 3 Oct 2025). The paper further highlights a model-specific comparison for Qwen2.5-VL: FDE = 285.90 on ROADWork work-zone planning versus 106.38 on normal commonsense driving cases in nuScenes, which localizes the problem to work-zone reasoning rather than general trajectory prediction (Liao et al., 3 Oct 2025).

This suggests that REACT-Drive is motivated less by generic VLM weakness than by a long-tail failure mode in which work-zone semantics systematically violate the assumptions under which direct multimodal planning remains stable. The framework therefore treats prior failures as reusable planning knowledge rather than isolated errors.

2. Failure-pattern mining and scene-graph analysis

A central component of REACT-Drive is its failure analysis pipeline, which transforms failed VLM outputs into a structured inventory of recurrent abnormal patterns (Liao et al., 3 Oct 2025). The procedure combines scene graph construction, candidate subgraph extraction, candidate merging, clustering, and human verification.

Each frame is encoded as a directed scene graph

$G = (V, E),$

where nodes correspond to work-zone-related entities and auxiliary structural nodes, and edges encode directional, proximity, and lane-membership relations (Liao et al., 3 Oct 2025). The auxiliary node set is

$$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$

while the work-zone category set includes drum, cone, work vehicle, ttc sign, fence, barricade, barrier, worker, tubular marker, and vertical panel (Liao et al., 3 Oct 2025). A YOLOv12 detector fine-tuned on ROADWork produces object instances

$\{(v_i,\, b_i)\},$

with class label $v_i$ and bounding box $b_i=[x_i,\,y_i,\,w_i,\,h_i]$ (Liao et al., 3 Oct 2025).

Spatial relations are computed after converting image coordinates into an ego-centered metric plane:

$$(x_i,\,y_i)=\left(\frac{x_i^{\text{px}}+\tfrac{w_i^{\text{px}}}{2}-c_x}{\mathrm{PPM}},\ \frac{c_y-\big(y_i^{\text{px}}+\tfrac{h_i^{\text{px}}}{2}\big)}{\mathrm{PPM}}\right),$$

with the $y$-axis pointing forward (Liao et al., 3 Oct 2025). Directional relations are defined through

$$\Delta\mathbf{p}_{i\to j}=\mathbf{p}_j-\mathbf{p}_i,\qquad \theta_{i\to j}=\mathrm{wrap}_{(-\pi,\pi]}\!\big(\mathrm{atan2}(\Delta y,\Delta x)-\psi_i\big),$$

yielding $inFrontOf$, $toLeftOf$, and $$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$0 relations by thresholding $$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$1 (Liao et al., 3 Oct 2025). Proximity relations are estimated using MiDaS depth and quantized into near_collision $$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$2 m, super_near $$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$3 m, very_near $$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$4 m, near $$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$5 m, and visible $$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$6 m (Liao et al., 3 Oct 2025).

Candidate abnormal subgraphs are extracted with depth-limited BFS centered on ego:

$$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$7

with $$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$8, and subgraphs with fewer than $$\mathcal{C}_{\mathrm{aux}}=\{ego,\ Left\ Lane,\ Middle\ Lane,\ Right\ Lane,\ Root\ Road\},$$9 nodes are discarded (Liao et al., 3 Oct 2025). These candidates are then merged using a structural signature

$\{(v_i,\, b_i)\},$0

together with threshold gating on average depth and average box-center distance, and relation-preserving directed subgraph isomorphism (Liao et al., 3 Oct 2025).

After clustering with K-means and choosing $\{(v_i,\, b_i)\},$1 by the elbow method on SSE, the authors obtain 10 initial abnormal clusters, then reduce them to 8 common failure patterns through human combination and semantic summarization (Liao et al., 3 Oct 2025). The final patterns are:

• P1: Dense drums or cones on sidewalk
• P2: Encounter dead end road
• P3: Interference from large work vehicles
• P4: Lane borrowing through work zone
• P5: Lane shift across work zones
• P6: Overreaction to signs
• P7: Accelerate through the exit in the work zone
• P8: Turning through work zone (Liao et al., 3 Oct 2025)

Each pattern is associated with one or more rule needs such as “follow the lane center,” “turn to avoid work zone,” “detour to bypass the work zone,” “return the origin lane after bypassing the work zone,” “cross the work zone,” “return center line after crossing work zone,” “follow the sign,” and “follow the front car” (Liao et al., 3 Oct 2025). A plausible implication is that REACT-Drive treats work-zone planning as a finite set of recurrent structural subproblems rather than a single undifferentiated multimodal inference task.

3. Rule induction and executable mitigation code

REACT-Drive is a two-stage framework. In the offline stage, prior failure cases are converted into a failure-case mitigation code database; in the online stage, similar cases are retrieved and their stored mitigation logic is executed for the new scene (Liao et al., 3 Oct 2025).

In the offline stage, the system uses a VLM to infer applicable work-zone constraints from a failure case and to generate executable Python code implementing those constraints (Liao et al., 3 Oct 2025). The input includes a front-view image, overlapping failure trajectories, a visible failure destination point, and a ground-truth annotated image (Liao et al., 3 Oct 2025). The paper states that this uses 8 predefined work-zone constraint templates based on work-zone traffic regulations from PennDOT guidance, although the full list of all eight fields is not fully enumerated in the provided text (Liao et al., 3 Oct 2025).

Rules are represented in a JSON-like schema with fields such as:

$b_i=[x_i,\,y_i,\,w_i,\,h_i]$0

where "no_cross_workzone" specifies whether crossing is allowed, "detour_side" is "left", "right", or "none", and "return_to_original_lane_after_workzone" is "True" or "False" (Liao et al., 3 Oct 2025).

The generated mitigation code must define exactly two functions:

$b_i=[x_i,\,y_i,\,w_i,\,h_i]$1

The paper also describes these functions as segment_drivable_mask and plan_destination; the spelling inconsistency between driveable and drivable appears in the source, but the intended semantics are stable (Liao et al., 3 Oct 2025). The first function adjusts the drivable road mask according to work-zone constraints and blocks undrivable regions; the second chooses a destination point consistent with those constraints (Liao et al., 3 Oct 2025).

The prompt template directs the VLM to output both completed constraints in JSON and executable Python code with no placeholders and no extra commentary (Liao et al., 3 Oct 2025). This is followed by self-verification using two explicit checks. First, the Drivability Constraint requires the predicted target $\{(v_i,\, b_i)\},$2 to satisfy

$\{(v_i,\, b_i)\},$3

where $\{(v_i,\, b_i)\},$4 is the Euclidean distance transform to the drivable set $\{(v_i,\, b_i)\},$5 (Liao et al., 3 Oct 2025). Second, the Destination Constraint requires the predicted destination to remain sufficiently close to the ground-truth target in pixel space:

$\{(v_i,\, b_i)\},$6

If validation fails, the system retries with actionable feedback; once it passes or reaches maximum retries, the rules and code are stored with the validation record (Liao et al., 3 Oct 2025).

This suggests that REACT-Drive does not merely retrieve prior trajectories. It retrieves a verified programmatic transformation of the planning problem.

4. Retrieval-augmented planning and online inference

The online stage of REACT-Drive uses RAG-style retrieval over the failure-case mitigation code database (Liao et al., 3 Oct 2025). For a new scenario, the system extracts multimodal scene features from images, semantic annotations, and temporal context, then retrieves similar prior cases and executes their stored mitigation code (Liao et al., 3 Oct 2025). If a highly similar case is not found, the system falls back to the base VLM reasoning pipeline (Liao et al., 3 Oct 2025).

The paper provides an explicit triplet-based similarity formulation in the pattern-analysis component. For an abnormal prototype $\{(v_i,\, b_i)\},$7 and a retrieved candidate subgraph $\{(v_i,\, b_i)\},$8, structural similarity is

$\{(v_i,\, b_i)\},$9

depth similarity is

$v_i$0

and bounding-box similarity is

$v_i$1

A priority rule labels a prototype non-independent when the minimum of the three similarities is at least $v_i$2 (Liao et al., 3 Oct 2025).

While the paper does not fully specify the production retrieval stack—such as ANN backend, exact online Top-$v_i$3, or multi-retrieval fusion policy—it is clear about what is retrieved and how it is used. The retrieved artifact is the associated stored mitigation code, which directly modifies the drivable mask and destination selection for the new scenario (Liao et al., 3 Oct 2025). In effect, retrieval influences planning by selecting which validated constraints and executable planning transformation should govern the current scene.

This distinguishes REACT-Drive from a generic retrieval-only memory system. Its retrieved knowledge is operational, not merely descriptive.

5. Trajectory planning formulation and empirical performance

After retrieval, REACT-Drive performs planning in three steps: retrieve a similar prior failure case, execute its mitigation code to modify the drivable road mask and compute a destination point, and generate a smoothed trajectory of 20 discrete points from the start to that destination (Liao et al., 3 Oct 2025). The paper does not provide a full optimization objective for the final smoothing stage, nor a controller-level motion model; the planning description is procedural rather than variational.

Trajectory quality is evaluated with:

$v_i$4

$v_i$5

$v_i$6

where $v_i$7 is the predicted position, $v_i$8 is the ground-truth position, and $v_i$9 is the collision indicator (Liao et al., 3 Oct 2025). Because accurate depth is not available in ROADWork, ADE and FDE are measured in pixel space (Liao et al., 3 Oct 2025).

The main dataset is ROADWork, described as the first dataset designed for driving through work zones, containing 1186 scenarios with multi-granularity work-zone annotations and drivable trajectory annotations from real driving videos (Liao et al., 3 Oct 2025). Baselines include GPT4o, Gemini-2.5, Qwen2.5-72B-VL, DriveLM, SimLingo, and RoboTron-Drive; additional mitigation baselines are fine-tuned Qwen2.5-72B-VL using QLoRA and a self-prediction variant without REACT’s constraint rules (Liao et al., 3 Oct 2025).

Average results on ROADWork are:

Method	ADE	FDE	CR
fine-tune VLM	207.97	384.31	0.11
VLM (Self) w/o Constraint Rules	201.09	350.75	0.03
REACT	54.73	101.64	0.04

These values ground the paper’s claim that REACT-Drive yields roughly a 3× reduction in average displacement error relative to VLM baselines under evaluation with Qwen2.5-VL (Liao et al., 3 Oct 2025). Pattern-specific results show especially strong gains on P1, P3, P6, and P7, where REACT reduces CR to 0.00 (Liao et al., 3 Oct 2025). For P2, the dead-end-road pattern, REACT reduces FDE to 110.44 versus 532.96 for the fine-tuned baseline (Liao et al., 3 Oct 2025).

The framework also transfers across VLM backbones. With REACT, GPT4o achieves ADE 54.73, FDE 101.64, CR 0.04, while Qwen2.5 achieves ADE 86.46, FDE 124.67, CR 0.07 (Liao et al., 3 Oct 2025). On efficiency, REACT reports 0.58 s inference time, compared with 17.90 s for the fine-tuning-based mitigation method (Liao et al., 3 Oct 2025). The paper further notes approximate latencies of around 5 s for GPT4o and 3 s for RoboTron-Drive, making REACT the fastest reported method in its comparison (Liao et al., 3 Oct 2025).

This suggests that REACT-Drive’s practical advantage is not only better geometric planning accuracy but also lower online computational burden, because its most expensive reasoning step—constraint and code generation—has been shifted offline into the database-construction phase.

6. Real-world evaluation, scope, and relation to adjacent paradigms

REACT-Drive is additionally evaluated on 15 real-world work zone scenarios collected from local driving environments different from ROADWork, totaling 100 images (Liao et al., 3 Oct 2025). Ground-truth trajectories are jointly annotated by two authors (Liao et al., 3 Oct 2025). In this physical-world evaluation, the reported results are:

Method	ADE	FDE	CR
GPT4o	127.68	225.06	0.03
DriveLM	134.53	247.28	0.06
SimLingo	182.61	271.56	0.03
RoboTron-Drive	143.48	253.19	0.05
REACT	64.28	106.47	0.00

The real-world transfer result of FDE 106.47 and CR 0.00 is one of the strongest empirical arguments for the method’s practicality (Liao et al., 3 Oct 2025). At the same time, the appendix explicitly states a limitation: despite using physical-world data collected from a real vehicle environment, the framework was not deployed on a real autonomous vehicle in active work zones due to safety concerns (Liao et al., 3 Oct 2025). The most precise reading is therefore that REACT-Drive has physical-world evaluation but not a closed-loop real-vehicle deployment.

The paper identifies several limitations. It does not systematically study extreme-weather or nighttime work zones, and evaluation is limited to ROADWork plus the self-collected physical data (Liao et al., 3 Oct 2025). The final planning and smoothing module is described procedurally rather than through a fully specified optimization or controller formulation. Performance also depends on pattern coverage: when training with only pattern 1 coverage, the system reports ADE 341.57, FDE 703.26, CR 0.14, whereas with 7 out of 8 patterns covered, FDE improves to 275.07 (Liao et al., 3 Oct 2025). This indicates that retrieval memory diversity materially affects generalization.

Conceptually, REACT-Drive belongs to a broader class of systems that place language reasoning above structured tool or program interfaces rather than relying on monolithic multimodal end-to-end inference. This suggests an architectural kinship with multimodal reasoning-and-tool-use systems such as MM-REACT, which likewise combine a LLM with external experts instead of relying solely on joint multimodal finetuning (Yang et al., 2023). A plausible implication is that REACT-Drive can be understood as a work-zone-specific instantiation of that broader modular pattern: identify what prior structured knowledge is missing, retrieve or execute the corresponding external procedure, and only then synthesize the final answer or plan.

In summary, REACT-Drive is best understood as a retrieval-enhanced, constraint-verified trajectory planning framework for work zones. Its distinctive contribution is to transform prior VLM failures into validated rules and executable code, then reuse those artifacts online to guide new planning problems. Within the scope studied, it demonstrates that failure memory, when lifted into programmatic constraints rather than stored as raw trajectories alone, can materially improve both robustness and runtime in one of autonomous driving’s most difficult long-tail settings (Liao et al., 3 Oct 2025).