CorrectAD: Automated Self-Correction for E2E Planning

• CorrectAD is a self-correction framework that addresses long-tail failures in E2E autonomous planning by iteratively diagnosing issues and generating targeted synthetic data.
• It employs a PM-Agent integrating vision-language models and LLMs to analyze failures and formulate precise multimodal data requirements.
• DriveSora, a diffusion-based video generator, creates spatiotemporally consistent multi-view data aligned with structured 3D layouts to reduce collision rates.

CorrectAD is a closed-loop, fully automated self-correction framework designed to improve the robustness of end-to-end (E2E) planners in autonomous driving by addressing the long-tail problem. The system iteratively diagnoses planner failures, formulates precise data requirements, generates high-fidelity synthetic driving data conditioned on structured 3D layouts, and retrains the planner, eliminating the need for manual intervention in rare or previously unmodeled scenarios. CorrectAD is planner-agnostic and combines vision-language/LLM-based analysis (the PM-Agent) with a diffusion-based video “world model” (DriveSora) to systematically reduce collision rates and address safety-critical edge cases (Ma et al., 17 Nov 2025).

1. Motivation: The Long-Tail Problem in End-to-End Autonomous Driving

End-to-end planners for autonomous driving are trained to map raw multi-view driving videos to future motion trajectories. While these models offer advantages over modular approaches—chiefly the avoidance of stepwise error accumulation—they exhibit significant brittleness under long-tail scenarios, including rare objects, severe weather, or dense traffic maneuvers. These scenarios are under-represented or entirely absent from typical training sets, leading to critical failures such as collisions or unanticipated trajectory deviations at deployment. Traditional remedies such as manual data collection are prohibitively expensive and hazardous, while corpus-based retrieval (e.g., AIDE) cannot address truly novel events due to corpus limitations (Ma et al., 17 Nov 2025). CorrectAD is designed to close this failure gap without human-in-the-loop curation or annotation by generating and leveraging synthetic data targeted to failure modes.

2. PM-Agent: Automated Failure Analysis and Data Requirement Generation

The PM-Agent component acts as the system’s diagnosis and specification module, simulating a product manager’s workflow. Its operation involves the following key stages:

• Failure Identification: Given a labeled training dataset $D^{\mathrm{train}}$ of multi-view video and corresponding 3D annotations, failures are quantized as instances where the planned trajectory collides with another actor within a predefined temporal horizon $T_{e2e}$:

$$D^{\mathrm{fail}} = \{(X, Y)\in D^{\mathrm{train}} : \exists t \le T_{e2e},\; \exists j:\| \mathbf{p}_{ego}(t) - \mathbf{p}_{other}^j(t)\| < \epsilon \}$$

• Multi-Round Failure Analysis: A combination of vision-LLMs (VLMs) and LLMs is employed. The analysis is multi-stage: first, a coarse failure category $h^{\mathrm{class}}$ (e.g., “Foreground,” “Background,” “Weather”) is assigned; second, a detailed natural-language description $h^{\mathrm{desc}}$ is generated. Categories are pre-clustered by expert annotation and adaptive clustering.
• Data Requirement Formulation: An LLM synthesizes $h^{\mathrm{class}}$ and $h^{\mathrm{desc}}$ into a concrete requirement $q$. The PM-Agent then retrieves $K$ nearest scene-caption/layout pairs whose scene captions $c_i$ match $q$:

$\hat r = \{(c_i,e_i) \mid s(c_i,q) \geq \delta\}$

This pipeline yields multimodal prompts—scene captions and their corresponding bird’s-eye-view (BEV) layouts—which condition the video generation model.

3. DriveSora: Controlled Diffusion World Model for Data Synthesis

DriveSora is the generative backbone of CorrectAD, tasked with producing spatiotemporally consistent multi-view driving videos tightly aligned with structured layout and textual prompts.

• Multimodal Encoding: Inputs comprise a foreground layout ($e^{\mathrm{fore}}$, denoting vehicle bounding boxes $B$, headings $M$, instance IDs $U$, per-box captions $V$), a background layout ($$e^{\mathrm{back}}\in\mathbb{R}^{H\times W \times 3}$$, i.e., road sketches), and a scene caption ($c$). Layout elements are embedded via MLPs and Fourier features:

$$\begin{align} \beta^{\mathrm{box}} &= \mathrm{MLP}(\mathrm{Fourier}(B) + \mathrm{Fourier}(M) + \mathrm{Fourier}(U) + E_{\mathrm{text}}(V)) \ \beta^{\mathrm{road}} &= E_{\mathrm{image}}(\alpha) \ \beta^{\mathrm{text}} &= E_{\mathrm{text}}(c) \end{align}$$

• Diffusion Process: Built atop the Spatial-Temporal Diffusion Transformer (STDiT), DriveSora operates via DDPM. At each denoising step, layout information is injected via ControlNet-Transformer modules and parameter-free multi-view spatial attention, enabling strict adherence to 3D layout primitives.
• Multi-Conditional Classifier-Free Guidance: Inference leverages joint guidance over text, foreground, and background conditions:

$$\hat\epsilon_\theta = \epsilon_\theta(z,\phi,\phi,\phi) + \lambda_{\text{text}}[\cdots] + \lambda_{\text{back}}[\cdots] + \lambda_{\text{fore}}[\cdots]$$

where $\lambda$ coefficients allow tuning conditional strength per modality.

• Implementation: DriveSora is adapted from OpenSora 1.1 and trained for 30K single-view and 25K multi-view iterations using HybridAdam, 5% probability classifier-free per condition, and an input resolution of $512 \times 512$, frame length $T=16$.

4. Closed-Loop Pipeline and Iterative Self-Correction

CorrectAD executes as an iterative agentic loop (“model $\rightarrow$ failure $\rightarrow$ generate $\rightarrow$ retrain”) and is agnostic to the specific planner architecture.

• Workflow:

1. Start with initial planner $\mathcal{F}$ and dataset $D^{\mathrm{train}}$.
2. Apply $\mathcal{F}$ to $D^{\mathrm{train}}$ to diagnose failures $D^{\mathrm{fail}}$.
3. PM-Agent transforms $D^{\mathrm{fail}}$ to multimodal requirements $R$.
4. DriveSora generates new labeled synthetic data $D^{\mathrm{gen}}$.
5. Augment $D^{\mathrm{train}}$ with $D^{\mathrm{gen}}$ and fine-tune $\mathcal{F}$.
6. Iterate until convergence (i.e., $D^{\mathrm{fail}}$ is empty or metrics stabilize).

• Pseudocode:

for iter in 1…N: D_fail = {} for x_i in D_train: y_pred = F(x_i) if collision(y_pred): D_fail.add(x_i) if D_fail.empty(): break R = PM_Agent(D_fail, D_train) D_gen = set() for r in R: x_gen = DriveSora.generate(r) D_gen.add((x_gen, annotations from r)) D_train.update(D_gen) F = finetune(F, D_train)

5. Experimental Evaluation and Metrics

CorrectAD is empirically validated on nuScenes and a large in-house dataset using multiple planners (e.g., UniAD, VAD). Key metrics include L2 displacement error (meters), collision rate (%), hit rate, and video quality scores (FID, FVD, CLIP, NDS).

Performance Results:

Model	nuScenes Avg L2 (m)	Collision Rate (%)	In-house L2 (m)	In-house Hit Rate
Baseline UniAD	1.03	0.31	0.85	0.77
AIDE	1.02	0.28	0.79	0.78
CorrectAD	0.98	0.19	0.62	0.82

CorrectAD corrects 62.5% of failure cases on nuScenes and 49.8% on the in-house set, reducing collision rates by 39% and 27% respectively. DriveSora-based generation achieves state-of-the-art video quality (e.g., FID 15.08, FVD 94.51), outperforming competing generators such as MagicDrive-V2 and Panacea when controlling for BEV layouts (Ma et al., 17 Nov 2025).

Ablation studies indicate that both DriveSora and PM-Agent are critical for maximal improvement; removing either reduces performance. Iterative application leads to additional decrements in collision rate and error, with convergent narrowing of the distribution gap (quantified by Hellinger distance).

6. System Design Analysis and Broader Implications

CorrectAD introduces a fully automated, LLM/VLM-driven self-correction paradigm for safety-critical long-tail challenges in E2E driving. The PM-Agent’s ability to structure failure analysis as a dialog and to formulate precise multimodal requirements enables high-fidelity, scenario-aligned data synthesis by DriveSora. The joint system, by being model-agnostic, is directly applicable to any E2E planner and is independent of domain-specific heuristics or human intervention.

A plausible implication is that such a closed agentic loop, which proactively diagnoses and corrects model behavior, represents a scalable methodological advance for continuous adaptation to real-world data shifts in safety domains.

Quantitative performance gains (39–27% collision reduction, 62.5–49.8% failure correction) on heterogeneous datasets substantiate the effectiveness of the architecture. DriveSora’s capacity for controlled, layout-conditional synthesis enables finer data distribution alignment with safety-relevant failure cases, which retrieval or generic video generation cannot provide. The PM-Agent’s use of LLMs and VLMs for semantic failure clustering, requirement articulation, and data retrieval demonstrates the utility of large multimodal models in autonomous system engineering.

7. Related Advancements and Connections

CorrectAD is distinct from adversarial detection/correction methods in vision classifiers, such as KL-driven autoencoder approaches (Vacanti et al., 2020), in that it targets structured, high-dimensional temporal sequences and operates in a model-training-loop, not merely at inference for single-task outputs. Unlike approaches limited to data retrieval (e.g., AIDE), CorrectAD directly synthesizes new training data, enforces 3D spatial-temporal alignment, and iteratively closes distributional gaps caused by rare events (Ma et al., 17 Nov 2025).

The system suggests a new direction for agentic, closed-loop adaptation in both autonomous driving and other domains where rare failure cases dominate deployment risk. Continued development could extend this paradigm to higher-dimensional or multi-task world models, and to broad classes of closed-loop AI safety interventions.