RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework

Abstract: High-level autonomous driving requires motion planners capable of modeling multimodal future uncertainties while remaining robust in closed-loop interactions. Although diffusion-based planners are effective at modeling complex trajectory distributions, they often suffer from stochastic instabilities and the lack of corrective negative feedback when trained purely with imitation learning. To address these issues, we propose RAD-2, a unified generator-discriminator framework for closed-loop planning. Specifically, a diffusion-based generator is used to produce diverse trajectory candidates, while an RL-optimized discriminator reranks these candidates according to their long-term driving quality. This decoupled design avoids directly applying sparse scalar rewards to the full high-dimensional trajectory space, thereby improving optimization stability. To further enhance reinforcement learning, we introduce Temporally Consistent Group Relative Policy Optimization, which exploits temporal coherence to alleviate the credit assignment problem. In addition, we propose On-policy Generator Optimization, which converts closed-loop feedback into structured longitudinal optimization signals and progressively shifts the generator toward high-reward trajectory manifolds. To support efficient large-scale training, we introduce BEV-Warp, a high-throughput simulation environment that performs closed-loop evaluation directly in Bird's-Eye View feature space via spatial warping. RAD-2 reduces the collision rate by 56% compared with strong diffusion-based planners. Real-world deployment further demonstrates improved perceived safety and driving smoothness in complex urban traffic.

• The paper introduces a joint generator-discriminator framework that integrates diffusion-based planning with reinforcement learning for robust autonomous driving.
• It leverages BEV-Warp for efficient closed-loop simulation, reducing computational costs and maintaining spatial accuracy during large ego-motion.
• Experimental results show a significant reduction in collision rates and improved efficiency, establishing a new baseline for multimodal trajectory optimization.

RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework

Introduction

RAD-2 introduces a unified architecture for motion planning in autonomous driving, leveraging a generator-discriminator framework coupled with scalable reinforcement learning (RL). Prior approaches—regression-based, selection-based, and diffusion-based planners—struggle either with unimodal predictions, limited candidate diversity, or instability from open-loop training. Diffusion-based imitation learning (IL) planners efficiently model multimodal trajectory distributions, but lack robustness due to stochastic instability and the absence of corrective feedback. RAD-2 addresses these deficits by integrating a diffusion-based trajectory generator with an RL-trained discriminator, setting a new standard in closed-loop, high-dimensional trajectory optimization for autonomous driving.

Figure 1: RAD-2 stabilizes RL optimization by projecting high-dimensional trajectories into low-dimensional score and longitudinal spaces and enables efficient closed-loop training with feature-level simulation.

Generator-Discriminator Architecture

RAD-2 decomposes planning into two decoupled components:

• Diffusion-Based Generator ($\mathcal{G}$): Conditions on BEV-encoded scene features to produce a diverse set of trajectory candidates via iterative denoising. Each candidate is a continuous trajectory over the planning horizon, exploiting multimodality inherent in complex traffic environments.
• RL-Optimized Discriminator ($\mathcal{D}$): Scores and reranks candidate trajectories using a Transformer encoder architecture, aggregating context from map, dynamic agents, and navigation instructions via cross-attention. The low-dimensional output aligns with reward signals, circumventing credit assignment issues endemic to high-dimensional action spaces.

The framework defines a joint policy distribution $\Pi_{\theta,\phi}(\tau|o)$, where generator and discriminator are iteratively co-optimized to approach the risk-neutral high-efficiency driving distribution.

Figure 2: Comparison of multimodal planning paradigms: (a) Vocabulary-based scoring, (b) Diffusion generation, (c) RAD-2’s joint generator-discriminator architecture.

BEV-Warp: Efficient Closed-Loop Simulation

RAD-2 introduces BEV-Warp, a novel simulation platform for efficient closed-loop training. BEV-Warp operates directly in the BEV feature space, applying spatial transformations to align logged scenes with simulated ego vehicle poses, eliminating the computational cost of image-level rendering and facilitating high-throughput data collection. BEV-Warp leverages spatial equivariance, ensuring that warping accurately maintains semantic alignment of scene elements under significant ego-motion.

Figure 3: BEV-Warp recursively warps spatial features via pose-determined matrices, synthesizing observations for closed-loop interaction without expensive rendering.

Figure 4: Warped BEV features maintain spatial correctness, as verified by consistent downstream perception outputs over large translations and rotations.

Reinforcement Learning and Policy Optimization

RAD-2’s RL strategy is underpinned by two critical algorithmic innovations:

Temporally Consistent Group Relative Policy Optimization (TC-GRPO):

The generator’s commitment to a selected trajectory is enforced over a trajectory latch horizon $H_{\text{reuse}}$, thereby preserving behavioral coherence and improving the fidelity of credit assignment. The RL discriminator is optimized by attributing sequence-level rewards (safety and efficiency) to these persistent trajectory intervals, with standardized groupwise advantages used to stabilize the policy gradient.

On-Policy Generator Optimization (OGO):

To adapt the generator $\mathcal{G}_\theta$ toward regions of higher expected reward, OGO leverages structured feedback from closed-loop rollouts. Trajectories with low reward are optimized over their longitudinal components—through acceleration or deceleration—generating new on-policy targets for generator fine-tuning. This gradual distribution shift increases feasible behavior space explored by the policy.

Figure 5: RAD-2’s pipeline synergizes a pre-trained diffusion generator and a RL-trained discriminator in a three-stage optimization loop leveraging BEV-Warp.

Replay Buffer and Joint Optimization

Closed-loop policy rollouts—including both safety-critical and efficiency-oriented scenarios—are managed in a FIFO replay buffer, ensuring diverse and balanced training data. Optimization alternates asynchronously: the discriminator is updated upon each new batch, while the generator is refined after a full buffer circulation, maintaining an approximately 8:1 optimization frequency in favor of the discriminator.

Figure 6: Rollout and optimization workflow, with separate triggers and balance between trajectory generator and discriminator updates in the replay buffer.

Experimental Evaluation

Closed-Loop Performance:

RAD-2 leads in all closed-loop metrics. In safety-oriented scenarios within BEV-Warp, the collision rate (CR) drops from 0.533 (ResAD) to 0.234 and Safety@1 rises from 0.418 to 0.730. In efficiency-oriented contexts, navigation metrics approach ideal (EP-Mean of 0.988, [email protected] of 0.736).

3DGS Evaluation:

Photorealistic simulation (3DGS) corroborates findings: RAD-2 achieves the lowest CR (0.250) and the highest safety margins (Safety@1: 0.723, Safety@2: 0.644), exceeding all prior IL/RL and hybrid architectures.

Open-Loop Metrics:

RAD-2 attains state-of-the-art ADE (0.208m) and FDE (0.553m), with minimal collision risk in predictive settings.

Figure 7: Policy scaling—synergistic joint optimization achieves steeper and higher reward scaling versus discriminator-only or sequential pipelines.

Training Pipeline Analysis and Ablation

Ablation demonstrates the necessity of coordinated generator-discriminator optimization. Discriminator-only or generator-only fine-tuning underperforms joint training. Key findings include:

• Replay Buffer Design: Reward-variance-based clip filtering prioritizes informative data, stabilizing RL and boosting efficiency ([email protected] +0.066).
• Trajectory Latch Horizon: Optimal at $H_{\text{reuse}}=8$, balancing temporal coherence and reactivity.
• Entropy Regularization: Adaptive inclusion of entropy term $\mathcal{H}$ in the RL loss prevents policy collapse, substantiated by improved stability and safety (CR down to 0.234).
• Scenario Composition: Integrated training on diverse scenario types yields generalized, robust policies, outperforming single-objective approaches which induce significant efficiency-safety trade-offs.

  Figure 8: Clip filtering significantly stabilizes RL training, leading to more consistent performance improvements.

  

  Figure 9: Adaptive entropy regularization maintains trajectory score diversity, preventing policy collapse.

  

  Figure 10: Mixed-objective scenario composition yields balanced policies across safety and efficiency; single-objective biases induce severe performance loss on complementary goals.

Qualitative Results

RAD-2 demonstrates superior safety in critical interactions, executing preemptive maneuvers to avoid collisions, and delivers improved tactical efficiency by proactively performing lane changes rather than exhibiting conservative hesitation in traffic.

Figure 11: RAD-2 avoids collisions through proactive deceleration, while the baseline fails under the same conditions.

Figure 12: RAD-2 achieves superior progression via lane change, whereas the baseline remains stuck in slow traffic.

Implications and Future Directions

RAD-2 sets a new baseline for scalable RL-driven planning in autonomous driving, advancing closed-loop interaction by resolving high-dimensional credit assignment via generator-discriminator decoupling and temporal optimization. Practically, it achieves low collision rates and high efficiency, with robust policy generalization across varying simulators.

Theoretically, RAD-2 validates the impact of RL-driven preference learning in complex continuous control, opening avenues for further integration with generative world models to mitigate sim-to-real gaps and enable large-scale scenario diversity. Challenges remain for architectures not leveraging BEV-centric representations, as BEV-Warp’s translation efficiency is currently representation-specific. Future research will likely target generalized feature transformation and bridging RL optimization with learned world models.

Conclusion

RAD-2 introduces an advanced generator-discriminator RL framework, addressing the instability and limited feedback inherent in diffusion-based imitation planners. Through a combination of TC-GRPO, OGO, and BEV-Warp simulation, RAD-2 achieves substantial gains in both safety and efficiency, establishing a scalable paradigm for closed-loop autonomous driving policy optimization (2604.15308).