ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generation

Abstract: We present ReinDriveGen, a framework that enables full controllability over dynamic driving scenes, allowing users to freely edit actor trajectories to simulate safety-critical corner cases such as front-vehicle collisions, drifting cars, vehicles spinning out of control, pedestrians jaywalking, and cyclists cutting across lanes. Our approach constructs a dynamic 3D point cloud scene from multi-frame LiDAR data, introduces a vehicle completion module to reconstruct full 360° geometry from partial observations, and renders the edited scene into 2D condition images that guide a video diffusion model to synthesize realistic driving videos. Since such edited scenarios inevitably fall outside the training distribution, we further propose an RL-based post-training strategy with a pairwise preference model and a pairwise reward mechanism, enabling robust quality improvement under out-of-distribution conditions without ground-truth supervision. Extensive experiments demonstrate that ReinDriveGen outperforms existing approaches on edited driving scenarios and achieves state-of-the-art results on novel ego viewpoint synthesis.

• The paper presents a novel RL-based post-training paradigm that refines out-of-distribution driving scene generation.
• It combines explicit 3D point cloud editing with vehicle geometry completion and a diffusion simulator to enhance photorealism.
• Empirical results show significant improvements in FID and IoU metrics, validating robustness in safety-critical scenarios.

ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generation

Introduction

The synthesis and controllable manipulation of photorealistic driving scenes are critical for the development, evaluation, and validation of autonomous driving systems, particularly for addressing safety-critical scenarios rarely represented in real-world datasets. Existing video generation and reconstruction-based simulators are fundamentally limited by their reliance on supervised learning over recorded data, which results in suboptimal performance and evident artifacts when presented with edited or out-of-distribution (OOD) scenes, such as counterfactual vehicle maneuvers, rare collisions, or novel trajectories.

ReinDriveGen introduces a unified framework to bridge this distribution gap through explicit editing in 3D point cloud space, vehicle geometry completion, and RL-based post-training for robust OOD generalization. The core innovations address both geometric and photorealistic fidelity, as well as reward modeling for reinforcement learning in domains where ground-truth supervision is unattainable.

Figure 1: ReinDriveGen enables photorealistic generation of OOD driving scenarios including safety-critical edits such as in-place spinning and complex collisions using RL-based post-training.

Methodology

Point Cloud-Conditioned Video Diffusion Simulator

ReinDriveGen’s foundational component is a hybrid pipeline that utilizes aggregated multi-frame LiDAR point clouds as a manipulable 3D representation. For dynamic actors (vehicles, pedestrians, cyclists), points are transformed and accumulated using annotated bounding boxes, while the static background is aggregated globally over entire sequences. The spatial incompleteness resulting from typical occlusions and limited sensor coverage is addressed by an AdaPoinTr-based 3D vehicle completion module, which is fine-tuned on synthetic and real data to reconstruct full $360^\circ$ geometry.

The completed point clouds are rendered as pseudo-images to serve as conditional input for a VACE-based video diffusion model. This architecture leverages the Video Condition Unit (VCU) for compositional integration of canonical reference frames and structural pseudo-images, ensuring that both textural and geometric consistency are preserved across multiple (potentially recursive) editing operations.

Figure 2: The pipeline integrates 3D point cloud editing and completion with RL-based post-training using a pairwise reward mechanism for contrastive supervision.

RL-Based Post-Training for OOD Quality

Traditional SFT fails when the edited configurations at test time deviate from the training set, inducing catastrophic degradation in generated vehicle realism (i.e., geometry, illumination, and texturing failures). ReinDriveGen extends DiffusionNFT to high-dimensional video generation, allowing policy optimization through efficient flow matching in the absence of paired supervision.

The central RL innovation is a pairwise reward aggregation procedure. A custom pairwise preference model (frozen DINOv3 ViT-H+ backbone plus MLP head) is trained on automatically constructed relative quality pairs, with degradation controlled via pseudo-image subsampling and adversarial augmentation strategies. During RL, for each OOD edit condition, $N$ sampled candidate videos are ranked via dense pairwise comparison over localized vehicle crops. Per-sample rewards are calculated by normalized win rate, which robustly discriminates fine-grained differences and is less prone to gradient instability and reward hacking observed in pointwise scoring approaches.

Figure 4: RL-based post-training results in continuous improvements to vehicle geometry, texture, and lighting, highlighted by increasing training steps across OOD scenarios.

Empirical Results

Evaluation is conducted across canonical and OOD driving scene modifications, including novel ego-vehicle viewpoint synthesis (lateral shifts up to 4m) and actor trajectory edits. Metrics for both settings include NTA-IoU, NTL-IoU, FID, and VBench-derived quantities (image quality, background consistency, motion smoothness). ReinDriveGen consistently exhibits numerically superior results, with SOTA FID ($51.99$ for lane-change scenarios), NTA-IoU ($0.549$), and clear improvements in photorealism and geometric integrity under aggressive actor edits.

Figure 5: Lane-change scenario comparison demonstrates ReinDriveGen's capability to maintain high spatial and textural fidelity under large lateral shifts.

Figure 6: Edited vehicle trajectories exhibit minimal geometric or shading artifacts with ReinDriveGen, in contrast to prominent failures in previous state-of-the-art methods.

Ablation studies confirm that geometry completion is critical for artifact-free actor manipulation; omission results in severe texture faults and visible geometry holes on occluded or rotated vehicles. Quantitative ablations further demonstrate that the RL post-training is the driving factor for closing the realism gap in rare/complex edits:

• Evaluation score curves show monotonic increases in preference-based quality over RL steps, especially for scenarios such as in-place spinning and vehicle rollovers.
• The pairwise reward model achieves higher test-set accuracy (95% vs. 91%) than the pointwise baseline and stable training dynamics.
• Qualitative comparison indicates suppression of reward hacking artifacts with the pairwise model, yielding more plausible and consistent outputs.

  Figure 8: Comparison of RL post-training with pointwise versus pairwise reward signals highlights the robustness and superiority of the pairwise approach on vehicle realism and coherence.

Theoretical and Practical Implications

ReinDriveGen’s RL-based post-training paradigm establishes a new robust route to generalization beyond coverage of available datasets, a fundamental concern in safety-critical domains such as autonomous driving. The tractable construction of relative preferences for policy optimization is domain-agnostic and can be generalized to any generative framework where absolute quality labels are either unattainable or uninformative.

Additionally, the modular decomposition of geometric manipulation and texture synthesis supports compositional and recursive editing pipelines without drift or degeneration in background context. In practice, this approach stands to enhance data augmentation for edge-case discovery, enable more comprehensive policy evaluation in closed-loop driving simulators, and support synthetic dataset scaling without dependence on rare event collection.

Conclusion

ReinDriveGen advances the state of art in controllable driving scene generation by integrating explicit 3D condition editing, learned completion, and RL-based post-training with robust pairwise reward modeling. It achieves leading perceptual and geometric quality in scenarios well beyond the training distribution, particularly for safety-critical edits where baseline models fail. The framework’s broader applicability suggests a scalable way forward for closing the distribution gap in generative modeling pipelines, with relevance to autonomous driving simulation, synthetic data generation, and RL from preference in domains without reference ground-truth (2604.01129).