Drive-3DAug: 3D Augmentation for Autonomous Driving

• Drive-3DAug is a 3D data augmentation approach that reconstructs scene geometry using voxel-based NeRFs, yielding detection accuracy gains of up to 1.7% on benchmarks.
• The method decomposes scenes into background and foreground objects, applies controlled 3D jitter and collision checks, and renders geometrically consistent images.
• Its reusable NeRF assets enable cross-detector training and versatile applications in 3D perception tasks, addressing limitations of 2D-only augmentation.

Searching arXiv for papers on Drive-3DAug and closely related driving-scene 3D augmentation work. arxiv_search(query="Drive-3DAug autonomous driving 3D data augmentation camera NeRF", max_results=10) Drive-3DAug is a 3D data augmentation approach for camera-based autonomous driving that augments driving scenes in 3D space rather than only in the 2D image plane. In its original formulation, it reconstructs 3D models of backgrounds and foreground objects with voxel-based Neural Radiance Fields (NeRF), inserts reconstructed objects into physically valid regions with controlled translation and rotation, and renders the composed scene back to the camera view for detector training (Tong et al., 2023). The method was introduced for camera-only monocular 3D detection on Waymo and nuScenes, where it yielded gains of 1.7% and 1.4% in detection accuracy, respectively, and produced reusable digital driving assets for other detectors and 3D perception tasks (Tong et al., 2023).

1. Problem formulation and scope

Drive-3DAug addresses a specific limitation of prior augmentation methods for autonomous driving cameras: existing methods were confined to the 2D image plane and therefore did not optimally increase data diversity in 3D real-world scenarios (Tong et al., 2023). The motivating premise is that driving scenes are highly diverse and complicated, making exhaustive collection infeasible with human effort alone. The framework therefore shifts augmentation from image-plane composition to explicit 3D scene manipulation.

In the original method, the input consists of monocular video, 3D box annotations, and optional LiDAR or estimated depth. Each scene is decomposed into background and foreground objects. Backgrounds are formed by masking out all moving objects via an instance-segmentation network, specifically SOLO v2. Foreground objects, including vehicles, pedestrians, and cyclists, are extracted with 2D masks and associated to 3D boxes via mask-box IoU, while only fully visible instances are retained (Tong et al., 2023).

A common misconception is to treat Drive-3DAug as a variant of 2D copy-paste. That characterization is inaccurate for the original method. Its central operations are 3D model construction, 3D object placement with geometric constraints, and view-consistent rendering back to 2D, rather than direct image-plane pasting (Tong et al., 2023).

2. Core pipeline: 3D model construction and scene synthesis

The original Drive-3DAug pipeline has two stages: 3D model construction and scene augmentation with rendering (Tong et al., 2023).

Stage	Key operations	Output
3D model construction	Scene decomposition; voxel-based NeRF learning for background and objects	Background and foreground NeRF assets
Scene augmentation and rendering	Background/object sampling; valid-region placement; 3D jitter; collision checks; rendering from camera pose	Augmented 2D training images

For 3D model construction, Drive-3DAug follows a voxel-based NeRF representation based on DVGO. It learns a density grid $$V_{\mathrm{density}} \in \mathbb{R}^{R \times R \times R}$$ and a feature grid $$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$. At a query point $x$ and view $d$, density and color are given by

$$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$

Volume rendering along a ray $r(t)=r_0+td$ is

$$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$

with

$$T(t)=\exp\!\left(-\int_{t_1}^{t}\sigma(r(s))\,ds\right).$$

The base training objective for background and object NeRFs is

$L = L_{\mathrm{color}} + L_{\mathrm{depth}},$

where

$$L_{\mathrm{color}}=\sum_{r\in\mathcal R}\|\hat C(r)-C(r)\|_2^2, \qquad L_{\mathrm{depth}}=\ell_1(\hat D(r),D(r)).$$

Here $$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$0 is supervised by LiDAR or structure-from-motion depth (Tong et al., 2023).

In scene augmentation, the method randomly samples one background NeRF and $$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$1–$$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$2 foreground NeRFs. Each inserted object can undergo 3D jitter with translation $$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$3 and rotation $$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$4. Two regimes are specified. In the “w/o RT” regime, $$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$5, $$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$6, $$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$7, with 1–2 cars per background. In the “w/ RT” regime, $$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$8 m, $$V_{\mathrm{color}} \in \mathbb{R}^{R \times R \times R \times C}$$9 m, and $x$0 (Tong et al., 2023).

The final output is obtained by rendering the composed 3D scene from the camera pose to produce a 2D training image. This preserves geometric consistency between object placement and camera projection in a way unavailable to purely image-plane augmentation (Tong et al., 2023).

3. Geometric validity, rectified loss, and symmetry-aware training

A defining component of Drive-3DAug is its explicit handling of physical plausibility in 3D placement. Valid-region identification is performed in bird’s-eye view. The ground plane is discretized into pillars of size $x$1 m $x$2 $x$3 m. For each pillar $x$4, let $x$5 denote the densities of voxels along the vertical axis. A pillar is marked valid if

$x$6

with $x$7 and $x$8 (Tong et al., 2023). This valid-region rule is designed to ensure physical plausibility, such as placing cars on roads rather than in occupied or implausible volumes. Collision checks with existing objects are then performed via 3D-IoU (Tong et al., 2023).

Drive-3DAug also modifies the original NeRF with a geometric rectified loss to suppress mask-edge artifacts in object NeRFs caused by imperfect instance masks. The probability that a ray $x$9 hits the object within the depth interval $d$0 of the object’s 3D box is

$d$1

The geometric rectified loss is

$d$2

The full object-model loss becomes

$d$3

This loss encourages density concentration inside the true object silhouette rather than around erroneous mask boundaries (Tong et al., 2023).

The second NeRF refinement is a symmetric-aware training strategy. Because many driving-scene objects, including cars and trucks, exhibit approximate left-right symmetry, the method introduces a virtual symmetric ray for each real camera ray through an object. If a ray direction in the object’s local frame is $d$4, the reflected ray has direction $d$5, and both rays are trained to predict the same color:

$d$6

According to the method description, this doubles the effective view coverage and permits rendering at larger yaw-rotation angles with fewer artifacts (Tong et al., 2023).

4. Assets, training protocol, and detector integration

The trained NeRFs are treated as reusable digital driving assets. Once trained, each background or object NeRF can be stored and reused to augment different detectors such as FCOS3D and SMOKE without retuning the NeRFs, and can serve downstream monocular 3D-perception tasks including detection, tracking, and BEV segmentation (Tong et al., 2023). The method description also notes that cross-dataset scene composition is possible, for example placing nuScenes cars on Waymo backgrounds (Tong et al., 2023).

The asset-construction protocol is explicit. For each dataset, 100 “sunny” scenes are selected to build the NeRF assets. Backgrounds use 30–40 consecutive frames spanning approximately 100–200 m, with a $d$7 grid, voxel size $d$8–$d$9 m, and 40k training iterations. Objects use 20–60 frames, about 1000 grid points, and the same voxel size. Runtime is reported as approximately 0.5 h for a background or a few minutes for an object on a single V100 (Tong et al., 2023).

The downstream detector-training setup is likewise specified. For FCOS3D, the backbone is ResNet-101-DCN, trained for 24 epochs with SGD at learning rate $$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$0, step decay at epochs 20 and 23, momentum $$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$1, weight decay $$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$2, and 1k-iteration warmup. For SMOKE, the backbone is DLA-34 with Adam at learning rate $$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$3 and the same schedule. The batch size is 16, and Waymo is sampled every 3 frames for efficiency. For each background model, 12 new images are generated offline and reused across experiments (Tong et al., 2023).

This design makes the augmentation pipeline modular: NeRF asset generation is decoupled from detector optimization. A plausible implication is that the method was intended not merely as an augmentation primitive for one benchmark configuration, but as a reusable asset-generation layer within a broader monocular 3D perception stack.

5. Empirical evaluation on Waymo and nuScenes

Drive-3DAug was evaluated on the camera-only monocular 3D detection task on Waymo and nuScenes (Tong et al., 2023). For Waymo Open Dataset, the reported split is 798 train and 202 val front-camera scenes at resolution $$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$4, with LET-AP, LET-APH, and LET-APL as detection metrics. For nuScenes, the split is 700 train and 150 val front-camera scenes at resolution $$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$5, with mAP at 2 m center distance and the error metrics ATE, ASE, and AOE (Tong et al., 2023).

The main quantitative result on the vehicle class for Waymo with FCOS3D is:

• Baseline LET-AP $$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$6
• Copy-paste $$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$7
• Drive-3DAug w/ RT $$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$8

This corresponds to a gain of 1.7 points over baseline. LET-APH improves from $$\sigma(x)=\mathrm{inter}(x,V_{\mathrm{density}}), \qquad c(x,d)=\mathrm{MLP}_{\theta}(\mathrm{inter}(x,V_{\mathrm{color}}),x,d).$$9 to $r(t)=r_0+td$0, and LET-APL from $r(t)=r_0+td$1 to $r(t)=r_0+td$2, both by 1.7 points. Rare headings of approximately $r(t)=r_0+td$3 improve by 2.2 points over baseline (Tong et al., 2023).

For nuScenes with FCOS3D, the results are:

• AP $r(t)=r_0+td$4 for baseline
• AP $r(t)=r_0+td$5 with Copy-paste
• AP $r(t)=r_0+td$6 with Drive-3DAug

This is a gain of 1.4 points. Translation error improves from ATE $r(t)=r_0+td$7 to $r(t)=r_0+td$8, and orientation error improves from AOE $r(t)=r_0+td$9 to $$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$0 (Tong et al., 2023).

The ablation study isolates the contributions of the NeRF refinements. On Waymo with FCOS3D and the w/ RT regime, base DVGO with depth supervision only yields LET-AP $$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$1; adding geometric rectified loss raises it to $$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$2; adding symmetric-aware training further raises it to $$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$3. A separate depth-supervision ablation reports LET-AP $$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$4 without depth and $$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$5 with depth, a gain of 0.5 (Tong et al., 2023).

Qualitative corner cases include synthetic occlusions, odd orientations, cars on slopes, and cross-dataset composites. These cases clarify the intended operating regime: augmentation is meant to expose detectors to geometric configurations that are uncommon or difficult to capture densely in real fleets (Tong et al., 2023).

6. Terminological extension and relation to later work

The designation “Drive-3DAug” primarily refers to the NeRF-based camera augmentation framework introduced in 2023 (Tong et al., 2023). Later work, however, uses the same designation in a broader sense for 3D-aware driving-scene augmentation.

In “DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes,” the editing framework is described as naturally serving as a 3D data-augmentation engine, and the details explicitly refer to this mode as Drive-3DAug (Liang et al., 2024). There, augmentation is driven by diffusion-based video editing rather than reusable NeRF assets. Position control is provided by a depth-aware projection of 3D bounding boxes into a six-channel pose map, and appearance is maintained by a three-tiered scheme consisting of cut-and-paste conditioning, CLIP features, and 3D priors from SV3D (Liang et al., 2024). For downstream detection on nuScenes, pre-training StreamPETR on two 50%-sized augmented subsets yields improvements from real-only performance of mAP $$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$6, mATE $$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$7 m, mAOE $$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$8 rad, NDS $$C(r)=\int_{t_1}^{t_2} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt,$$9 to $$T(t)=\exp\!\left(-\int_{t_1}^{t}\sigma(r(s))\,ds\right).$$0, $$T(t)=\exp\!\left(-\int_{t_1}^{t}\sigma(r(s))\,ds\right).$$1 m, $$T(t)=\exp\!\left(-\int_{t_1}^{t}\sigma(r(s))\,ds\right).$$2 rad, and $$T(t)=\exp\!\left(-\int_{t_1}^{t}\sigma(r(s))\,ds\right).$$3 when both repositioning and replacement augmentations are used (Liang et al., 2024).

A separate implementation-level summary also applies the label “Drive-3DAug” to the viewpoint-augmentation strategy of VR-Drive, an end-to-end driving system that uses feed-forward 3D Gaussian Splatting and online view synthesis for robustness to novel camera viewpoints (Cho et al., 27 Oct 2025). In that formulation, random camera extrinsic offsets are sampled during training, novel views are rendered from predicted 3D Gaussians, and the downstream planning backbone is trained on either original or synthesized inputs with probability 50% (Cho et al., 27 Oct 2025). This usage is methodologically distinct from the original NeRF asset-construction pipeline.

This suggests that “Drive-3DAug” functions in the literature both as the name of a specific NeRF-based method and as a broader label for 3D-aware driving-scene augmentation. The original contribution remains distinguished by three properties: voxel-based NeRF asset construction, physically plausible object insertion in 3D space, and the reuse of learned background and object models across detectors, tasks, and even datasets (Tong et al., 2023).