nuScenes-SG: Driving Scene Generation & Labeling

• nuScenes-SG is a nuScenes-style scene generation and labeling workflow that reconstructs driving scenes with image-only data for novel-view synthesis and auto label transfer.
• It extends 3D Gaussian Splatting with modules like DEM, OEM, and GPS to effectively model dynamic objects, improve opacity prediction, and manage large-scale scenes.
• The pipeline shows state-of-the-art rendering metrics and boosts detection performance by ensuring accurate pose transformation and annotation consistency.

Searching arXiv for papers relevant to "nuScenes-SG" and the methods named in the provided data. Within the available source material, nuScenes-SG is best understood as an Editor's term for a nuScenes-style driving-scene generation and auto-labeling setting built around large-scale scene reconstruction, novel-view synthesis, and annotation transfer. In that setting, the central problem is not only to render plausible novel views of dynamic driving scenes, but also to generate corresponding annotations that remain usable for downstream 2D and 3D perception. The most explicit formulation of this pipeline in the provided evidence is EGSRAL, a 3D Gaussian Splatting-based renderer and automated labeling system that is designed to work with only training images for reconstruction while transferring annotations from original dataset coordinates to synthesized views (Huo et al., 2024). A plausible implication is that nuScenes-SG is better characterized as a rendering-and-labeling workflow than as a purely visual synthesis benchmark.

1. Conceptual scope

For nuScenes-style applications, the defining contribution described in the source is not just better novel-view synthesis, but an explicit mechanism to transfer labels from the original dataset coordinate system to synthesized views, making generated data usable for 2D/3D detection training (Huo et al., 2024). This establishes a two-part objective: first, reconstruct a large-scale driving scene with sufficient fidelity to support novel-view rendering; second, preserve annotation consistency under pose transformation so that synthesized images can be paired with valid labels.

The pipeline is presented as extending standard 3D Gaussian Splatting (3D GS) in three directions. It introduces enhanced rendering for complex driving scenes, automatic novel-view labeling, and a reconstruction stage that uses no extra supervision beyond images. The source contrasts this with prior driving-scene 3D GS methods that often depend on LiDAR, trajectories, or object-level annotations. The distinction is important: reconstruction is image-only, but auto-labeling still depends on existing annotations that are transformed into the synthesized view. This clarifies a likely misconception. The method does not claim annotation-free supervision for downstream tasks; rather, it claims no extra annotation for the reconstruction stage.

This framing also helps delimit the meaning of “SG” in this context. The source does not define a canonical expansion for the label nuScenes-SG. A cautious reading is that it denotes a nuScenes-like scene generation and label synthesis regime in which rendering quality, coordinate transfer, and detection utility are jointly relevant.

2. Rendering model and 3D Gaussian Splatting extensions

The rendering backbone is Deformable 3D GS, augmented by three components: a Deformation Enhancement Module (DEM), an Opacity Enhancement Module (OEM), and a Grouping Strategy (GPS) (Huo et al., 2024). DEM is introduced to improve modeling of dynamic objects and static backgrounds. OEM is introduced to improve opacity prediction. GPS is introduced to address large-scale scenes with unreasonable view frustums.

The paper retains the standard 3D GS compositing equation:

$C_{\{u,v\}} = \sum_{i \in N} T_i \alpha_i c_i,$

where $C_{\{u,v\}}$ is the rendered pixel color, $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$ is transmittance, $c_i$ is Gaussian color, and $\alpha_i$ is opacity from the 2D projected Gaussian.

The baseline deformation field is written as

$$(\delta\mathbf{x}, \delta\mathbf{r}, \delta\mathbf{s}) = \mathcal{F_\theta}(\gamma(\operatorname{sg}(\mathbf{x})), \gamma(t), \mathbf{d}),$$

where $\mathbf{x}$, $\mathbf{r}$, and $\mathbf{s}$ are Gaussian position, rotation, and scale, $t$ is time, $C_{\{u,v\}}$0 is positional encoding, $C_{\{u,v\}}$1 is stop-gradient, and $C_{\{u,v\}}$2 is a learned state attribute indicating whether a primitive is static or dynamic.

DEM predicts two factors,

$C_{\{u,v\}}$3

$C_{\{u,v\}}$4

and updates Gaussian attributes as

$C_{\{u,v\}}$5

OEM replaces directly learned opacity with

$C_{\{u,v\}}$6

Taken together, these modules indicate that the nuScenes-SG formulation in the source is not a thin wrapper around vanilla 3D GS. It is a driving-scene-specific extension aimed at dynamic content, scale, and annotation-preserving synthesis. This suggests that visual fidelity and geometric consistency are treated as coupled requirements.

3. Large-scale scene handling through grouping

A major technical issue in large-scale driving scenes is that a standard rendering frustum may include very distant or occluded Gaussians that should not participate in a given view. According to the source, this can produce unrealistic perspectives and increase optimization difficulty (Huo et al., 2024). EGSRAL addresses this with a grouping strategy in which the scene is divided into $C_{\{u,v\}}$7 groups using a fixed image interval, each Gaussian primitive is assigned a group id, and rendering and training are performed only with the Gaussians belonging to the relevant group.

The grouping algorithm is described procedurally. It splits images into groups $C_{\{u,v\}}$8, uses a distance threshold $C_{\{u,v\}}$9 to select Gaussians near each group’s image set, assigns a group id to those Gaussians, and trains with multi-group joint optimization rather than sequential training. The stated effects are that it reduces the active field of view, avoids including distant occluded Gaussians in the current rendering, reduces optimization burden, and speeds up rendering.

The source explicitly distinguishes this design from DrivingGaussian. DrivingGaussian is described as using static incremental training, with group order that matters, and with separate models for dynamic objects and groups. EGSRAL instead uses shuffled multi-group training with a single model, and it states that there is no need for extra object annotations. To improve early-group quality, it further uses overlap training: for group $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$0, it samples images from group $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$1 and the last $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$2 images of group $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$3.

The reported grouping ablation on KITTI City gives PSNR 22.64 with 4 groups, 23.10 with 6 groups, 23.60 with 8 groups, and 23.38 with 10 groups; the best result is with 8 groups. On a KITTI scene with approximately 4 million Gaussians on NVIDIA V100, the timing is 0.768 s/frame for Deformable 3D GS, 0.254 s/frame for EGSRAL with 8 groups, and 0.199 s/frame for Deformable 3D GS with grouping. Although these results are not from nuScenes itself, the source uses them to motivate the scaling behavior that supports nuScenes-style multi-view driving scenes.

4. Coordinate systems, adaptor design, and annotation transfer

The auto-labeling mechanism centers on a coordinate-system mismatch. Dataset annotations and poses are in the Original World Coordinate System (OWCS), whereas SfM-reconstructed point clouds and camera poses are in the Estimated World Coordinate System (EWCS). Since the renderer is trained in EWCS, novel cameras must be expressed there; however, annotations originate in OWCS. The proposed solution is a trainable adaptor that maps poses from OWCS to EWCS (Huo et al., 2024).

The adaptor takes camera pose matrices $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$4 as input and output, uses an 8-layer MLP backbone with a linear layer output head, and is trained with three constraints: an existing pose constraint $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$5, a novel pose 3D constraint $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$6, and a novel pose projection constraint $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$7. For a known camera pose $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$8 in OWCS and its SfM counterpart $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$9 in EWCS, the existing-pose constraint is

$c_i$0

where $c_i$1 is the adaptor prediction.

To generalize beyond observed poses, the method uses a Random Position Transformation (RPT) module to sample a nearby novel pose $c_i$2. The predicted pose in EWCS is $c_i$3, and a corresponding implicit SfM pose $c_i$4 is assumed to exist but is not directly available. Projection-consistency training then enforces consistency of nearby camera relations under projection. For a nearby pose $c_i$5 and novel pose $c_i$6 in OWCS, the relative pose conversion is

$c_i$7

The transformed pose is projected using intrinsics:

$c_i$8

and pixel coordinates are

$c_i$9

The losses are

$\alpha_i$0

$\alpha_i$1

and

$\alpha_i$2

with $\alpha_i$3, $\alpha_i$4, $\alpha_i$5, and $\alpha_i$6 neighboring poses.

At inference time, the workflow is explicit: start from original pose $\alpha_i$7 in OWCS; apply affine random transformation via RPT to get $\alpha_i$8; feed $\alpha_i$9 into the adaptor to get $$(\delta\mathbf{x}, \delta\mathbf{r}, \delta\mathbf{s}) = \mathcal{F_\theta}(\gamma(\operatorname{sg}(\mathbf{x})), \gamma(t), \mathbf{d}),$$0 in EWCS; render novel image $$(\delta\mathbf{x}, \delta\mathbf{r}, \delta\mathbf{s}) = \mathcal{F_\theta}(\gamma(\operatorname{sg}(\mathbf{x})), \gamma(t), \mathbf{d}),$$1 from $$(\delta\mathbf{x}, \delta\mathbf{r}, \delta\mathbf{s}) = \mathcal{F_\theta}(\gamma(\operatorname{sg}(\mathbf{x})), \gamma(t), \mathbf{d}),$$2; apply the same affine transformation to original annotations $$(\delta\mathbf{x}, \delta\mathbf{r}, \delta\mathbf{s}) = \mathcal{F_\theta}(\gamma(\operatorname{sg}(\mathbf{x})), \gamma(t), \mathbf{d}),$$3 to produce $$(\delta\mathbf{x}, \delta\mathbf{r}, \delta\mathbf{s}) = \mathcal{F_\theta}(\gamma(\operatorname{sg}(\mathbf{x})), \gamma(t), \mathbf{d}),$$4; and pair $$(\delta\mathbf{x}, \delta\mathbf{r}, \delta\mathbf{s}) = \mathcal{F_\theta}(\gamma(\operatorname{sg}(\mathbf{x})), \gamma(t), \mathbf{d}),$$5 with $$(\delta\mathbf{x}, \delta\mathbf{r}, \delta\mathbf{s}) = \mathcal{F_\theta}(\gamma(\operatorname{sg}(\mathbf{x})), \gamma(t), \mathbf{d}),$$6. The source emphasizes an important nuance: it does not provide a detailed object-instance taxonomy mapping rule such as a nuScenes class remapping table. Instead, it focuses on camera pose transfer, 3D box coordinate transformation, and generating corresponding 2D/3D annotations for the synthesized view. This directly addresses another common misunderstanding: the method is geometric label transfer, not ontology remapping.

5. nuScenes configurations and reported performance

The source reports results on two nuScenes setups, nuScenes-S and nuScenes-D (Huo et al., 2024). nuScenes-S is based on the setup of S-NeRF, uses only front camera, uses resolution 1600 × 900, takes every 4th image for the test split, and uses scene tokens 164, 209, 359, 916. Averaged over these scenes, EGSRAL reports PSNR 34.43, SSIM 0.939, and LPIPS 0.205. The compared methods are Deformable 3D GS with PSNR 33.43, SSIM 0.932, LPIPS 0.224; 3D GS with PSNR 32.82, SSIM 0.925, LPIPS 0.225; and Mip-Splatting with PSNR 32.22, SSIM 0.928, LPIPS 0.224.

Per-scene EGSRAL metrics on nuScenes-S are reported as 36.32 / 0.946 / 0.193 for scene 164, 36.48 / 0.963 / 0.216 for 209, 31.94 / 0.911 / 0.218 for 359, and 32.99 / 0.936 / 0.191 for 916. The paper also states that, compared against S-NeRF, SUDS, EmerNeRF, and 3D GS, EGSRAL is best on the nuScenes benchmark used by S-NeRF, with 34.43 PSNR, 0.939 SSIM, and 0.205 LPIPS.

nuScenes-D is based on DrivingGaussian, uses synchronized images from 6 surrounding cameras, uses resolution 1600 × 900, takes every 5th image for the test split, and uses scene tokens 103, 168, 212, 220, 228, 687. On this setup, the average for EGSRAL is PSNR 29.04, SSIM 0.883, and LPIPS 0.162, while DrivingGaussian reports PSNR 28.74, SSIM 0.865, and LPIPS 0.237. The source explicitly interprets this as a notable LPIPS improvement, suggesting better perceptual quality.

In the context of nuScenes-SG, these results matter because they combine multi-view rendering quality with a downstream-compatible labeling mechanism. The source’s broader takeaway is that the pipeline achieves state-of-the-art rendering quality on both nuScenes-S and nuScenes-D, with particularly strong nuScenes relevance in the combination of synthesis and annotation transfer.

6. Downstream detection impact and interpretive boundaries

One of the strongest practical claims in the source is that synthesized images with auto-generated labels improve downstream 2D and 3D detection (Huo et al., 2024). The evaluation uses Co-DETR for 2D detection and MonoLSS for 3D monocular detection, with categories car, bus, truck, and trailer. The baseline dataset is constructed from 17 scenes, 3,898 images, and 674 annotated sample images. Using the adaptor, the dataset size is then increased to 2× and 3×.

For 2D detection on the 17-scene dataset, Co-DETR on sample set reports mAP 23.7 at 1×, 25.9 at 2×, and 26.8 at 3×. Co-DETR on all set reports mAP 28.1 at 1×, 29.9 at 2×, and 31.3 at 3×. For 3D detection, MonoLSS on sample set reports AP3D 17.21 at 1×, 19.78 at 2×, and 20.31 at 3×; MonoLSS on all set reports 21.69 at 1×, 22.15 at 2×, and 22.87 at 3×.

The paper also includes a direct validation of label quality by comparing the trainable adaptor against a plain matrix-based transform. The Matrix method gives AP 21.71 and AD 1.867 m, whereas the Adaptor gives AP 72.53 and AD 0.605 m. The source uses this to support the claim that learned pose transfer is much better than a simple rigid alignment, especially when SfM contains outliers or imperfect correspondences.

These results support a specific interpretation of nuScenes-SG. It is not merely a pipeline for producing visually plausible images; it is a mechanism for producing synthesized images whose paired labels are sufficiently consistent to improve detector training. At the same time, the interpretive boundary is clear. The system focuses on pose transformation consistency and 3D box coordinate transformation, not on class remapping or richer semantic relabeling. This suggests that its strongest use case is geometrically grounded data augmentation for nuScenes-like driving perception.