---
title: 'nuScenes-SG: Driving Scene Generation & Labeling'
url: https://www.emergentmind.com/topics/nuscenes-sg
type: topic
---

# nuScenes-SG: Driving Scene Generation & Labeling

Searching arXiv for papers relevant to "nuScenes-SG" and the methods named in the provided data.
Within the available source material, **nuScenes-SG** is best understood as an *Editor's term* for a **nuScenes-style driving-scene generation and auto-labeling setting** built around large-scale scene reconstruction, novel-view synthesis, and annotation transfer. In that setting, the central problem is not only to render plausible novel views of dynamic driving scenes, but also to generate corresponding annotations that remain usable for downstream 2D and 3D perception. The most explicit formulation of this pipeline in the provided evidence is **EGSRAL**, a 3D Gaussian Splatting-based renderer and automated labeling system that is designed to work with **only training images** for reconstruction while transferring annotations from original dataset coordinates to synthesized views [2412.15550]. A plausible implication is that nuScenes-SG is better characterized as a **rendering-and-labeling workflow** than as a purely visual synthesis benchmark.

## 1. Conceptual scope

For **nuScenes-style applications**, the defining contribution described in the source is **not just better novel-view synthesis**, but **an explicit mechanism to transfer labels from the original dataset coordinate system to synthesized views**, making generated data usable for **2D/3D detection training** [2412.15550]. This establishes a two-part objective: first, reconstruct a large-scale driving scene with sufficient fidelity to support novel-view rendering; second, preserve annotation consistency under pose transformation so that synthesized images can be paired with valid labels.

The pipeline is presented as extending standard **3D Gaussian Splatting (3D GS)** in three directions. It introduces **enhanced rendering for complex driving scenes**, **automatic novel-view labeling**, and a **reconstruction stage that uses no extra supervision beyond images**. The source contrasts this with prior driving-scene 3D GS methods that often depend on **LiDAR**, **trajectories**, or **object-level annotations**. The distinction is important: reconstruction is image-only, but auto-labeling still depends on **existing annotations** that are transformed into the synthesized view. This clarifies a likely misconception. The method does **not** claim annotation-free supervision for downstream tasks; rather, it claims **no extra annotation** for the reconstruction stage.

This framing also helps delimit the meaning of “SG” in this context. The source does not define a canonical expansion for the label **nuScenes-SG**. A cautious reading is that it denotes a **nuScenes-like scene generation and label synthesis regime** in which rendering quality, coordinate transfer, and detection utility are jointly relevant.

## 2. Rendering model and 3D Gaussian Splatting extensions

The rendering backbone is **Deformable 3D GS**, augmented by three components: a **Deformation Enhancement Module (DEM)**, an **Opacity Enhancement Module (OEM)**, and a **Grouping Strategy (GPS)** [2412.15550]. DEM is introduced to improve modeling of **dynamic objects and static backgrounds**. OEM is introduced to improve **opacity prediction**. GPS is introduced to address **large-scale scenes with unreasonable view frustums**.

The paper retains the standard 3D GS compositing equation:
$$
C_{\{u,v\}} = \sum_{i \in N} T_i \alpha_i c_i,
$$
where $C_{\{u,v\}}$ is the rendered pixel color, $T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$ is transmittance, $c_i$ is Gaussian color, and $\alpha_i$ is opacity from the 2D projected Gaussian.

The baseline deformation field is written as
$$
(\delta\mathbf{x}, \delta\mathbf{r}, \delta\mathbf{s}) =
\mathcal{F_\theta}(\gamma(\operatorname{sg}(\mathbf{x})), \gamma(t), \mathbf{d}),
$$
where $\mathbf{x}$, $\mathbf{r}$, and $\mathbf{s}$ are Gaussian position, rotation, and scale, $t$ is time, $\gamma(\cdot)$ is positional encoding, $\operatorname{sg}(\cdot)$ is stop-gradient, and $\mathbf{d}$ is a learned state attribute indicating whether a primitive is static or dynamic.

DEM predicts two factors,
$$
\alpha_p = \sigma\!\left(MLP_1(\mathcal{F_\beta}(\mathbf{d}, \gamma(t), \mathbf{x}-\mathbf{c}_{view}))\right),
$$
$$
\alpha_\sigma = \tanh\!\left(MLP_2(\mathcal{F_\beta}(\mathbf{d}, \gamma(t), \mathbf{x}-\mathbf{c}_{view}))\right),
$$
and updates Gaussian attributes as
$$
G_2(\mathbf{x} + \alpha_p \delta\mathbf{x}, \mathbf{r} + \delta\mathbf{r}, \mathbf{s} + \delta\mathbf{s}, \alpha_\sigma \sigma).
$$
OEM replaces directly learned opacity with
$$
\sigma = \mathcal{F_\eta}(\boldsymbol{\sigma}').
$$

Taken together, these modules indicate that the nuScenes-SG formulation in the source is not a thin wrapper around vanilla 3D GS. It is a driving-scene-specific extension aimed at dynamic content, scale, and annotation-preserving synthesis. This suggests that visual fidelity and geometric consistency are treated as coupled requirements.

## 3. Large-scale scene handling through grouping

A major technical issue in **large-scale driving scenes** is that a standard rendering frustum may include **very distant or occluded Gaussians** that should not participate in a given view. According to the source, this can produce **unrealistic perspectives** and increase **optimization difficulty** [2412.15550]. EGSRAL addresses this with a **grouping strategy** in which the scene is divided into $N$ groups using a **fixed image interval**, each Gaussian primitive is assigned a **group id**, and rendering and training are performed only with the Gaussians belonging to the relevant group.

The grouping algorithm is described procedurally. It **splits images into groups $G_{img}[i]$**, uses a **distance threshold $d$** to select Gaussians near each group’s image set, assigns a **group id** to those Gaussians, and trains with **multi-group joint optimization** rather than sequential training. The stated effects are that it **reduces the active field of view**, **avoids including distant occluded Gaussians in the current rendering**, **reduces optimization burden**, and **speeds up rendering**.

The source explicitly distinguishes this design from **DrivingGaussian**. DrivingGaussian is described as using **static incremental training**, with **group order** that **matters**, and with **separate models for dynamic objects and groups**. EGSRAL instead uses **shuffled multi-group training** with a **single model**, and it states that there is **no need for extra object annotations**. To improve early-group quality, it further uses **overlap training**: for group $j$, it samples images from group $j$ and the last $N_o$ images of group $j-1$.

The reported grouping ablation on **KITTI City** gives **PSNR 22.64** with **4 groups**, **23.10** with **6 groups**, **23.60** with **8 groups**, and **23.38** with **10 groups**; the best result is with **8 groups**. On a KITTI scene with approximately **4 million Gaussians** on **NVIDIA V100**, the timing is **0.768 s/frame** for **Deformable 3D GS**, **0.254 s/frame** for **EGSRAL with 8 groups**, and **0.199 s/frame** for **Deformable 3D GS with grouping**. Although these results are not from nuScenes itself, the source uses them to motivate the scaling behavior that supports nuScenes-style multi-view driving scenes.

## 4. Coordinate systems, adaptor design, and annotation transfer

The auto-labeling mechanism centers on a coordinate-system mismatch. Dataset annotations and poses are in the **Original World Coordinate System (OWCS)**, whereas **SfM-reconstructed point clouds and camera poses** are in the **Estimated World Coordinate System (EWCS)**. Since the renderer is trained in EWCS, novel cameras must be expressed there; however, annotations originate in OWCS. The proposed solution is a trainable **adaptor** that maps poses from OWCS to EWCS [2412.15550].

The adaptor takes **camera pose matrices $3\times 4$** as input and output, uses an **8-layer MLP** backbone with a **linear layer** output head, and is trained with three constraints: an **existing pose constraint $L_p$**, a **novel pose 3D constraint $L_{3D}$**, and a **novel pose projection constraint $L_{proj}$**. For a known camera pose $P$ in OWCS and its SfM counterpart $P_S$ in EWCS, the existing-pose constraint is
$$
L_p = smooth_{L_1}(P_A, P_S),
$$
where $P_A$ is the adaptor prediction.

To generalize beyond observed poses, the method uses a **Random Position Transformation (RPT)** module to sample a nearby novel pose $P_{nov}$. The predicted pose in EWCS is $P_{nov\_A}$, and a corresponding implicit SfM pose $P_{nov\_S}$ is assumed to exist but is **not directly available**. Projection-consistency training then enforces consistency of nearby camera relations under projection. For a nearby pose $P_w^{n+i}$ and novel pose $P_w^{n_{nov}}$ in OWCS, the relative pose conversion is
$$
\hat P_{n_{nov}}^{n+i} = (P_w^{n_{nov}})^{-1} * P_w^{n+i}.
$$
The transformed pose is projected using intrinsics:
$$
\acute{P}_{n_{nov}}^{n+i} = K_{intr} * \begin{bmatrix} T_{\hat P_{n_{nov}}^{n+i}} \end{bmatrix},
$$
and pixel coordinates are
$$
\tilde{P}_{n_{nov}}^{n+i} =
\begin{bmatrix}
x_{\acute{P}_{n_{nov}}^{n+i}} / z_{\acute{P}_{n_{nov}}^{n+i}} \\
y_{\acute{P}_{n_{nov}}^{n+i}} / z_{\acute{P}_{n_{nov}}^{n+i}}
\end{bmatrix}.
$$
The losses are
$$
L_{proj} = \sum_{i=1}^{N} MSE(\tilde{P}_{n_{nov\_A}}^{n+i}, \tilde{P}_{n_{nov\_S}}^{n+i}),
$$
$$
L_{3D} = \sum_{i=1}^{N} smooth_{L_1}(\hat{P}_{n_{nov\_A}}^{n+i}, \hat{P}_{n_{nov\_S}}^{n+i}),
$$
and
$$
L_{all} = w_1 L_p + w_2 L_{3D} + w_3 L_{proj},
$$
with **$w_1 = 50$**, **$w_2 = 0.1$**, **$w_3 = 1$**, and **$N = 15$** neighboring poses.

At inference time, the workflow is explicit: start from original pose $P_{ori}$ in OWCS; apply affine random transformation via RPT to get $P_{nov}$; feed $P_{nov}$ into the adaptor to get $P_{nov\_S}$ in EWCS; render novel image $image_{nov}$ from $P_{nov\_S}$; apply the same affine transformation to original annotations $anns_{ori}$ to produce $anns_{nov}$; and pair $image_{nov}$ with $anns_{nov}$. The source emphasizes an important nuance: it does **not** provide a detailed object-instance taxonomy mapping rule such as a **nuScenes class remapping table**. Instead, it focuses on **camera pose transfer**, **3D box coordinate transformation**, and generating corresponding **2D/3D annotations** for the synthesized view. This directly addresses another common misunderstanding: the method is geometric label transfer, not ontology remapping.

## 5. nuScenes configurations and reported performance

The source reports results on two nuScenes setups, **nuScenes-S** and **nuScenes-D** [2412.15550]. **nuScenes-S** is based on the setup of **S-NeRF**, uses **only front camera**, uses resolution **1600 × 900**, takes every **4th** image for the test split, and uses scene tokens **164, 209, 359, 916**. Averaged over these scenes, **EGSRAL** reports **PSNR 34.43**, **SSIM 0.939**, and **LPIPS 0.205**. The compared methods are **Deformable 3D GS** with **PSNR 33.43**, **SSIM 0.932**, **LPIPS 0.224**; **3D GS** with **PSNR 32.82**, **SSIM 0.925**, **LPIPS 0.225**; and **Mip-Splatting** with **PSNR 32.22**, **SSIM 0.928**, **LPIPS 0.224**.

Per-scene EGSRAL metrics on nuScenes-S are reported as **36.32 / 0.946 / 0.193** for scene **164**, **36.48 / 0.963 / 0.216** for **209**, **31.94 / 0.911 / 0.218** for **359**, and **32.99 / 0.936 / 0.191** for **916**. The paper also states that, compared against **S-NeRF**, **SUDS**, **EmerNeRF**, and **3D GS**, EGSRAL is best on the nuScenes benchmark used by S-NeRF, with **34.43 PSNR**, **0.939 SSIM**, and **0.205 LPIPS**.

**nuScenes-D** is based on **DrivingGaussian**, uses synchronized images from **6 surrounding cameras**, uses resolution **1600 × 900**, takes every **5th** image for the test split, and uses scene tokens **103, 168, 212, 220, 228, 687**. On this setup, the average for **EGSRAL** is **PSNR 29.04**, **SSIM 0.883**, and **LPIPS 0.162**, while **DrivingGaussian** reports **PSNR 28.74**, **SSIM 0.865**, and **LPIPS 0.237**. The source explicitly interprets this as a **notable LPIPS improvement**, suggesting **better perceptual quality**.

In the context of nuScenes-SG, these results matter because they combine **multi-view rendering quality** with a downstream-compatible labeling mechanism. The source’s broader takeaway is that the pipeline achieves **state-of-the-art rendering quality on both nuScenes-S and nuScenes-D**, with particularly strong nuScenes relevance in the combination of synthesis and annotation transfer.

## 6. Downstream detection impact and interpretive boundaries

One of the strongest practical claims in the source is that synthesized images with auto-generated labels improve downstream **2D** and **3D** detection [2412.15550]. The evaluation uses **Co-DETR** for **2D detection** and **MonoLSS** for **3D monocular detection**, with categories **car**, **bus**, **truck**, and **trailer**. The baseline dataset is constructed from **17 scenes**, **3,898 images**, and **674 annotated sample images**. Using the adaptor, the dataset size is then increased to **2×** and **3×**.

For **2D detection** on the **17-scene dataset**, **Co-DETR on sample set** reports **mAP 23.7** at **1×**, **25.9** at **2×**, and **26.8** at **3×**. **Co-DETR on all set** reports **mAP 28.1** at **1×**, **29.9** at **2×**, and **31.3** at **3×**. For **3D detection**, **MonoLSS on sample set** reports **AP3D 17.21** at **1×**, **19.78** at **2×**, and **20.31** at **3×**; **MonoLSS on all set** reports **21.69** at **1×**, **22.15** at **2×**, and **22.87** at **3×**.

The paper also includes a direct validation of label quality by comparing the trainable adaptor against a **plain matrix-based transform**. The **Matrix method** gives **AP 21.71** and **AD 1.867 m**, whereas the **Adaptor** gives **AP 72.53** and **AD 0.605 m**. The source uses this to support the claim that learned pose transfer is much better than a simple rigid alignment, especially when **SfM contains outliers or imperfect correspondences**.

These results support a specific interpretation of nuScenes-SG. It is not merely a pipeline for producing visually plausible images; it is a mechanism for producing synthesized images whose paired labels are sufficiently consistent to improve detector training. At the same time, the interpretive boundary is clear. The system focuses on **pose transformation consistency** and **3D box coordinate transformation**, not on class remapping or richer semantic relabeling. This suggests that its strongest use case is **geometrically grounded data augmentation** for nuScenes-like driving perception.

Source: https://www.emergentmind.com/topics/nuscenes-sg