ExtraGS: Trajectory Extrapolation in Driving Scenes

• ExtraGS is a framework for trajectory extrapolation that integrates geometric Gaussian nodes and a neural scene graph paradigm to enhance view synthesis from driving logs.
• It employs Road Surface and Far Field Gaussians to target near and distant scene regions, ensuring consistent lane markings and stable far-field geometry during extrapolation.
• A self-supervised uncertainty mechanism selectively fuses generative outputs with geometric rendering, effectively reducing artifacts like texture smear and geometry warp.

Searching arXiv for ExtraGS and closely related trajectory-extrapolation / 3DGS context papers. ExtraGS is a framework for trajectory extrapolation in autonomous-driving scene rendering that integrates geometric and generative priors to synthesize extrapolated views from recorded driving logs while preserving high fidelity along the original trajectory (Tan et al., 21 Aug 2025). It is designed for the setting in which standard volumetric-rendering approaches, including NeRF and 3DGS, are effective for interpolation but degrade under extrapolation, producing texture smear, geometry warp, and inconsistencies in scene elements such as lane markings and distant buildings. ExtraGS addresses this by combining a Neural Scene Graph decomposition with two additional geometric nodes—Road Surface Gaussians and Far Field Gaussians—and a self-supervised uncertainty mechanism that selectively trusts a generative prior only in regions where extrapolation artifacts occur (Tan et al., 21 Aug 2025).

1. Problem setting and design rationale

Trajectory extrapolation differs from novel-view interpolation along a recorded path because the vehicle exits the logged trajectory and the renderer must synthesize views under geometric configurations insufficiently constrained by the training observations (Tan et al., 21 Aug 2025). In this regime, purely geometric methods tend to break down, while an emerging alternative—using a pre-trained generative model as pseudo ground truth—often over-smooths fine geometry, introduces semantic drift, and does not strictly adhere to the scene’s 3D structure (Tan et al., 21 Aug 2025).

ExtraGS is formulated as a hybrid solution to this failure mode. It builds on the Neural Scene Graph paradigm in which a scene is decomposed into multiple nodes, including static background, dynamic cars, and sky, and augments that decomposition with Road Surface Gaussians and Far Field Gaussians (Tan et al., 21 Aug 2025). A self-supervised uncertainty head then controls where generative corrections should be applied. This suggests that the core design objective is not unconditional hallucination, but selective correction under an explicitly learned confidence model.

At render time, the framework combines geometric rendering and generative pseudo ground truth through per-pixel alpha blending: $$\hat C(x) \;=\;(1 - U(x))\,C_{\rm geom}(x)\;+\;U(x)\,C_{\rm gen}(x),$$ where $C_{\rm geom}(x)$ is the color from pure Gaussian splatting, $C_{\rm gen}(x)$ is the pseudo–ground-truth from a diffusion or flow model, and $U(x)\in[0,1]$ is the learned uncertainty weighting (Tan et al., 21 Aug 2025). A common misconception is that ExtraGS is fundamentally a generative renderer; the formulation shows instead that geometric rendering remains primary, with the generative prior used only where the learned uncertainty is high.

2. System architecture and scene decomposition

The architecture extends Neural Scene Graph by adding two geometric nodes tailored to driving scenes: Road Surface Gaussians, which target the road region, and Far Field Gaussians, which target distant structures such as buildings and clouds (Tan et al., 21 Aug 2025). The stated motivation is that road surfaces are large, mostly planar, low-texture, and often observed at grazing angles, while distant content exhibits huge depth variation and poor LiDAR coverage (Tan et al., 21 Aug 2025).

Road Surface Gaussians are based on a hybrid Gaussian-Signed Distance Function design. The SDF is used to place and orient Gaussians during training iterations, but only the Gaussians are rendered at test time (Tan et al., 21 Aug 2025). Far Field Gaussians introduce a learnable scalar scaling factor per Gaussian to permit efficient adjustment of distant geometry (Tan et al., 21 Aug 2025). These additions are paired with a self-supervised uncertainty head based on spherical harmonics, which estimates how well a given view direction was covered during training (Tan et al., 21 Aug 2025).

Generative priors are treated as black boxes whose outputs are fused through learned uncertainty masks (Tan et al., 21 Aug 2025). The paper explicitly instantiates this with MagicDriveV2 in ExtraGS-M and Difix3D+ in ExtraGS-D (Tan et al., 21 Aug 2025). This suggests a modular interface in which the geometric backbone and the uncertainty-guided fusion are central, while the upstream generative model is replaceable.

3. Road Surface Gaussians

The Road Surface Gaussian representation is intended to model the road as a structured geometric entity rather than as generic volumetric content (Tan et al., 21 Aug 2025). ExtraGS assumes local planarity and uses a dimension-reduced SDF: $$\phi(\mathbf p)\;=\;d(\mathbf p) \;=\;|\cos\theta(\mathbf p)|\;\bigl(p_z - \mathcal H(p_x,p_y)\bigr),$$ where $\mathcal H(p_x,p_y)$ is a learned elevation MLP mapping horizontal coordinates to surface height, and $|\cos\theta(p_x,p_y)|$ is a learned scalar MLP giving the cosine between the road normal and vertical (Tan et al., 21 Aug 2025).

A set of Gaussians $\{G_i\}$ is placed at

$$\mu_i=(\mu_{x,i},\mu_{y,i},\mathcal H(\mu_{x,i},\mu_{y,i}))$$

with anisotropic covariances $\Sigma_i$ constrained to lie in the tangent plane (Tan et al., 21 Aug 2025). Each Gaussian density is

$C_{\rm geom}(x)$0

The hybrid occupancy may be written as

$C_{\rm geom}(x)$1

where the factor $C_{\rm geom}(x)$2 constrains Gaussian support to the zero-level set of the road SDF, although in practice only the Gaussians are rendered (Tan et al., 21 Aug 2025). Within the stated framework, the SDF acts as a geometric scaffold for placement and orientation rather than as a rendered field.

This representation is significant because roads in autonomous-driving scenes are especially vulnerable to extrapolation artifacts: low texture weakens photometric constraints, planar extent amplifies warping, and grazing-angle observations make geometric misalignment visually salient. The explicit road-specific node is intended to retain lane markings and curb boundaries that purely generative approaches tend to blur (Tan et al., 21 Aug 2025).

4. Far Field Gaussians and extrapolation geometry

ExtraGS uses Far Field Gaussians for distant objects because such content exhibits depth variation and weak direct geometric constraints (Tan et al., 21 Aug 2025). Rather than optimizing raw Gaussian position and covariance alone, the method introduces a learnable scalar $C_{\rm geom}(x)$3 per Gaussian: $C_{\rm geom}(x)$4 The resulting density is

$C_{\rm geom}(x)$5

The stated effect is that distant Gaussians can rapidly “zoom out” or “in” to match high-level scene layouts (Tan et al., 21 Aug 2025). A plausible implication is that this parameterization regularizes optimization in the far field by constraining motion and scale updates to a coordinated form, which is especially useful when distant appearance is only sparsely observed or poorly triangulated.

In the overall scene graph, FFG complements RSG by handling the opposite geometric regime. RSG focuses on near-ground, structured, large-support geometry; FFG addresses distant content that is semantically important for realism but difficult to reconstruct stably under trajectory shifts. The paper’s qualitative summary attributes to ExtraGS stable far-field geometry even at 3 m lateral offsets (Tan et al., 21 Aug 2025).

5. Uncertainty-guided integration of generative priors

The uncertainty mechanism is the central control signal that decides where the generative prior should influence the output. ExtraGS models the empirical distribution of training-view directions for each Gaussian node $C_{\rm geom}(x)$6 using a spherical-harmonics expansion: $C_{\rm geom}(x)$7 where

$C_{\rm geom}(x)$8

This provides a learned approximation of view-density coverage (Tan et al., 21 Aug 2025).

Given front-to-back opacity accumulation along a ray, the per-pixel uncertainty is

$C_{\rm geom}(x)$9

If a ray reaches a region from a view direction rarely seen in training, $C_{\rm gen}(x)$0 is small, $C_{\rm gen}(x)$1 becomes large, and the generative prior is trusted more strongly (Tan et al., 21 Aug 2025). The paper also gives an equivalent variance-style expression,

$C_{\rm gen}(x)$2

with coefficients tied to the SH representation (Tan et al., 21 Aug 2025).

This mechanism directly addresses a second misconception: ExtraGS does not fuse generative outputs globally. The weighting is per pixel and conditioned on view-direction coverage, so the framework preserves original-trajectory fidelity where the geometric renderer is reliable and resorts to pseudo ground truth only in unreliable extrapolated regions. The paper summarizes this as selective integration of generative priors only where extrapolation artifacts occur (Tan et al., 21 Aug 2025).

6. Training objectives, priors, and evaluation

ExtraGS uses distinct objectives depending on whether a viewpoint lies on the original trajectory or is extrapolated (Tan et al., 21 Aug 2025). For original-trajectory pixels, the framework applies a reconstruction loss,

$C_{\rm gen}(x)$3

together with a semantic mask loss

$C_{\rm gen}(x)$4

road SDF regularization,

$C_{\rm gen}(x)$5

and an uncertainty negative log-likelihood term,

$C_{\rm gen}(x)$6

For extrapolated viewpoints, the framework uses generated pseudo ground truth and optimizes

$C_{\rm gen}(x)$7

where $C_{\rm gen}(x)$8 is described as a small adversarial term and $C_{\rm gen}(x)$9 as a learnable residual color encoder that mitigates saturation mismatches (Tan et al., 21 Aug 2025). The complete objective is

$U(x)\in[0,1]$0

The implementation includes a multi-level hash-grid for the SDF head, three small MLPs for elevation, slope, and color, and two sets of spherical-harmonic coefficients per Gaussian—one for color and one for uncertainty (Tan et al., 21 Aug 2025). A tiny CNN or MLP serves as the pseudo color encoder (Tan et al., 21 Aug 2025).

The reported evaluation covers multiple datasets, diverse multi-camera setups, and various generative priors (Tan et al., 21 Aug 2025). The datasets and protocols include nuScenes, with six $U(x)\in[0,1]$1 surround cameras over 10 s clips and 3 m lateral-shift extrapolation, and the Waymo Open Dataset, with four-second clips at resolution $U(x)\in[0,1]$2 and 3 m lateral shifts (Tan et al., 21 Aug 2025). The key reported metrics are summarized below.

Setting	Reported metrics
Original trajectory	PSNR $U(x)\in[0,1]$3 dB, SSIM $U(x)\in[0,1]$4, LPIPS $U(x)\in[0,1]$5, FID $U(x)\in[0,1]$6
Extrapolation	FID@3 m shift $U(x)\in[0,1]$7, downstream mAP$U(x)\in[0,1]$8 (BEVFormer v2) $U(x)\in[0,1]$9, mAP$$\phi(\mathbf p)\;=\;d(\mathbf p) \;=\;|\cos\theta(\mathbf p)|\;\bigl(p_z - \mathcal H(p_x,p_y)\bigr),$$0 $$\phi(\mathbf p)\;=\;d(\mathbf p) \;=\;|\cos\theta(\mathbf p)|\;\bigl(p_z - \mathcal H(p_x,p_y)\bigr),$$1
Waymo downstream	NTA-IoU $$\phi(\mathbf p)\;=\;d(\mathbf p) \;=\;|\cos\theta(\mathbf p)|\;\bigl(p_z - \mathcal H(p_x,p_y)\bigr),$$2, NTL-IoU $$\phi(\mathbf p)\;=\;d(\mathbf p) \;=\;|\cos\theta(\mathbf p)|\;\bigl(p_z - \mathcal H(p_x,p_y)\bigr),$$3

The comparisons reported in the paper state that ExtraGS-D surpasses prior generative-prior extrapolation methods including DriveDreamer4D, ReconDreamer, FreeVS, DriveX, and StreetCrafter, while using no external boxes or LiDAR conditioning (Tan et al., 21 Aug 2025).

7. Significance, limitations, and interpretation

The principal contribution of ExtraGS is the explicit coupling of geometry-specialized Gaussian nodes with a learned uncertainty gate for generative fusion (Tan et al., 21 Aug 2025). In the paper’s qualitative characterization, this yields crisp lane markings, realistic curb boundaries, and stable far-field geometry even at 3 m lateral offsets, in contrast to blur or drift in purely generative approaches (Tan et al., 21 Aug 2025). The framework is therefore positioned as a method for preserving geometric consistency under extrapolation without giving up the realism benefits of modern generative priors.

Its design also clarifies the division of labor among components. RSG handles the road as a structured surface; FFG handles distant content through scale-aware Gaussian adaptation; the SH uncertainty head estimates view-support reliability; and the generative prior fills only those regions where the current geometry is unreliable (Tan et al., 21 Aug 2025). This suggests that ExtraGS is best understood not as a single renderer, but as a fusion architecture over geometric and generative evidence.

At the same time, the framework depends on pseudo ground truth from external generative models such as MagicDriveV2 and Difix3D+ (Tan et al., 21 Aug 2025). A plausible implication is that part of its extrapolation performance remains conditioned on the capabilities and failure modes of the chosen prior, even though the uncertainty-guided blending is designed to reduce semantic drift and over-smoothing. The paper’s own framing emphasizes that the method treats such priors as black boxes and relies on learned uncertainty masks to constrain their influence (Tan et al., 21 Aug 2025).

Within the broader 3DGS literature, ExtraGS occupies the subproblem of autonomous-driving trajectory extrapolation rather than generic novel-view synthesis or scene compression. Its stated summary is a “judicious marriage” of strong geometric Gaussian nodes and a self-supervised uncertainty mask for selective, artifact-free use of generative priors, with the goal of state-of-the-art realism and geometric consistency in driving-scene trajectory extrapolation (Tan et al., 21 Aug 2025).