Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)
Papers
Topics
Authors
Recent
Search
2000 character limit reached

DrivingForward 3D Scene Reconstruction

Updated 7 January 2026
  • DrivingForward is a feed-forward, self-supervised model that reconstructs 3D driving scenes from sparse, multi-camera data using 3D Gaussian splatting.
  • It jointly trains Pose, Depth, and Gaussian networks with a differentiable renderer to achieve fast, real-time inference and metric consistency.
  • Its design mitigates challenges of limited view overlap and uncertain camera extrinsics, enabling robust automotive scene reconstruction.

DrivingForward is a feed-forward model designed for real-time driving scene reconstruction using 3D Gaussian splatting from flexible surround-view camera input. It addresses the challenges inherent to vehicle-mounted camera imagery, which is typically sparse with limited view overlap, and where vehicle motion makes reliable camera extrinsics difficult to obtain. DrivingForward reconstructs 3D scenes by jointly training three neural networks—Pose, Depth, and Gaussian—along with a differentiable 3D Gaussian-splatting renderer, all operating in a self-supervised regime without ground-truth depth or extrinsics, and is capable of fast, feed-forward inference from arbitrary subsets of multi-frame, multi-camera data (Tian et al., 2024).

1. Model Architecture

DrivingForward comprises three main components—Pose Network (P), Depth Network (D), and Gaussian Network (G)—co-trained with a differentiable renderer.

  • Pose Network (P): Receives pairs of images from temporal (same camera, different times), spatial (different cameras, same time), or spatio-temporal contexts. Outputs a relative 3×43 \times 4 transformation Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3) estimated using a ResNet-18 backbone encoder followed by a two-layer MLP producing a 6D vector (axis-angle rotation + translation). Photometric reprojection losses supervise this transformation.
  • Depth Network (D): Consumes a single image and outputs a dense depth map DitD_i^t, along with intermediate latent features FimageF_{image}. Its backbone consists of ResNet-18, a volumetric feature-fusion module, and a U-Net decoder supporting multi-scale depth prediction.
  • Gaussian Network (G): For each input pixel, predicts the parameters of a Gaussian primitive gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}.
    • The center μk\mu_k is formed by unprojecting the depth Di(u)D_i(u): μi(u)=Π−1(u,Di(u))\mu_i(u) = \Pi^{-1}(u, D_i(u)), where Π−1=Ki−1uDi(u)\Pi^{-1} = K_i^{-1} u D_i(u), all in vehicle coordinates.
    • Covariance Σk=R(rk)diag(sk)R(rk)⊤\Sigma_k = R(r_k) \mathrm{diag}(s_k) R(r_k)^\top with Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3)0 (axis-scales), Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3)1 (unit quaternion).
    • Opacity Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3)2 and color coefficients Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3)3 in a spherical-harmonics basis.
    • The network has a depth encoder (U-Net on Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3)4), a fusion decoder (merging Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3)5 and Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3)6), and four output "heads" for Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3)7 (softplus), Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3)8 (Tit→t′∈SE(3)T_{i^t \rightarrow t'} \in SE(3)9 normalization), DitD_i^t0 (sigmoid), and DitD_i^t1 (linear).

2. 3D Gaussian Splatting Formulation

Splatting is the core rendering mechanism whereby 3D Gaussian primitives, predicted from input images, are rendered into novel target views:

  • Projection: Each primitive's mean DitD_i^t2 and covariance DitD_i^t3 (in vehicle coordinates) are projected into 2D using the target camera intrinsics and extrinsics. The image-plane Gaussian parameters are:
    • DitD_i^t4
    • DitD_i^t5, with DitD_i^t6
  • Per-pixel Splatting Weight:

    DitD_i^t7

    Each primitive contributes weighted color to pixels in the target view.

  • Compositing: Primitives are sorted by increasing depth (DitD_i^t8). At each pixel DitD_i^t9:

    FimageF_{image}0

    Final color is accumulated via:

    FimageF_{image}1

    All splatting and compositing are differentiable on GPU, following the 3DGS paradigm.

3. Self-supervision and Loss Functions

Training is fully self-supervised using only RGB surround-camera images (nuScenes dataset, no LiDAR/extrinsics). The overall loss is:

FimageF_{image}2

  • Pose/Depth self-supervision (FimageF_{image}3):
    • Reprojection loss:

    FimageF_{image}4 - Targeted reprojection:

    FimageF_{image}5 - Multiple contexts: temporal (FimageF_{image}6), spatial (FimageF_{image}7), and spatial-temporal (FimageF_{image}8). - Depth smoothness:

    FimageF_{image}9 - Loss weights: gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}0.

  • Rendering supervision (gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}1):

    gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}2

    • gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}3.
  • Regularization: Enforced using quaternion normalization (for gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}4), softplus/sigmoid activations (for gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}5), and depth smoothness.

4. Inference Workflow

Inference is strictly feed-forward, requiring only the Depth and Gaussian networks (and the differentiable renderer). The sequence proceeds as follows for a set of gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}6 images gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}7:

  1. For each input image gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}8:
    • Depth network infers gk={μk,Σk,αk,ck}g_k = \{\mu_k, \Sigma_k, \alpha_k, c_k\}9 and μk\mu_k0.
    • Depth unprojection yields μk\mu_k1 for each pixel μk\mu_k2.
    • Gaussian network predicts per-pixel μk\mu_k3, μk\mu_k4, μk\mu_k5.
  2. Pools all pixel-Gaussians into one set μk\mu_k6.
  3. Renderer produces the novel view via splatting/compositing.

Feed-forward operation ensures outputs remain metrically consistent for arbitrary frame/camera combinations, without per-scene optimization. Inference on six surround cameras executes in approximately 0.3–0.6 seconds on a single A6000 GPU.

Below is a summary table of inference flow:

Step Operation Output
Per image DNet: μk\mu_k7 Depth map, latent feature
Per pixel Unproject μk\mu_k8 3D primitive center μk\mu_k9
Per pixel GNet: fusion, ouputs Di(u)D_i(u)0, Di(u)D_i(u)1, Di(u)D_i(u)2
All images/pixels Aggregate Di(u)D_i(u)3 Set of all Gaussian primitives
All primitives Render view Splatting-composited RGB image

5. Parameterizations and Differentiable Rendering

DrivingForward assigns a 3D Gaussian primitive to every input pixel:

  • Center (Di(u)D_i(u)4): By depth unprojection in vehicle coordinates.
  • Covariance (Di(u)D_i(u)5): Parameterized by an axis-aligned scale Di(u)D_i(u)6 and rotation Di(u)D_i(u)7 (as quaternion, converted via Di(u)D_i(u)8).
  • Opacity (Di(u)D_i(u)9): Scaled to [0,1] by sigmoid.
  • Color coefficients (μi(u)=Π−1(u,Di(u))\mu_i(u) = \Pi^{-1}(u, D_i(u))0): Spherical harmonics basis up to degree μi(u)=Π−1(u,Di(u))\mu_i(u) = \Pi^{-1}(u, D_i(u))1, for view-dependent effects.

The differentiable renderer implements:

  • 3D→2D Gaussian projection (using μi(u)=Π−1(u,Di(u))\mu_i(u) = \Pi^{-1}(u, D_i(u))2 and its Jacobian),
  • Per-pixel Gaussian evaluation and weighted summation across sorted depths,
  • Composite image assembly.

This enables fully end-to-end differentiable optimization during training, as well as fast, optimization-free inference.

6. Application Domain and Significance

DrivingForward is tailored to the automotive context, leveraging sparse, surround-view camera configurations on moving vehicles. Real-world data, such as the nuScenes dataset, is used for training and evaluation—no LiDAR or ground-truth extrinsics are required.

The architecture robustly handles the low overlap and extrinsic uncertainty typical of automotive multi-camera systems. Comparative experiments demonstrate superior scene reconstruction quality over existing feed-forward and scene-optimized methods, particularly in real-scale depth recovery and real-time novel-view synthesis (Tian et al., 2024). The model’s flexibility to accept arbitrary combinations of frames and views, while maintaining metric consistency and operational speed (<1 s), is notable for practical deployment.

7. Implementation Notes and Inference Pseudocode

The inference process is purely feed-forward, with no test-time optimization. The following pseudocode summarizes the steps:

μi(u)=Π−1(u,Di(u))\mu_i(u) = \Pi^{-1}(u, D_i(u))3

Block diagram representation:

μi(u)=Π−1(u,Di(u))\mu_i(u) = \Pi^{-1}(u, D_i(u))4

All architecture, loss, and inference pipeline details are fully specified in the original source (Tian et al., 2024).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to DrivingForward Architecture.