---
title: 'VoxelTTO: Geometrically Accurate 3D Reconstruction'
url: https://www.emergentmind.com/topics/voxeltto
type: topic
---

# VoxelTTO: Geometrically Accurate 3D Reconstruction

VoxelTTO is a feed-forward framework for reconstructing geometrically accurate three-dimensional Gaussian Splatting (3DGS) scenes from an arbitrary number of RGB images, with optional camera parameters. It aggregates dense multiview image features into a global voxel representation, decodes Gaussian primitives from voxel features rather than associating them independently with image pixels, and uses stochastic solid volume rendering during training and inference. When camera parameters are available, pose-supervised test-time optimization (TTO) adapts lightweight LoRA modules while keeping the visual foundation model (VFM) frozen. The resulting system combines voxel-aligned scene representation, sparse 3D processing, differentiable volumetric rendering, and camera-conditioned test-time adaptation [2609.21498].

## 1. Motivation and conceptual foundations

VoxelTTO addresses limitations of feed-forward 3DGS methods that predict Gaussian primitives in a pixel-aligned manner. In such systems, each image pixel or patch may generate one or more Gaussians, even when multiple views observe the same physical surface. The resulting primitives can overlap excessively, have inconsistent scales and orientations, and accumulate opacity inappropriately. Typical artifacts include floating blobs, blurred surfaces, holes, ghosting, and incorrect boundaries.

Pixel-aligned prediction also weakens multiview consistency. A Gaussian generated from one view is not necessarily colocated with the Gaussian generated from another view for the same surface element. Consequently, the number and spatial arrangement of primitives depend heavily on image sampling rather than on persistent three-dimensional scene structure.

A second difficulty concerns camera pose. VFM-based reconstruction systems may predict camera extrinsics and intrinsics with residual rotation, translation, or scale errors. These errors can be partially compensated in input-view rendering while producing severe misalignment in novel views. VoxelTTO therefore separates scene representation from image indexing: observations are back-projected into a common three-dimensional coordinate system, aggregated spatially, and processed as voxel features.

The method is distinct from conventional per-scene 3DGS optimization. It does not optimize every Gaussian independently for each scene. Instead, the primary reconstruction path is feed-forward, while optional TTO adapts lightweight LoRA parameters in the VFM using pose supervision. The large pretrained VFM parameters remain frozen.

## 2. Multiview feature extraction and voxel construction

For input images $\mathbf I=\{\mathbf I_m\}_{m=1}^{M}$, the VFM and prediction head produce point maps, depth maps, camera parameters, and confidence maps:

$$
(\mathbf P_m,\mathbf D_m,\mathbf g_m)
=
\Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).
$$

Here, $\mathbf P_m\in\mathbb R^{H\times W\times3}$ is a point map, $\mathbf D_m\in\mathbb R^{H\times W}$ is a depth map, and $\mathbf g_m$ contains camera extrinsics and intrinsics. The backbone is initialized from DA3-GIANT-style pretrained weights and contains 40 attention layers. Layers 1–12 use single-view attention, whereas layers 13–40 alternate single-view and global attention. DINOv2 features are used as the visual representation, and a DPT feature decoder upsamples patch tokens to dense image features.

For view $m$, the dense feature map is

$$
\mathbf F_m\in\mathbb R^{H\times W\times C}.
$$

Each sampled image location contributes a three-dimensional point, RGB color, dense feature, and confidence. The resulting dense feature point cloud is

$$
\mathcal P =
\left\{
(\mathbf p_l,\mathbf a_l,\mathbf f_l,\kappa_l)
\right\}_{l=1}^{L},
$$

where $\mathbf p_l$ is a world-coordinate point, $\mathbf a_l$ is RGB color, $\mathbf f_l$ is an image feature, and $\kappa_l$ is confidence.

VoxelTTO uses regular voxels with a default side length of $0.002$ m. If $\mathcal P_v$ is the set of observations assigned to voxel $v$, the voxel feature center is confidence-weighted:

$$
\mathbf x_v =
\frac{\sum_{l\in\mathcal P_v}\kappa_l\mathbf p_l}
{\sum_{l\in\mathcal P_v}\kappa_l}.
$$

Voxel color and feature are aggregated analogously:

$$
\mathbf a_v =
\frac{\sum_{l\in\mathcal P_v}\kappa_l\mathbf a_l}
{\sum_{l\in\mathcal P_v}\kappa_l},
\qquad
\mathbf f_v =
\frac{\sum_{l\in\mathcal P_v}\kappa_l\mathbf f_l}
{\sum_{l\in\mathcal P_v}\kappa_l}.
$$

Voxel confidence is conservatively assigned as

$$
\kappa_v=\min_{l\in\mathcal P_v}\kappa_l.
$$

This aggregation changes the primitive-generation unit from the image pixel to the persistent three-dimensional voxel. Observations from different views that occupy the same spatial region are fused before Gaussian decoding. The voxel center is learned from observations and need not coincide with the nominal center of the regular grid cell.

The aggregated voxel features are processed by a sparse three-dimensional convolutional U-Net. Sparse processing omits empty grid cells while providing local and larger-context spatial reasoning. The resulting refined voxel features are passed to separate geometry and appearance decoder branches.

## 3. Voxel-aligned Gaussian representation

Each occupied voxel decodes $n$ Gaussian primitives. The reported default and best-performing configuration uses

$$
n=2.
$$

For Gaussian $q$, the geometry branch predicts a center $\boldsymbol\mu_q$, scale $\mathbf s_q$, rotation quaternion $\mathbf r_q$, and opacity $\sigma_q$:

$$
(\boldsymbol\mu_q,\mathbf s_q,\mathbf r_q,\sigma_q)
=
\mathcal D_{\mathrm{geo}}(\mathbf f_v).
$$

The appearance branch predicts second-order spherical-harmonic coefficients:

$$
\mathbf c_q=\mathcal D_{\mathrm{app}}(\mathbf f_v).
$$

The decoder is divided into geometry and appearance branches so that appearance optimization does not excessively interfere with geometry.

A conventional covariance construction from scale and quaternion is

$$
\boldsymbol\Sigma_q
=
\mathbf R(\mathbf r_q)
\operatorname{diag}(\mathbf s_q^2)
\mathbf R(\mathbf r_q)^\mathsf T,
$$

where $\mathbf R(\mathbf r_q)$ is the rotation matrix obtained from the normalized quaternion. The paper does not specify in the supplied text whether Gaussian centers are predicted as absolute world coordinates or as offsets from $\mathbf x_v$; this implementation detail requires the released code.

Second-order RGB spherical harmonics contain 27 coefficients under the conventional dimensionality $3(2+1)^2$. Directional appearance can therefore be expressed as

$$
\mathbf c_q(\boldsymbol\omega)
=
\sum_{l=0}^{2}\sum_{m=-l}^{l}
\mathbf c_{q,lm}Y_{lm}(\boldsymbol\omega),
$$

where $\boldsymbol\omega$ is the viewing direction.

The central representational property is voxel alignment. Multiple Gaussians may be decoded from one voxel, but their generation is conditioned on a shared spatial feature rather than on independent image pixels. This reduces duplicate primitive generation and encourages a consistent three-dimensional organization. It does not imply that each voxel corresponds to a single physical surface point; the voxel is a spatial support for one or more Gaussian primitives.

## 4. Pose-supervised test-time optimization

VoxelTTO optionally performs TTO when camera parameters are available. The objective is to adapt pose-sensitive VFM computations without updating the VFM backbone or pose decoder. LoRA modules are inserted into the query, key, and value projections of attention layers 13–40. The original VFM parameters remain frozen.

A generic LoRA update is

$$
W'=W+BA,
$$

up to the usual scaling convention. The down-projection matrix $A$ is shared, while the up-projection matrices $B$ are independent across layers.

The first camera defines the scene coordinate frame, and the first ground-truth camera pose is fixed to the identity. Because reconstruction has scale ambiguity, predicted and ground-truth translations are separately normalized by their mean norms. The rotation loss is

$$
\mathcal L_{\mathrm{rot}}
=
\frac{1}{2M}
\sum_{i=1}^{M}
\left[
3-
\operatorname{tr}
\left(
(\mathbf R_i^p)^\mathsf T\mathbf R_i^g
\right)
\right].
$$

The translation loss uses SmoothL1 after scale normalization:

$$
\mathcal L_{\mathrm{trans}}
=
\operatorname{SmoothL1}
\left(
\left\{\frac{\mathbf t_i^p}{s_p}\right\}_{i=1}^{M},
\left\{\frac{\mathbf t_i^g}{s_g}\right\}_{i=1}^{M}
\right).
$$

Intrinsics are represented by

$$
\mathbf k=(f_x,f_y,c_x,c_y),
$$

with direct SmoothL1 supervision:

$$
\mathcal L_{\mathrm{intrinsics}}
=
\operatorname{SmoothL1}(\mathbf k^p,\mathbf k^g).
$$

The TTO objective is

$$
\mathcal L_{\mathrm{TTO}}
=
\lambda_{\mathrm{pose}}
\left(
\mathcal L_{\mathrm{rot}}
+
w_t\mathcal L_{\mathrm{trans}}
\right)
+
\lambda_K\mathcal L_{\mathrm{intrinsics}}.
$$

The supplied manuscript does not report numerical values for $\lambda_{\mathrm{pose}}$, $w_t$, or $\lambda_K$.

VoxelTTO uses 20 TTO iterations. For ten $518\times518$ images, the reported runtime is 2.304 s without TTO, 0.2662 s for one TTO forward pass, and 0.075 s for one TTO backward pass. Twenty TTO iterations require under 7 s, with total runtime under 7.4 s. Frozen attention computations use BF16, whereas LoRA calculations use FP32. The reported backbone runtime decreases from 1.959 s to 0.1956 s when using BF16.

TTO is therefore a targeted adaptation of the VFM rather than a full scene-optimization procedure. Its gradients flow from pose losses through the pose decoder and VFM attention computations into LoRA parameters only. The resulting adapted VFM predictions are then used for voxel construction and Gaussian decoding.

## 5. Stochastic solid volume rendering

VoxelTTO replaces vanilla 3DGS rasterization with stochastic solid volume rendering during both training and inference. The motivation is that RGB renderings can remain close to ordinary 3DGS rasterization while depth rendering provides stronger geometric behavior at boundaries and in regions containing holes.

Let $\mathbf x(t)$ be a point on a ray, $\boldsymbol\omega$ its direction, $v(\mathbf x)$ a vacancy function, $\sigma(\mathbf x,\boldsymbol\omega)$ attenuation, and $T(t)$ transmittance. Stochastic solid attenuation is defined by

$$
\sigma(\mathbf x,\boldsymbol\omega)
=
\left|
\boldsymbol\omega^\mathsf T\nabla\log v(\mathbf x)
\right|
=
\frac{
\left|
\boldsymbol\omega^\mathsf T\nabla v(\mathbf x)
\right|
}{
v(\mathbf x)
}.
$$

Transmittance is

$$
T(t)
=
\exp
\left(
-\int_{t_n}^{t}
\sigma(\mathbf x(s),\boldsymbol\omega)\,ds
\right),
$$

and rendered color is

$$
\mathbf C
=
\int_{t_n}^{t_f}
T(t)
\sigma(\mathbf x(t),\boldsymbol\omega)
\mathbf c(\mathbf x(t),\boldsymbol\omega)\,dt.
$$

For a Gaussian field $G(\mathbf x)$, the paper uses

$$
v(\mathbf x)=\sqrt{1-G(\mathbf x)}.
$$

For Gaussian primitive $i$, with $t_i^*$ denoting the location of the Gaussian maximum along the ray, the accumulated transmittance is

$$
T_i(t)=
\begin{cases}
\sqrt{1-G_i(t)}, & t\le t_i^*,\\[4pt]
\dfrac{1-G_i(t_i^*)}{\sqrt{1-G_i(t)}}, & t>t_i^*.
\end{cases}
$$

This construction produces RGB renderings close to Gaussian rasterization but yields a depth distribution more suitable for geometric supervision.

VoxelTTO renders median depth. The median location $t_{\mathrm{med}}$ satisfies

$$
T(t_{\mathrm{med}})=0.5.
$$

It is found using a fixed number of bisection iterations. Differentiation through the implicit condition gives

$$
\frac{\partial T}{\partial t}
\frac{\partial t_{\mathrm{med}}}{\partial\theta}
+
\frac{\partial T}{\partial\theta}
=0,
$$

and therefore

$$
\frac{\partial t_{\mathrm{med}}}{\partial\theta}
=
-
\frac{\partial T/\partial\theta}
{\partial T/\partial t}.
$$

This permits depth gradients to reach Gaussian centers, scales, rotations, opacities, voxel features, and upstream image features.

The reconstruction objective combines RGB and depth terms:

$$
\mathcal L
=
\lambda_{\ell_1^d}\mathcal L_{\mathrm{depth}}^{\ell_1}
+
\lambda_{\mathrm{align}}\mathcal L_{\mathrm{depth}}^{\mathrm{align}}
+
\lambda_{\ell_1}\mathcal L_{\mathrm{rgb}}^{\ell_1}
+
\lambda_{\mathrm{ssim}}\mathcal L_{\mathrm{rgb}}^{\mathrm{ssim}},
$$

with

$$
\lambda_{\ell_1^d}=0.25,\qquad
\lambda_{\mathrm{align}}=0.25,\qquad
\lambda_{\ell_1}=0.9,\qquad
\lambda_{\mathrm{ssim}}=1.0.
$$

The depth loss contains an $\ell_1$ difference between rendered and ground-truth depth and an alignment term involving VFM-predicted depth. RGB supervision consists of pixelwise $\ell_1$ and SSIM losses. No separate geometric, opacity, scale, pose, or Gaussian regularization loss is reported for the standard feed-forward training objective.

## 6. Training, evaluation, and limitations

VoxelTTO is trained on ScanNet, Infinigen, ARKitScenes, and DL3DV-10K. Training sequences are sampled with 3–16 frames resized to $224\times448$. The model uses Adam for 30,000 iterations, an initial learning rate of $3\times10^{-6}$, cosine annealing, a 2,000-iteration warm-up, and a minimum learning rate of $10^{-8}$. Approximately 75.3 million parameters are trainable during standard training, compared with a full model size of 1.39 billion parameters. Training requires 40 hours on two RTX PRO 6000 GPUs, corresponding to 80 GPU hours.

The default voxel side length is $0.002$ m, and each occupied voxel decodes two Gaussian primitives. Evaluation uses Replica, Tanks and Temples, and DTU. Reported metrics include PSNR, SSIM, LPIPS, depth RMSE, relative depth error, threshold accuracy, camera rotation and translation error, and DTU surface alignment.

On Replica, VoxelTTO with TTO obtains input-view PSNR/SSIM/LPIPS of 29.91/.896/.170 and test-view values of 28.54/.876/.172. Without TTO, the corresponding test-view values are 25.83/.827/.196. On Tanks and Temples, the full method obtains test-view PSNR/SSIM/LPIPS of 19.23/.565/.391, compared with 14.71/.485/.475 without TTO.

Depth performance is particularly strong on Replica. The full method obtains test-view RMSE 6.945, relative error .866, and $\delta_{1.05}$, $\delta_{1.10}$, and $\delta_{1.25}$ values of 98.86, 99.24, and 99.58, respectively. On Tanks and Temples, it obtains test-view RMSE 46.22, relative error 3.235, and corresponding threshold values of 86.75, 95.16, and 98.33.

For camera pose, the full method reports mean and maximum rotation errors of .220 and .316 degrees on Replica, with mean and maximum translation errors of .002 and .006 m. On Tanks and Temples, the corresponding rotation errors are .262 and .356 degrees, and translation errors are .020 and .057 m. On DTU, camera alignment improves surface metrics from 5.556 accuracy, 9.159 completeness, and 7.357 overall without TTO to 2.952, 3.880, and 3.416 with TTO.

Ablations identify several design dependencies. Two Gaussians per voxel outperform one and four in the reported balance between appearance, depth, and computation. Training and inference with stochastic solid volume rendering produce PSNR 28.54, SSIM .876, LPIPS .172, RMSE 6.95, and AbsRel .87, whereas vanilla rasterization in both stages produces 21.52, .774, .269, 27.58, and 4.05. Reducing voxel size to .0015 m improves the reported metrics, while increasing it to .005 m causes substantial degradation. Removing the depth loss increases test-view RMSE from 6.95 to 7.49 and relative error from .87 to .94.

The method has several limitations. TTO requires aligned camera supervision; without camera parameters, the system operates as the no-TTO variant. Performance depends on the VFM’s depth, point-map, confidence, and pose predictions, as well as on image coverage and overlap. Voxel size creates a fidelity–efficiency trade-off: smaller voxels preserve more detail but increase occupied voxel count, sparse-convolution memory, Gaussian count, and rendering cost. The method also retains the computational burden of a 1.39-billion-parameter VFM, and TTO increases inference time from approximately 2.3 s to under 7.4 s for ten images.

Several reproduction-critical details are unspecified in the supplied manuscript, including the dense feature dimension, exact LoRA rank, TTO learning rate, TTO loss weights, ray-sampling distribution, activations enforcing positive scales and opacities, total Gaussian counts per scene, and whether Gaussian centers are predicted absolutely or relative to voxel centers. These omissions do not alter the principal definition of VoxelTTO, but they prevent complete implementation specification without the released code.

VoxelTTO’s central contribution is the integration of three mechanisms: voxel-aligned multiview feature aggregation, stochastic solid volume rendering, and pose-supervised LoRA test-time optimization. The first addresses primitive overlap and multiview inconsistency, the second supplies depth-sensitive geometric supervision, and the third adapts camera-dependent VFM computations without updating the foundation model. Together they define a feed-forward 3DGS reconstruction system with optional, lightweight test-time adaptation.

Source: https://www.emergentmind.com/topics/voxeltto