VoxelTTO: Geometrically Accurate 3D Reconstruction

• VoxelTTO is a feed-forward framework for reconstructing 3D scenes using multiple RGB images, even with optional camera parameters, by aggregating dense image features into a global voxel representation and decoding Gaussian primitives. It ensures multiview consistency and maintains geometric accuracy and offers test-time pose adaptation when camera parameters are available.
• The voxels are decoded from either 2Gaussians (2G) per occupied voxel at a side-length of 0.002m as standard, this results in consistency in decoding the geometric structure and color data across сценs, samples for each Gaussian contains 27-coefficient order 2 RGB spherical harmonics
• The system uses stochastic solid volume rendering in both training and inference to enhance depth and geometric details, and for the adaptive parts, lighter models such as LoRA are used.

VoxelTTO is a feed-forward framework for reconstructing geometrically accurate three-dimensional Gaussian Splatting (3DGS) scenes from an arbitrary number of RGB images, with optional camera parameters. It aggregates dense multiview image features into a global voxel representation, decodes Gaussian primitives from voxel features rather than associating them independently with image pixels, and uses stochastic solid volume rendering during training and inference. When camera parameters are available, pose-supervised test-time optimization (TTO) adapts lightweight LoRA modules while keeping the visual foundation model (VFM) frozen. The resulting system combines voxel-aligned scene representation, sparse 3D processing, differentiable volumetric rendering, and camera-conditioned test-time adaptation (Zhao et al., 18 Sep 2026).

1. Motivation and conceptual foundations

VoxelTTO addresses limitations of feed-forward 3DGS methods that predict Gaussian primitives in a pixel-aligned manner. In such systems, each image pixel or patch may generate one or more Gaussians, even when multiple views observe the same physical surface. The resulting primitives can overlap excessively, have inconsistent scales and orientations, and accumulate opacity inappropriately. Typical artifacts include floating blobs, blurred surfaces, holes, ghosting, and incorrect boundaries.

Pixel-aligned prediction also weakens multiview consistency. A Gaussian generated from one view is not necessarily colocated with the Gaussian generated from another view for the same surface element. Consequently, the number and spatial arrangement of primitives depend heavily on image sampling rather than on persistent three-dimensional scene structure.

A second difficulty concerns camera pose. VFM-based reconstruction systems may predict camera extrinsics and intrinsics with residual rotation, translation, or scale errors. These errors can be partially compensated in input-view rendering while producing severe misalignment in novel views. VoxelTTO therefore separates scene representation from image indexing: observations are back-projected into a common three-dimensional coordinate system, aggregated spatially, and processed as voxel features.

The method is distinct from conventional per-scene 3DGS optimization. It does not optimize every Gaussian independently for each scene. Instead, the primary reconstruction path is feed-forward, while optional TTO adapts lightweight LoRA parameters in the VFM using pose supervision. The large pretrained VFM parameters remain frozen.

2. Multiview feature extraction and voxel construction

For input images $\mathbf I=\{\mathbf I_m\}_{m=1}^{M}$, the VFM and prediction head produce point maps, depth maps, camera parameters, and confidence maps:

$$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$

Here, $\mathbf P_m\in\mathbb R^{H\times W\times3}$ is a point map, $\mathbf D_m\in\mathbb R^{H\times W}$ is a depth map, and $\mathbf g_m$ contains camera extrinsics and intrinsics. The backbone is initialized from DA3-GIANT-style pretrained weights and contains 40 attention layers. Layers 1–12 use single-view attention, whereas layers 13–40 alternate single-view and global attention. DINOv2 features are used as the visual representation, and a DPT feature decoder upsamples patch tokens to dense image features.

For view $m$, the dense feature map is

$\mathbf F_m\in\mathbb R^{H\times W\times C}.$

Each sampled image location contributes a three-dimensional point, RGB color, dense feature, and confidence. The resulting dense feature point cloud is

$$\mathcal P = \left\{ (\mathbf p_l,\mathbf a_l,\mathbf f_l,\kappa_l) \right\}_{l=1}^{L},$$

where $\mathbf p_l$ is a world-coordinate point, $\mathbf a_l$ is RGB color, $$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$0 is an image feature, and $$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$1 is confidence.

VoxelTTO uses regular voxels with a default side length of $$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$2 m. If $$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$3 is the set of observations assigned to voxel $$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$4, the voxel feature center is confidence-weighted:

$$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$5

Voxel color and feature are aggregated analogously:

$$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$6

Voxel confidence is conservatively assigned as

$$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$7

This aggregation changes the primitive-generation unit from the image pixel to the persistent three-dimensional voxel. Observations from different views that occupy the same spatial region are fused before Gaussian decoding. The voxel center is learned from observations and need not coincide with the nominal center of the regular grid cell.

The aggregated voxel features are processed by a sparse three-dimensional convolutional U-Net. Sparse processing omits empty grid cells while providing local and larger-context spatial reasoning. The resulting refined voxel features are passed to separate geometry and appearance decoder branches.

3. Voxel-aligned Gaussian representation

Each occupied voxel decodes $$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$8 Gaussian primitives. The reported default and best-performing configuration uses

$$(\mathbf P_m,\mathbf D_m,\mathbf g_m) = \Gamma_\psi\!\left(\Phi_\theta(\mathbf I)\right).$$9

For Gaussian $\mathbf P_m\in\mathbb R^{H\times W\times3}$0, the geometry branch predicts a center $\mathbf P_m\in\mathbb R^{H\times W\times3}$1, scale $\mathbf P_m\in\mathbb R^{H\times W\times3}$2, rotation quaternion $\mathbf P_m\in\mathbb R^{H\times W\times3}$3, and opacity $\mathbf P_m\in\mathbb R^{H\times W\times3}$4:

$\mathbf P_m\in\mathbb R^{H\times W\times3}$5

The appearance branch predicts second-order spherical-harmonic coefficients:

$\mathbf P_m\in\mathbb R^{H\times W\times3}$6

The decoder is divided into geometry and appearance branches so that appearance optimization does not excessively interfere with geometry.

A conventional covariance construction from scale and quaternion is

$\mathbf P_m\in\mathbb R^{H\times W\times3}$7

where $\mathbf P_m\in\mathbb R^{H\times W\times3}$8 is the rotation matrix obtained from the normalized quaternion. The paper does not specify in the supplied text whether Gaussian centers are predicted as absolute world coordinates or as offsets from $\mathbf P_m\in\mathbb R^{H\times W\times3}$9; this implementation detail requires the released code.

Second-order RGB spherical harmonics contain 27 coefficients under the conventional dimensionality $\mathbf D_m\in\mathbb R^{H\times W}$0. Directional appearance can therefore be expressed as

$\mathbf D_m\in\mathbb R^{H\times W}$1

where $\mathbf D_m\in\mathbb R^{H\times W}$2 is the viewing direction.

The central representational property is voxel alignment. Multiple Gaussians may be decoded from one voxel, but their generation is conditioned on a shared spatial feature rather than on independent image pixels. This reduces duplicate primitive generation and encourages a consistent three-dimensional organization. It does not imply that each voxel corresponds to a single physical surface point; the voxel is a spatial support for one or more Gaussian primitives.

4. Pose-supervised test-time optimization

VoxelTTO optionally performs TTO when camera parameters are available. The objective is to adapt pose-sensitive VFM computations without updating the VFM backbone or pose decoder. LoRA modules are inserted into the query, key, and value projections of attention layers 13–40. The original VFM parameters remain frozen.

A generic LoRA update is

$\mathbf D_m\in\mathbb R^{H\times W}$3

up to the usual scaling convention. The down-projection matrix $\mathbf D_m\in\mathbb R^{H\times W}$4 is shared, while the up-projection matrices $\mathbf D_m\in\mathbb R^{H\times W}$5 are independent across layers.

The first camera defines the scene coordinate frame, and the first ground-truth camera pose is fixed to the identity. Because reconstruction has scale ambiguity, predicted and ground-truth translations are separately normalized by their mean norms. The rotation loss is

$\mathbf D_m\in\mathbb R^{H\times W}$6

The translation loss uses SmoothL1 after scale normalization:

$\mathbf D_m\in\mathbb R^{H\times W}$7

Intrinsics are represented by

$\mathbf D_m\in\mathbb R^{H\times W}$8

with direct SmoothL1 supervision:

$\mathbf D_m\in\mathbb R^{H\times W}$9

The TTO objective is

$\mathbf g_m$0

The supplied manuscript does not report numerical values for $\mathbf g_m$1, $\mathbf g_m$2, or $\mathbf g_m$3.

VoxelTTO uses 20 TTO iterations. For ten $\mathbf g_m$4 images, the reported runtime is 2.304 s without TTO, 0.2662 s for one TTO forward pass, and 0.075 s for one TTO backward pass. Twenty TTO iterations require under 7 s, with total runtime under 7.4 s. Frozen attention computations use BF16, whereas LoRA calculations use FP32. The reported backbone runtime decreases from 1.959 s to 0.1956 s when using BF16.

TTO is therefore a targeted adaptation of the VFM rather than a full scene-optimization procedure. Its gradients flow from pose losses through the pose decoder and VFM attention computations into LoRA parameters only. The resulting adapted VFM predictions are then used for voxel construction and Gaussian decoding.

5. Stochastic solid volume rendering

VoxelTTO replaces vanilla 3DGS rasterization with stochastic solid volume rendering during both training and inference. The motivation is that RGB renderings can remain close to ordinary 3DGS rasterization while depth rendering provides stronger geometric behavior at boundaries and in regions containing holes.

Let $\mathbf g_m$5 be a point on a ray, $\mathbf g_m$6 its direction, $\mathbf g_m$7 a vacancy function, $\mathbf g_m$8 attenuation, and $\mathbf g_m$9 transmittance. Stochastic solid attenuation is defined by

$m$0

Transmittance is

$m$1

and rendered color is

$m$2

For a Gaussian field $m$3, the paper uses

$m$4

For Gaussian primitive $m$5, with $m$6 denoting the location of the Gaussian maximum along the ray, the accumulated transmittance is

$m$7

This construction produces RGB renderings close to Gaussian rasterization but yields a depth distribution more suitable for geometric supervision.

VoxelTTO renders median depth. The median location $m$8 satisfies

$m$9

It is found using a fixed number of bisection iterations. Differentiation through the implicit condition gives

$\mathbf F_m\in\mathbb R^{H\times W\times C}.$0

and therefore

$\mathbf F_m\in\mathbb R^{H\times W\times C}.$1

This permits depth gradients to reach Gaussian centers, scales, rotations, opacities, voxel features, and upstream image features.

The reconstruction objective combines RGB and depth terms:

$\mathbf F_m\in\mathbb R^{H\times W\times C}.$2

with

$\mathbf F_m\in\mathbb R^{H\times W\times C}.$3

The depth loss contains an $\mathbf F_m\in\mathbb R^{H\times W\times C}.$4 difference between rendered and ground-truth depth and an alignment term involving VFM-predicted depth. RGB supervision consists of pixelwise $\mathbf F_m\in\mathbb R^{H\times W\times C}.$5 and SSIM losses. No separate geometric, opacity, scale, pose, or Gaussian regularization loss is reported for the standard feed-forward training objective.

6. Training, evaluation, and limitations

VoxelTTO is trained on ScanNet, Infinigen, ARKitScenes, and DL3DV-10K. Training sequences are sampled with 3–16 frames resized to $\mathbf F_m\in\mathbb R^{H\times W\times C}.$6. The model uses Adam for 30,000 iterations, an initial learning rate of $\mathbf F_m\in\mathbb R^{H\times W\times C}.$7, cosine annealing, a 2,000-iteration warm-up, and a minimum learning rate of $\mathbf F_m\in\mathbb R^{H\times W\times C}.$8. Approximately 75.3 million parameters are trainable during standard training, compared with a full model size of 1.39 billion parameters. Training requires 40 hours on two RTX PRO 6000 GPUs, corresponding to 80 GPU hours.

The default voxel side length is $\mathbf F_m\in\mathbb R^{H\times W\times C}.$9 m, and each occupied voxel decodes two Gaussian primitives. Evaluation uses Replica, Tanks and Temples, and DTU. Reported metrics include PSNR, SSIM, LPIPS, depth RMSE, relative depth error, threshold accuracy, camera rotation and translation error, and DTU surface alignment.

On Replica, VoxelTTO with TTO obtains input-view PSNR/SSIM/LPIPS of 29.91/.896/.170 and test-view values of 28.54/.876/.172. Without TTO, the corresponding test-view values are 25.83/.827/.196. On Tanks and Temples, the full method obtains test-view PSNR/SSIM/LPIPS of 19.23/.565/.391, compared with 14.71/.485/.475 without TTO.

Depth performance is particularly strong on Replica. The full method obtains test-view RMSE 6.945, relative error .866, and $$\mathcal P = \left\{ (\mathbf p_l,\mathbf a_l,\mathbf f_l,\kappa_l) \right\}_{l=1}^{L},$$0, $$\mathcal P = \left\{ (\mathbf p_l,\mathbf a_l,\mathbf f_l,\kappa_l) \right\}_{l=1}^{L},$$1, and $$\mathcal P = \left\{ (\mathbf p_l,\mathbf a_l,\mathbf f_l,\kappa_l) \right\}_{l=1}^{L},$$2 values of 98.86, 99.24, and 99.58, respectively. On Tanks and Temples, it obtains test-view RMSE 46.22, relative error 3.235, and corresponding threshold values of 86.75, 95.16, and 98.33.

For camera pose, the full method reports mean and maximum rotation errors of .220 and .316 degrees on Replica, with mean and maximum translation errors of .002 and .006 m. On Tanks and Temples, the corresponding rotation errors are .262 and .356 degrees, and translation errors are .020 and .057 m. On DTU, camera alignment improves surface metrics from 5.556 accuracy, 9.159 completeness, and 7.357 overall without TTO to 2.952, 3.880, and 3.416 with TTO.

Ablations identify several design dependencies. Two Gaussians per voxel outperform one and four in the reported balance between appearance, depth, and computation. Training and inference with stochastic solid volume rendering produce PSNR 28.54, SSIM .876, LPIPS .172, RMSE 6.95, and AbsRel .87, whereas vanilla rasterization in both stages produces 21.52, .774, .269, 27.58, and 4.05. Reducing voxel size to .0015 m improves the reported metrics, while increasing it to .005 m causes substantial degradation. Removing the depth loss increases test-view RMSE from 6.95 to 7.49 and relative error from .87 to .94.

The method has several limitations. TTO requires aligned camera supervision; without camera parameters, the system operates as the no-TTO variant. Performance depends on the VFM’s depth, point-map, confidence, and pose predictions, as well as on image coverage and overlap. Voxel size creates a fidelity–efficiency trade-off: smaller voxels preserve more detail but increase occupied voxel count, sparse-convolution memory, Gaussian count, and rendering cost. The method also retains the computational burden of a 1.39-billion-parameter VFM, and TTO increases inference time from approximately 2.3 s to under 7.4 s for ten images.

Several reproduction-critical details are unspecified in the supplied manuscript, including the dense feature dimension, exact LoRA rank, TTO learning rate, TTO loss weights, ray-sampling distribution, activations enforcing positive scales and opacities, total Gaussian counts per scene, and whether Gaussian centers are predicted absolutely or relative to voxel centers. These omissions do not alter the principal definition of VoxelTTO, but they prevent complete implementation specification without the released code.

VoxelTTO’s central contribution is the integration of three mechanisms: voxel-aligned multiview feature aggregation, stochastic solid volume rendering, and pose-supervised LoRA test-time optimization. The first addresses primitive overlap and multiview inconsistency, the second supplies depth-sensitive geometric supervision, and the third adapts camera-dependent VFM computations without updating the foundation model. Together they define a feed-forward 3DGS reconstruction system with optional, lightweight test-time adaptation.