---
title: UniForge3D 3D Reconstruction and Generation
url: https://www.emergentmind.com/topics/uniforge3d
type: topic
---

# UniForge3D 3D Reconstruction and Generation

UniForge3D is a joint 3D reconstruction-and-generation framework for producing complete textured 3D assets from sparse, unposed RGB images. Given four input images with unknown camera poses, it combines a VGGT-based reconstruction branch with a TRELLIS.2-initialized sparse-voxel generation branch in an anchor-guided shared object space. The framework preserves observed geometry and appearance while completing occluded or unseen regions, and uses generated structure to establish pixel-to-voxel correspondences and select the most informative input view for each voxel [2610.15357].

## 1. Problem formulation and objectives

UniForge3D addresses the tension between reconstruction and generation. Reconstruction methods can accurately recover visible surfaces but generally produce incomplete geometry, whereas generative models can complete unseen regions using learned shape priors but may replace input-specific details with merely plausible geometry. The framework jointly optimizes these capabilities to generate a complete asset that preserves the geometry and appearance visible in the input images while maintaining consistency under novel viewpoints.

The input is a sparse set of unposed images,

$$
\mathcal I=\{I_i\}_{i=1}^{N},
$$

where the camera poses are unknown and $N$ is small. The output is a complete textured 3D asset supporting novel-view rendering. UniForge3D decomposes the task into two coupled stages:

1. **Anchor-guided joint reconstruction and generation**: a VGGT reconstruction branch predicts per-view point maps and reconstruction features, while a DiT-based sparse-voxel branch predicts a complete coarse structure.
2. **Structure-guided view selection and conditioning**: the predicted structure conditions reconstruction, establishes pixel-to-voxel correspondences, and selects the most informative view for local shape and texture generation.

This yields the processing chain

$$
\text{unposed images}
\rightarrow
\text{shared-space reconstruction and generation}
\rightarrow
\text{sparse structure}
\rightarrow
\text{structure-conditioned point maps}
\rightarrow
\text{pixel-to-voxel correspondences}
\rightarrow
\text{voxel-wise view selection}
\rightarrow
\text{detailed shape}
\rightarrow
\text{texture}.
$$

The central design problem is coordinate compatibility. VGGT predicts geometry in a camera-relative coordinate system, whereas native 3D generators such as TRELLIS.2 operate in canonical object space. Direct interaction between these branches therefore requires an additional spatial alignment. UniForge3D instead defines a common anchor-guided space in which both reconstructed points and generated voxels are represented.

## 2. Anchor-guided shared object space

The first input image is designated as the anchor view. The shared coordinate system follows the anchor view’s azimuth while preserving the object’s upright direction. For a canonical-space point $p^{\mathrm{can}}\in\mathbb R^3$, the anchor-guided coordinate is

$$
p^\star=R_a p^{\mathrm{can}},
$$

where $R_a\in SO(3)$ is a rotation about the canonical vertical axis chosen so that the anchor-view azimuth defines the shared-space front direction.

This transformation changes horizontal orientation without tilting the object. It relies on the upright-object assumption: the object’s vertical direction is known or consistently aligned with the canonical vertical axis. Under this assumption, azimuth alignment is sufficient and a full arbitrary three-dimensional camera rotation is unnecessary.

The same transformed geometry is used to construct both training targets:

- visible surface points become point-map targets;
- complete object geometry is voxelized into sparse-structure targets.

Consequently, reconstructed points and generated voxels share the same coordinates without requiring a separately predicted rigid or similarity transformation. The shared space is therefore both a geometric interface between the two branches and the basis for later pixel-to-voxel correspondence.

The reported ablation results support this design. On GSO, anchor-guided joint training obtains an IoU of $0.5827$ and a CD of $1.0563\times10^{-2}$ for the sparse structure, compared with an IoU of $0.4563$ and a CD of $1.6291\times10^{-2}$ for cross-space joint training, and an IoU of $0.4601$ and a CD of $1.5825\times10^{-2}$ for canonical-space joint training.

## 3. Reconstruction, generation, and bidirectional interaction

### Reconstruction branch

The reconstruction branch is based on VGGT. For each view $v$, it produces image features, cross-view geometric features, reconstruction tokens, per-pixel 3D point predictions, and confidence values. Its point-map output is

$$
P_v\in\mathbb R^{H\times W\times3},
$$

where each pixel predicts a 3D point in the shared anchor-guided space. The corresponding confidence map is

$$
\hat\rho_v\in\mathbb R^{H\times W}.
$$

The paper does not specify numerical values for $H$, $W$, or the internal VGGT channel dimensions.

### Generation branch

The generation branch is a sparse-voxel DiT initialized from TRELLIS.2. It predicts a complete coarse sparse voxel structure rather than a dense voxel grid. The generated structure supplies:

- global spatial support for detailed shape generation;
- a coarse complete geometry prior;
- structure tokens for conditioning reconstruction;
- voxel locations for pixel-to-voxel correspondence and view selection.

Detailed shape generation subsequently operates at 512 and 1024 resolutions.

### Reconstruction-to-generation exchange

Let $F_v^l$ denote VGGT features for view $v$ at layer $l$, $D_v^l$ denote corresponding DINOv3 patch features, and $\phi^l$ denote an adapter mapping VGGT features into the DINOv3 feature space. View-specific conditioning is formed as

$$
C_v^l=D_v^l+\phi^l(F_v^l).
$$

The conditions from all views are concatenated:

$$
C_{\mathrm{mv}}^l=\operatorname{Concat}_v[C_v^l].
$$

Generation tokens use cross-attention with $C_{\mathrm{mv}}^l$ as keys and values. This provides the generator with both DINOv3 visual information and VGGT geometric information. The residual outputs of the newly introduced adapters are zero-initialized, keeping the initial model close to the pretrained components.

### Generation-to-reconstruction exchange

After a generation block, reconstruction tokens attend to generation tokens:

$$
R^l
=
R^l+
g^l(t)\odot
A_{g\rightarrow r}(Q=R^l,K=G^l,V=G^l),
$$

where $R^l$ denotes concatenated reconstruction tokens, $G^l$ denotes generation tokens, $g^l(t)$ is a time- or noise-level-dependent gate, and $t$ is the diffusion or flow-matching time.

This interaction provides the reconstruction branch with complete-shape context. It is intended to improve point-map prediction in regions where individual views offer weak or ambiguous evidence. The first-stage objective is

$$
\mathcal L_{\mathrm{joint}}
=
\mathcal L_{\mathrm{flow}}
+
\mathcal L_{\mathrm{point}}.
$$

The flow-matching term trains sparse-structure generation, while the point-map term trains reconstruction.

## 4. Structure-guided reconstruction and view selection

After the initial sparse structure is generated, UniForge3D feeds it back into a structure-conditioned reconstruction network. For each voxel center $x_j$, a Fourier positional encoding followed by an MLP produces a structure token:

$$
S_j=\operatorname{MLP}(\operatorname{PE}(x_j)).
$$

The complete sequence of structure tokens is

$$
S=\{S_j\}_j.
$$

Reconstruction tokens attend to these tokens according to

$$
R'=R+\operatorname{CA}(R,S,S).
$$

The resulting point maps and confidence maps are expressed in the same coordinate system as the sparse structure. UniForge3D then establishes explicit pixel-to-voxel correspondences:

1. retain valid foreground pixels;
2. confidence-filter predictions, retaining the highest-confidence $95\%$;
3. quantize each predicted point onto the sparse voxel grid.

For voxel $x_j$ and view $v$, let $\mathcal Q_{v,j}$ denote the pixels in view $v$ that quantize to that voxel. The view-support score is

$$
s_v(x_j)=|\operatorname{LCC}(\mathcal Q_{v,j})|,
$$

where $\operatorname{LCC}$ denotes the largest connected component of the corresponding image pixels. The selected view is

$$
v_j^\star=\arg\max_v s_v(x_j).
$$

The largest connected component criterion favors views in which a voxel is supported by a spatially coherent image region rather than by isolated or noisy pixels. Ties favor the anchor view and, if necessary, the view with higher mean confidence in the selected connected component. If no directly supported pixels exist, the voxel inherits the selected view of the most strongly supported voxel in its 26-neighborhood; if no neighboring voxel has support, the anchor view is used.

This selection is a geometric correspondence-based rule rather than a learned categorical classifier. It addresses the fact that a surface can be clearly visible in one image, oblique in another, and occluded in a third.

## 5. Detailed shape and texture generation

The detailed shape generator combines global multi-view attention with voxel-specific selected-view attention. Reconstruction features from multiple VGGT levels are fused with DINOv3 features:

$$
C_v^l
=
D_v^l+
\psi^l\left(\operatorname{Concat}_k F_v^k\right),
$$

where $F_v^k$ are multi-level structure-conditioned reconstruction features and $\psi^l$ performs multi-level mapping, fusion, and spatial alignment.

Two forms of cross-attention are alternated:

- **Global multi-view attention**: every voxel token attends to concatenated feature sequences from all input views, providing global context and learned completion.
- **Voxel-specific attention**: each voxel attends to the feature sequence of its selected view $v_j^\star$, supplying local evidence from the image that most clearly observes that region.

The combination is intended to preserve observed local details while maintaining global multi-view consistency and completing unseen surfaces using the learned 3D prior. The shape DiT updates the shape latent at the sparse locations defined by the generated structure, and a shape decoder converts the latent representation into a mesh.

Texture generation uses the same structure-guided and voxel-wise view-conditioning principle. The final mesh is encoded into a fixed shape latent, which conditions a texture DiT operating on a noisy texture latent. The decoded outputs include:

- base color;
- metallic;
- roughness;
- opacity.

These attributes are baked into UV textures.

The point-map loss is defined as

$$
\mathcal L_{\mathrm{point}}
=
\mathcal L_{\mathrm{reg}}
+
\mathcal L_{\mathrm{conf}}
+
\mathcal L_{\mathrm{normal}}.
$$

For a valid foreground pixel,

$$
e_i=\|P_i-P_i^\ast\|_2.
$$

The confidence term is

$$
\mathcal L_{\mathrm{conf}}
=
\mathcal R\left(
\left\{
\hat\rho_i e_i-0.2\log\hat\rho_i
\right\}
\right),
$$

where $\mathcal R$ is robust averaging with a $98\%$ quantile filter. The normal term is the mean cosine error between point-map normals evaluated at three scales. Standalone structure-conditioned reconstruction uses the same regression, confidence, and normal terms.

The detailed shape and texture generators use latent flow matching. Classifier-free guidance is enabled by randomly dropping image conditioning with probability $0.1$.

## 6. Training, evaluation, and limitations

UniForge3D is trained on a 210K-object subset of TRELLIS-500K. For each object, two random azimuthal orientations are sampled, six candidate anchor views are rendered for each orientation, and 48 additional random viewpoints are rendered for multi-view training.

The joint reconstruction-and-generation stage uses VGGT and TRELLIS.2 initialization, LoRA ranks of 64 for generation and 32 for reconstruction, 60K iterations, global batch size 128, and peak learning rate $5\times10^{-5}$. DINOv3 is frozen, while the new interaction modules are fully trained. Structure-guided reconstruction is initialized from VGGT and fully fine-tuned for 40K iterations before being frozen for subsequent generation training. Detailed shape generation uses separate 512- and 1024-resolution generators, rank-128 denoiser LoRA, 60K iterations, global batch size 64, and learning rate $5\times10^{-5}$. Texture generation uses analogous rank-128 LoRA adaptation with the shape latent fixed. All stages use AdamW with cosine learning-rate decay and are trained on 32 NVIDIA A800 GPUs.

Evaluation uses Toys4K and 500 objects from GSO, with four randomly selected unposed RGB images supplied to each method. Meshes are aligned to ground truth using a GICP-based Sim(3) transformation before the principal metrics are computed. Appearance is evaluated using PSNR, SSIM, and LPIPS; geometry is evaluated using Chamfer Distance and F-score at distance threshold $0.01$.

UniForge3D reports the following results:

| Dataset | PSNR | SSIM | LPIPS | CD $\times10^{-2}$ | F-score |
|---|---:|---:|---:|---:|---:|
| Toys4K | 27.428 | 0.9616 | 0.0401 | 1.4196 | 0.8899 |
| GSO | 24.826 | 0.9363 | 0.0591 | 1.4047 | 0.8897 |

The framework is reported as best on every listed appearance and geometry metric relative to Stream3D + TRELLIS.2, Hunyuan3D-2MV, MV-SAM3D, ReconViaGen v0.2, ReconViaGen v0.5, and Mix3R. On GSO, the voxel-space ablation obtains an IoU of $0.5827$ and a CD of $1.0563\times10^{-2}$ for anchor-guided joint training. Structure-conditioned reconstruction achieves an overall point-map error of $3.313\times10^{-3}$, compared with $4.846\times10^{-3}$ for image-only VGGT. Voxel-wise view selection improves GSO shape-generation performance from PSNR $26.892$, SSIM $0.9649$, LPIPS $0.0401$, CD $1.4739\times10^{-2}$, and F-score $0.8758$ to PSNR $27.207$, SSIM $0.9657$, LPIPS $0.0388$, CD $1.4047\times10^{-2}$, and F-score $0.8897$.

The principal limitation is the reliance on synthetic renderings for training. Performance remains challenged by severe occlusion, strong reflections, extreme lighting, and difficult real-world appearance conditions. The method also depends on the upright-object assumption and a fixed anchor-guided orientation strategy. Its sparse-voxel representation provides a coarse complete structure, while detailed generation remains dependent on learned priors and may be affected by errors in the initial structure or point maps.

UniForge3D’s central contribution is therefore the integration of reconstruction and generation through a shared geometric interface. The anchor-guided space resolves the coordinate mismatch between camera-relative reconstruction and canonical-space generation; bidirectional feature exchange allows observed evidence and complete-shape priors to influence one another; structure-conditioned point maps establish explicit pixel-to-voxel correspondences; and voxel-wise view selection routes the most informative local evidence into detailed shape and texture generation. Together, these mechanisms provide a framework for complete 3D asset generation that combines observation fidelity with learned completion.

Source: https://www.emergentmind.com/topics/uniforge3d