UniForge3D: Joint 3D Reconstruction and Generation in a Shared Anchor-Guided Space

Abstract: Generating complete 3D assets from sparse, unposed images requires preserving observed geometry while completing unseen regions. Joint reconstruction and generation offers a way to address both goals: reconstruction recovers the geometry observed in the input images, while generation contributes learned shape priors. However, combining camera-relative reconstruction with canonical-space generation introduces a coordinate mismatch that makes direct geometric interaction more difficult. Moreover, conditioning each 3D location on all views without accounting for their relevance may mix useful geometric evidence with unrelated image content, hindering local detail recovery. We present UniForge3D, which jointly models reconstruction and generation in a shared anchor-guided space, resolving their coordinate mismatch. This space follows the anchor view's azimuth while keeping the object upright. The two branches exchange features bidirectionally to predict per-view point maps and a complete sparse voxel structure. We further condition point-map prediction on the generated structure to establish pixel-to-voxel correspondences and select the most informative input view for each voxel. Features from the selected views then guide detailed shape generation, helping preserve observed geometric details. Our method achieves state-of-the-art appearance and geometry results on Toys4K and GSO, generating complete 3D assets with improved fidelity. Project webpage: https://xishuxishu.github.io/uniforge3d-project/

• The paper introduces UniForge3D, a method for jointly reconstructing and generating 3D assets from sparse, unposed RGB images, improving both completion and input fidelity.
• UniForge3D uses an anchor-guided shared space and bidirectional feature exchange between reconstruction and generation branches to enhance 3D asset creation, achieving top performance metrics on Toys4K and GSO datasets
• The method’s architecture includes a VGGT-based reconstruction branch and a TRELLIS.2-based sparse-voxel generation branch, both conditioned on the same anchor-guided space, reducing error propagation and improving local geometry integrity.

UniForge3D addresses a central tension in sparse-view 3D asset generation: reconstruction provides input-faithful geometry but is incomplete in unobserved regions, whereas generative modeling supplies shape priors but can replace input-specific structures with merely plausible alternatives. The paper’s principal claim is that these functions should be modeled jointly rather than arranged in a unidirectional reconstruction-then-generation pipeline. Its solution combines bidirectional interaction between reconstruction and generation with a shared coordinate system and voxel-wise selection of informative image evidence (2610.15357).

Problem formulation and contribution

The task is to generate a complete textured 3D asset from four sparse, unposed RGB images. The output should satisfy two requirements simultaneously. First, surfaces visible in the input images should be reconstructed with high geometric and appearance fidelity. Second, occluded or unseen regions should be completed using learned 3D priors while remaining consistent with the observed object.

The paper identifies two deficiencies in existing approaches. Image-conditioned 3D generators can preserve semantic plausibility without enforcing exact correspondence to observed structures. Conversely, methods that first estimate geometry and then condition a generator are vulnerable to error propagation: inaccurate reconstructed geometry becomes an unreliable generation condition. Joint reconstruction and generation offers a potential remedy, but prior systems such as Mix3R retain different coordinate representations for their reconstruction and generation branches, making geometric interaction dependent on an additional alignment process.

UniForge3D makes three architectural contributions. It defines an anchor-guided shared object space in which both branches represent geometry. It introduces bidirectional feature exchange between a VGGT-based reconstruction branch and a TRELLIS.2-based sparse-voxel generation branch. Finally, it uses structure-conditioned point maps to establish pixel-to-voxel correspondences and select the most informative input view independently for each voxel (2610.15357).

Anchor-guided shared space

The shared coordinate system is designed to reconcile two competing requirements: alignment with image observations and compatibility with object-centered 3D generation. The horizontal orientation follows the azimuth of a designated anchor image, while the vertical axis remains aligned with the canonical upright direction. Canonical object coordinates are therefore rotated around the vertical axis according to the anchor-view azimuth.

This design differs from both a fully camera-relative representation and a fixed canonical representation. A first-camera coordinate system is naturally suited to feed-forward reconstruction but varies with the observation configuration. A canonical space is convenient for native 3D generation but may not directly encode the orientation implied by the input views. The anchor-guided space instead provides a stable object-centered reference whose horizontal orientation is tied to an actual input observation.

The practical consequence is that point maps and sparse voxel structures are supervised in the same coordinates. Reconstructed 3D points can therefore be associated with generated voxels without predicting a separate similarity transformation. This is important because the correspondence is used not merely for visualization but as an internal mechanism for conditioning detailed shape generation.

Bidirectional joint reconstruction and generation

The joint model combines a VGGT reconstruction branch with a diffusion-style sparse-structure generation branch initialized from TRELLIS.2. DINOv3 image features provide visual and semantic conditioning, while VGGT features provide cross-view geometric information.

The first interaction direction is reconstruction-to-generation. Multi-level VGGT features are spatially aligned with DINOv3 patch features through learned adapters. The resulting residual features are added to the DINOv3 tokens and concatenated across views. Generation tokens then attend to these features, allowing sparse-structure prediction to use estimated point geometry and multi-view relationships rather than relying only on image appearance.

The second direction is generation-to-reconstruction. After generation blocks update the sparse-structure features, reconstruction tokens attend to them through gated cross-attention. The gate is conditioned on the diffusion noise level, so the amount of injected generative information can vary during denoising. This allows the reconstruction branch to use global shape context when estimating the 3D positions of image pixels.

The branches are optimized jointly using latent flow-matching loss for sparse-structure generation and point-map losses for reconstruction. The point-map objective includes robust point regression, confidence estimation, and multi-scale normal consistency. The paper’s design thus treats reconstruction and generation as mutually informative predictions rather than as a fixed sequence of independent estimators.

The ablation results support this interaction mechanism. On GSO, DINO-only conditioning achieves sparse-structure IoU of 0.4261 and Chamfer distance of 2.0019 times $10^{-2}$. Adding VGGT features improves these values to 0.5533 and 1.1710, respectively. Full anchor-guided joint training further improves them to 0.5827 and 1.0563. The result supports the claim that geometric reconstruction features are useful to generation and that bidirectional optimization provides an additional benefit beyond one-way conditioning.

The coordinate-space comparison is also consequential. Canonical-space joint training obtains IoU 0.4601 and Chamfer distance 1.5825 times $10^{-2}$, while cross-space training obtains IoU 0.4563 and Chamfer distance 1.6291 times $10^{-2}$. Both are substantially below the anchor-guided configuration. This supports the paper’s stronger architectural claim that the coordinate system itself, not only the presence of cross-branch features, determines the effectiveness of joint modeling.

Structure-guided reconstruction and voxel-wise view selection

The second major component uses the generated sparse structure to improve reconstruction. The sparse voxel centers are encoded with Fourier positional features, and reconstruction tokens attend to these structure features. The resulting point maps are expressed in the same space as the voxel structure, enabling direct quantization of predicted 3D points onto the voxel grid.

For each view and voxel, the method retains high-confidence foreground pixels and computes a support score based on the number of pixels in the largest connected component mapped to that voxel. The view with the largest support is selected for that voxel. Unsupported voxels inherit the selected view of a supported 26-neighborhood voxel; if no neighboring support exists, the anchor view is used.

This selection rule is intentionally local. It does not assume that one image is globally optimal for the entire object. A surface may be directly visible in one view, oblique in another, and occluded in a third. The method therefore assigns visual evidence at voxel resolution, using pixel support as a proxy for the reliability and informativeness of each observation.

Detailed shape generation alternates between global multi-view attention and voxel-specific selected-view attention. Global attention exposes every voxel to all input images and supports shape completion. Selected-view attention restricts each voxel’s local conditioning to the image judged most informative for its spatial region. This separation is intended to prevent unrelated image evidence from being mixed into local feature representations.

The ablation isolates the contribution of this mechanism while holding the predicted sparse structure fixed. Full multi-view conditioning produces normal-rendering PSNR of 26.892, SSIM of 0.9649, LPIPS of 0.0401, Chamfer distance of 1.4739 times $10^{-2}$, and F-score of 0.8758. Voxel-wise selection improves these values to 27.207, 0.9657, 0.0388, 1.4047 times $10^{-2}$, and 0.8897. The improvement is modest in perceptual metrics but consistent across all reported measures, indicating that region-specific conditioning primarily benefits local geometry and detail fidelity.

Structure conditioning also improves point-map reconstruction. On GSO, image-only VGGT obtains accuracy 5.025, completeness 4.667, and overall error 4.846, with distances reported in normalized coordinates scaled by $10^{-3}$. Joint training reduces these to 4.053, 4.250, and 4.152. Explicit structure-conditioned VGGT reduces them further to 3.539, 3.086, and 3.313. The larger improvement in completeness is consistent with the intended role of the generated structure: it supplies spatial context that helps reconstruction infer coherent geometry beyond directly visible evidence.

Quantitative evaluation

Training uses 210,000 objects from a subset of TRELLIS-500K. Evaluation uses 1,000 held-out objects drawn from Toys4K and GSO, with four randomly selected unposed views per object. The model is compared with Stream3D combined with TRELLIS.2, Hunyuan3D-2MV, MV-SAM3D, ReconViaGen, and Mix3R. Appearance is evaluated with PSNR, SSIM, and LPIPS; geometry is evaluated with Chamfer distance and F-score.

UniForge3D reports the best result for every metric on both datasets. On Toys4K, it achieves PSNR 27.428, SSIM 0.9616, LPIPS 0.0401, Chamfer distance 1.4196 times $10^{-2}$, and F-score 0.8899. Mix3R, the strongest listed competitor on several geometry and appearance measures, achieves PSNR 23.907, SSIM 0.9498, LPIPS 0.0506, Chamfer distance 2.3445 times $10^{-2}$, and F-score 0.7343. Thus, relative to Mix3R, UniForge3D improves PSNR by 3.521 dB, reduces LPIPS by 0.0105, reduces Chamfer distance by approximately 39.4%, and increases F-score by 0.1556.

On GSO, UniForge3D achieves PSNR 24.826, SSIM 0.9363, LPIPS 0.0591, Chamfer distance 1.4047 times $10^{-2}$, and F-score 0.8897. Mix3R obtains 21.040, 0.9239, 0.0688, 2.3202 times $10^{-2}$, and 0.7174, respectively. The corresponding improvements are 3.786 dB in PSNR, 0.0124 in SSIM, 0.0097 lower LPIPS, approximately 39.5% lower Chamfer distance, and a 0.1723 increase in F-score.

These results substantiate the paper’s principal claim that the method improves both completion and input fidelity rather than trading one for the other. However, the evaluation protocol aligns generated meshes to ground truth with a GICP-based Sim(3) transformation before computing the principal appearance and geometry metrics. This makes the reported scores appropriate for shape and rendering comparison after registration, but it does not by itself measure absolute pose accuracy in the original input coordinate system. The shared-space visualizations and point-map experiments provide additional evidence of alignment, but the aggregate benchmark numbers should be interpreted in light of this registration step.

Implementation and modeling choices

The method is assembled from pretrained components rather than trained from scratch. The joint reconstruction-generation stage uses TRELLIS.2 and VGGT, with LoRA adaptation ranks of 64 and 32 and fully trained interaction modules. DINOv3 remains frozen. The structure-conditioned reconstruction stage fully fine-tunes VGGT, after which it is frozen for subsequent shape and texture generation. The 512- and 1024-resolution shape generators and the texture generator use rank-128 LoRA adaptation.

The staged design separates coarse structure prediction, structure-conditioned detailed shape generation, and texture synthesis. This is computationally substantial: training is performed on 32 NVIDIA A800 GPUs, with multiple 60,000-iteration stages. The use of pretrained models and parameter-efficient adaptation reduces the number of trainable parameters, but the system remains a multi-stage pipeline with significant infrastructure requirements.

The texture stage conditions a texture diffusion model on the final shape latent while keeping the shape fixed. It predicts base color, metallic, roughness, and opacity attributes that are baked into UV textures. This extends the shared-space and view-selection strategy beyond geometry, although the reported ablations focus primarily on shape and normal-map fidelity rather than isolating texture-specific improvements.

Limitations and open questions

The paper explicitly states that training is conducted on synthetic renderings. Severe occlusion, strong reflections, and extreme lighting remain challenging. This limitation directly affects the interpretation of the real-world qualitative results: although the method generalizes visually to real images, the training distribution does not fully represent the appearance variation encountered in unconstrained captures.

The view-selection score is also a heuristic. Connected-component pixel support measures the quantity of predicted evidence assigned to a voxel, but it does not directly estimate visibility correctness, photometric reliability, or geometric uncertainty. A view can provide many pixels while still being affected by reflection, segmentation error, or inaccurate point-map prediction. The method’s confidence filtering partially addresses this issue, but the paper does not report calibration analyses or comparisons with uncertainty-aware view selection.

The evaluation further leaves several questions open. The experiments use four input images, so robustness as the number of views varies is not established. The paper does not quantify inference cost relative to competing methods, despite the multiple reconstruction and generation stages. It also does not isolate how performance changes under incorrect anchor selection or under substantially non-upright objects, where preserving a canonical vertical direction may be less appropriate. Finally, because the main metrics use Sim(3) alignment, the extent to which the anchor-guided space improves absolute pose consistency rather than post-alignment reconstruction quality remains incompletely measured.

Conclusion

UniForge3D presents a coherent integration of reconstruction and generation for sparse, unposed multi-view 3D asset creation. Its central technical decision is to place both branches in an anchor-guided shared space, enabling direct geometric interaction without an additional cross-space alignment module. Bidirectional feature exchange improves sparse-structure prediction and point-map estimation, while structure-conditioned voxel-wise view selection supplies localized visual evidence for detailed shape generation.

The reported results are strong across both Toys4K and GSO: the method achieves the best appearance and geometry scores among the compared systems, with Chamfer distance near 1.41 times $10^{-2}$0 and F-score near 0.89 on both datasets. The ablations further indicate that shared coordinates, joint bidirectional modeling, explicit structure conditioning, and voxel-wise view selection each contribute materially to performance. The principal unresolved issue is robustness outside the synthetic training distribution, particularly under severe occlusion, reflections, and extreme illumination (2610.15357).