ReconViaGen: 3D Reconstruction via Generation

• The paper introduces ReconViaGen, which combines a feed-forward reconstruction prior with a diffusion-based generative model to produce 3D objects that are both complete and faithful to input images.
• The framework employs cross-view-aware conditioning and rendering-aware velocity compensation during denoising, significantly reducing errors such as holes and oversmoothing, as evidenced by improvements in Chamfer Distance and F-score.
• ReconViaGen integrates VGGT for multi-view feature extraction and TRELLIS for structured latent generation, outperforming reconstruction-only and generation-only baselines in both geometric accuracy and image fidelity.

Searching arXiv for the primary paper and closely related reconstruction-via-generation work. ReconViaGen is a multi-view 3D object reconstruction framework that explicitly combines a strong feed-forward reconstruction prior with a diffusion-based 3D generative prior to produce reconstructions that are both complete and faithful to the input images. It addresses pose-free multi-view 3D object reconstruction from a set of uncalibrated object images $I=\{I_i\}_{i=1}^N$, with the goal of predicting a complete 3D object $O$. Its central claim is that pure reconstruction methods and pure generation methods fail in complementary ways: reconstruction methods preserve image consistency but often leave holes or oversmooth invisible regions, while diffusion-based 3D generators can plausibly complete missing geometry and texture but often drift away from the actual object shown in the inputs (Chang et al., 27 Oct 2025).

1. Problem setting and conceptual framing

ReconViaGen is motivated by sparse real multi-view captures in which overlap is limited and occlusions arise from self-occlusion, support surfaces, or incomplete camera coverage. Under those conditions, standard reconstruction systems, whether classic multi-view stereo, NeRF-like optimization, or newer feed-forward reconstruction models, struggle because visible correspondences become weak or absent. The paper highlights failure modes such as holes, missing object parts, blurred local details, and poor completeness, especially for weak-texture objects and for unseen portions that no image observes directly (Chang et al., 27 Oct 2025).

The method reframes this regime as a controlled generative reconstruction problem. Rather than using a 3D generator only for post hoc completion, ReconViaGen injects reconstruction priors into both the conditioning pathway and the denoising dynamics of a 3D generative model. In the paper’s formulation, this is necessary because accurate reconstruction requires consistency with the specific input object at pixel-level detail, not merely global plausibility. A plausible implication is that ReconViaGen should be read not as a generic image-to-3D generator with stronger conditioning, but as a reconstruction system whose generative component is explicitly constrained by a reconstruction backbone (Chang et al., 27 Oct 2025).

This positioning distinguishes ReconViaGen from later systems that claim a deeper 3D-space unification. UniRecGen, for example, explicitly contrasts itself with ReconViaGen by stating that ReconViaGen injects implicit features from a reconstruction branch into the generative process, whereas UniRecGen provides explicit guidance using intermediate reconstructed geometry (Huang et al., 1 Apr 2026). That comparison is useful because it clarifies what ReconViaGen contributes: it does not canonicalize an explicit 3D scaffold for generator-native conditioning, but it does make a 3D generator reconstruction-aware at both coarse and fine levels.

2. Failure analysis of prior reconstruction and generation pipelines

The paper identifies two reasons why existing diffusion-based 3D generative methods fail to achieve reconstruction-grade consistency. The first is the insufficiency in constructing and leveraging cross-view connections when extracting multi-view image features as conditions. In methods like TRELLIS or Hunyuan3D-2.0-mv, multi-view images are typically encoded with DINO-like image features and then passed to the generative model with weak or simplistic fusion. According to the paper, this is inadequate because the conditioning does not encode strong enough multi-view stereo understanding: correspondences, 3D lifting cues, camera relations, depth implications, and tracking consistency are not explicitly aggregated into the latent conditions (Chang et al., 27 Oct 2025).

The second is the poor controllability of iterative denoising during local detail generation. Diffusion-based generators are stochastic, and fine geometry and texture details emerge gradually through denoising. ReconViaGen argues that, in prior methods, the conditioning signal is too weak to constrain this process tightly enough, especially for local, pixel-sensitive details. The resulting outputs may be plausible but inconsistent with the observations, particularly in edge geometry, texture patterns, and small shape features (Chang et al., 27 Oct 2025).

The ablations support this diagnosis. On Dora-Bench, the proposed global geometry condition reduces Chamfer Distance from $0.144$ in the TRELLIS-M baseline to $0.093$ and increases F-score from $0.843$ to $0.941$. Adding rendering-aware velocity compensation on top of global geometry conditioning and per-view conditioning improves PSNR from $21.045$ to $22.632$, SSIM from $0.905$ to $0.911$, LPIPS from $O$0 to $O$1, and geometry from CD $O$2 to $O$3 and F-score from $O$4 to $O$5 (Chang et al., 27 Oct 2025). This division of labor is central to the method: cross-view-aware conditioning stabilizes coarse structure, while denoising-time correction constrains fine detail.

3. Architecture and information flow

ReconViaGen integrates two pretrained foundations: VGGT as a reconstruction prior and TRELLIS as a generation prior. VGGT provides multi-view reconstruction understanding, including image features, camera parameters, depth, point maps, and tracking cues. TRELLIS provides a 3D generative prior through its Structured LATents (SLAT) representation and coarse-to-fine flow-based generation. The full system has three stages: reconstruction-based conditioning, reconstruction-conditioned generation, and inference-time rendering-aware refinement (Chang et al., 27 Oct 2025).

In the first stage, input images are passed through a fine-tuned VGGT. VGGT uses a DINO-based ViT tokenizer and a 24-layer transformer with alternating frame-wise and global self-attention, producing multi-view 3D-aware features from several depths: $O$6 VGGT is fine-tuned with LoRA adapters on the aggregator while keeping the pretrained geometric priors, with objective

$O$7

From these features, a Condition Net constructs two distinct conditioning token families. For global structure, it constructs a fixed-length global geometry token list $O$8 by iterative cross-attention: $O$9 with $0.144$0. For fine detail generation, it constructs local per-view token lists $0.144$1: $0.144$2 The paper stresses that explicit reconstructions like point clouds are not used as conditions because the features carry richer cues including camera, depth, point maps, and tracking (Chang et al., 27 Oct 2025).

In the second stage, ReconViaGen uses TRELLIS’s two-stage generation pipeline. Sparse structure Flow predicts the active sparse voxel structure $0.144$3 and is conditioned on $0.144$4. It uses TRELLIS’s rectified flow formulation with conditional flow matching: $0.144$5 SLAT Flow then predicts latent vectors for the active sparse voxels,

$0.144$6

which are decoded into the final 3D object

$0.144$7

This fine stage is conditioned on all local per-view tokens $0.144$8. Inside each SLAT transformer block, the noisy SLAT representation cross-attends separately to each view condition and fuses them with learned weights: $0.144$9 This is the mechanism by which local detail synthesis remains multi-view aware (Chang et al., 27 Oct 2025).

Stage	Core operation	Conditioning
Reconstruction-based conditioning	VGGT feature extraction and Condition Net	$0.093$0 for global structure, $0.093$1 for per-view detail
Coarse-to-fine 3D generation	TRELLIS SS Flow and SLAT Flow	Global geometry tokens, then local per-view tokens
Inference-time refinement	Rendering-aware velocity compensation	Refined camera poses and rendering losses

This architecture makes ReconViaGen a direct 3D latent generation system under reconstruction-aware conditioning rather than a view-synthesis-first pipeline. Coarse geometry is represented as sparse voxels in SS Flow, while fine geometry and appearance are represented as structured latents over active voxels in SLAT Flow (Chang et al., 27 Oct 2025).

4. Rendering-aware velocity compensation

The third stage, Rendering-aware Velocity Compensation (RVC), addresses the controllability problem of denoising directly. During later denoising steps, when fine details are formed, the method periodically decodes the current SLAT estimate into a 3D object, renders it from estimated input camera poses, compares the rendered images with the observed inputs, and uses the rendering loss gradient to correct the predicted denoising velocity (Chang et al., 27 Oct 2025).

To make this possible, ReconViaGen first estimates and refines camera poses in the TRELLIS generation space. It renders 30 images from known spherical viewpoints around the generated object, concatenates them with the input images, and feeds all images to VGGT. Because the rendered views have known poses, VGGT can infer coarse input-camera poses in the TRELLIS coordinate system. These are then refined using image matching between rendered and input images, depth maps from the rendered views, and a PnP solver with RANSAC to obtain refined poses $0.093$2 (Chang et al., 27 Oct 2025).

When the denoising time $0.093$3, ReconViaGen decodes the current SLAT latent to a textured object $0.093$4, renders images using the refined poses $0.093$5, and computes an image similarity loss

$0.093$6

Let $0.093$7 be the current predicted clean SLAT latent: $0.093$8 The compensation term is then

$0.093$9

where $0.843$0 abbreviates $0.843$1. The next latent is updated as

$0.843$2

with $0.843$3 in practice (Chang et al., 27 Oct 2025).

This mechanism is inference-only. It does not retrain the generator; instead it modifies the denoising trajectory so that local latent updates reduce rendering error against the actual input images. The paper interprets this as the main answer to poor controllability in iterative generation. A plausible implication is that ReconViaGen views consistency not solely as a property of the learned condition encoder, but as a property that must also be enforced during sampling.

5. Training protocol, benchmarks, and empirical performance

Training and supervision are deliberately modular. For VGGT fine-tuning and TRELLIS sparse structure transformer fine-tuning, the paper uses 390k objects from Objaverse. Each object mesh is rendered from 150 views at $0.843$4, and 60 views per object are used for fine-tuning. VGGT fine-tuning samples $0.843$5 views randomly and uses camera, depth, and point-map supervision. For SS Flow, VGGT is frozen and the Condition Net plus the diffusion transformer are trained (Chang et al., 27 Oct 2025).

Implementation details are unusually specific. LoRA fine-tuning uses rank 64, scaling alpha 128, and zero dropout; adapters are applied to qkv mappings and projector layers in attention modules. VGGT fine-tuning uses AdamW with learning rate $0.843$6. TRELLIS transformer fine-tuning also uses AdamW at $0.843$7, classifier-free guidance drop rate $0.843$8, 8 NVIDIA A800 80GB GPUs, 40k training steps, and batch size 192. At inference, classifier-free guidance strengths are 7.5 for sparse-structure generation and 3.0 for SLAT generation, with 30 and 12 sampling steps respectively (Chang et al., 27 Oct 2025).

Evaluation uses Dora-Bench and OmniObject3D in a deliberately challenging 4-view sparse setting. On OmniObject3D, 24 views are rendered and 4 are chosen as inputs; on Dora-Bench, 40 views are rendered and 4 uniformly spaced views indexed 0, 9, 19, 29 are used. Metrics are PSNR, SSIM, and LPIPS for novel-view image fidelity, and Chamfer Distance plus F-score for geometry accuracy and completeness. CD and F-score are computed from 100k sampled points with normalized coordinates in $0.843$9, and F-score radius $0.941$0 (Chang et al., 27 Oct 2025).

The reported results place ReconViaGen ahead of both reconstruction-only and generation-only baselines. On Dora-Bench it reports

$0.941$1

The closest generative baseline, Hunyuan3D-2.0-mv, has $0.941$2, $0.941$3, and PSNR $0.941$4. TRELLIS-M is markedly worse with $0.941$5, $0.941$6, and $0.941$7. VGGT, which outputs point clouds rather than images, achieves $0.941$8 and $0.941$9 (Chang et al., 27 Oct 2025).

On OmniObject3D, ReconViaGen reports

$21.045$0

compared with TRELLIS-M at $21.045$1, $21.045$2, and VGGT at $21.045$3, $21.045$4 (Chang et al., 27 Oct 2025).

The ablations are equally central. Starting from TRELLIS-M on Dora-Bench, the baseline gives PSNR $21.045$5, SSIM $21.045$6, LPIPS $21.045$7, CD $21.045$8, and F-score $21.045$9. Adding the global geometry condition improves these to $22.632$0, $22.632$1, $22.632$2, $22.632$3, and $22.632$4. Adding local per-view condition yields $22.632$5, $22.632$6, $22.632$7, $22.632$8, and $22.632$9. Adding RVC reaches $0.905$0, $0.905$1, $0.905$2, $0.905$3, and $0.905$4 (Chang et al., 27 Oct 2025). The paper interprets this as showing that global geometry condition is responsible for the biggest jump in shape quality and completeness, local per-view condition improves local view alignment and image fidelity, and RVC supplies the final consistency boost.

6. Interpretation, related directions, and limitations

ReconViaGen matters because it reframes multi-view reconstruction under sparse and imperfect observations as a controlled generation problem rather than a pure correspondence problem. Its key insight is not simply to use a generative prior, but to inject reconstruction priors into both the conditioning and the denoising dynamics of a 3D generator. This places it within a broader lineage of reconstruction with learned priors, from image-space MAP inference under an autoregressive density model in compressive sensing (Dave et al., 2016) to later 3D systems that condition generation on coarse geometric scaffolds (Park et al., 30 Apr 2026, Zadaianchuk et al., 29 Apr 2026).

Within the immediate literature, ReconViaGen occupies an intermediate position. It is more reconstruction-driven than TRELLIS or Hunyuan3D-2.0-mv, because it derives both global and local conditions from VGGT. It is less explicitly geometry-anchored than UniRecGen, which maps reconstruction outputs into a shared canonical 3D space and conditions a generator with canonical point clouds and latent-augmented view features (Huang et al., 1 Apr 2026). This suggests that “reconstruction via generation” in current 3D work spans a spectrum from implicit feature injection to explicit canonical geometric control.

The paper also acknowledges concrete trade-offs. The method depends on strong pretrained priors, specifically VGGT and TRELLIS. It requires pose refinement using rendered intermediate geometry and matching, which adds complexity during inference. Rendering-aware guidance is only applied at later denoising steps and only after camera pose estimation, implying additional runtime overhead. The exact runtime is not reported. Hallucination can still fail in extremely ambiguous cases; the paper does not claim guaranteed correctness of invisible regions, only improved completeness and stronger consistency. The experiments focus on objects rather than arbitrary scenes, and the method relies implicitly on known or recoverable object masks in the object-centered setup (Chang et al., 27 Oct 2025).

A final limitation is methodological rather than numerical. ReconViaGen uses synthetic renderings for large-scale training, and its empirical strength derives partly from that regime. This suggests that out-of-distribution object structure, severe viewpoint ambiguity, or systematic pose-estimation errors remain significant stressors. Even so, the framework establishes a specific technical template for reconstruction-via-generation: VGGT-derived cross-view-aware global and local conditions for coarse and fine stages, coupled with denoising-time rendering constraints that explicitly pull generation back toward the observations (Chang et al., 27 Oct 2025).