GenRec: Knowing Where to Reconstruct and Where to Generate

Abstract: Generative novel view synthesis from sparse input images is rarely all reconstruction or all generation: pixels visible in some source view have a unique correct value modulated only by view-dependent shading, while pixels in disocclusions or beyond the captured volume admit a distribution of plausible completions. Existing generative novel-view-synthesis methods conflate these regimes under a single uniform loss, blurring the line between geometric fidelity and creative hallucinations even when scene geometry is injected through warped point clouds or projected depth. We introduce GenRec, a multi-view flow matching model that builds the reconstruction--generation split directly into its architecture, supervision, and gradient flow. Guided by an observation mask derived from the source cameras and a monocular depth estimator, a flow matching backbone jointly denoises RGB and scene-coordinate maps across all target views, while a pixel-space refinement stage restores high-frequency detail on observed pixels; the same mask gates supervision so regression signals do not contaminate the generative prior. Across RealEstate10K, DL3DV-10K, and Mip-NeRF~360, in both single-view extrapolation and two-view interpolation, GenRec attains the best reconstruction fidelity in observed regions while also surpassing purely generative baselines on perceptual quality in unobserved ones, showing the effectiveness of our approach.

• The paper introduces a mask-guided architecture that separates deterministic reconstruction from uncertain generation, using independent modules, losses, and gradient paths to improve both observed and unobserved regions.
• GenRec achieves 17.05 dB global PSNR and 20.85 dB observed-region PSNR on RealEstate10K, while reaching a best FID of 33.09 and running roughly 100 times faster than Gen3C.
• The results show that gradient isolation and observation-mask gating are both necessary, while the method’s main limitation is its dependence on monocular depth accuracy for reliable warping and region classification.

Motivation and problem structure

GenRec addresses generative novel view synthesis (NVS) from sparse posed inputs by observing that a single synthesized view is rarely uniformly reconstructive or uniformly generative. Pixels visible in at least one source view have a near-deterministic target, modulated only by view-dependent shading; pixels in disocclusions or beyond the captured frustum admit an entire posterior over plausible completions. The authors argue that existing generative NVS methods—whether conditioned on Plücker rays (He et al., 2024, Gao et al., 2024) or on forward-warped depth/point-cloud renders (Yu et al., 2024), [2503.gen3c]—apply a single distribution-matching loss to every pixel, forcing one network to satisfy two structurally different objectives simultaneously. A regression-style signal penalizes fidelity where only one answer exists; conversely, a distribution-matching objective cannot anchor pixels whose value geometry uniquely determines.

The paper's central claim is that this split should be enforced architecturally: separate modules, separate losses, and isolated gradient paths, all governed by a per-pixel observation mask derived from source cameras and monocular metric depth (Depth Anything 3). The result is a model that achieves the best reconstruction fidelity among compared methods in observed regions while also surpassing purely generative baselines on perceptual quality in unobserved ones—and does so roughly two orders of magnitude faster than the strongest baseline (~11 s vs. ~1200 s per scene for Gen3C).

Method

Observation mask and conditioning

Given $N_{\text{src}}$ posed source views and target poses, GenRec runs an off-the-shelf monocular depth estimator, back-projects source pixels, and forward-warps RGB and scene coordinates into each target frame. This yields three precomputed signals per target view: a soft observation mask $O_i$, warped RGB $W_i^I$, and warped scene coordinates $W_i^{SC}$. The mask is the linchpin of the method—it gates architecture, supervision, and gradient flow.

Multi-view flow matching backbone

The backbone is initialized from Stable Diffusion 2.1 and converted to Conditional Flow Matching via the Diff2Flow reparameterization (Liu et al., 2022), which maps $\mathbf{v}$-predictions to flow velocities in closed form while preserving pretrained weights. Two architectural modifications follow SpatialGen: self-attention is rewired into cross-view attention over all $N$ target views for single-pass multi-view consistency, and zero-initialized cross-modal attention couples each view's RGB tokens with its scene-coordinate tokens. Both modalities are denoised jointly in one trajectory; decoded scene coordinates later supply 3D positions for the refinement stage's attention. Conditioning concatenates the noisy latent, Plücker ray embeddings, the area-pooled observation mask, and VAE encodings of the modality-specific warps.

Notably, the backbone generates every pixel—including observed ones—under the flow matching objective. The reconstruction split is not achieved by masking the generative output but by adding a correction pathway whose gradients never reach the backbone.

Pixel-space refinement of observed regions

Two components refine observed pixels without disturbing the generative prior:

Geometry-aware decoder adapter: zero-initialized $1{\times}1$ skip convolutions inject features from the encoded warp before each up-block of the frozen RGB decoder, complemented by LoRA adapters ($r=8$) on decoder convolutions and attention projections. Skip connections copy high-frequency photometric detail that latent compression destroys—a fidelity ceiling the paper attributes to VAE latents, echoing pixel-space NVS diffusion work [2611.elata]. The LoRA weights shift the global output sharpness, so perceptual benefits extend beyond the mask.

Sparse 3D reconstruction branch: a lightweight network predicts a residual $\Delta_i$ added only where $O_i > 0$. It uses sparse cross-attention restricted to $O_i$0 geometrically nearest neighbours, with additive log-priors $O_i$1 derived from inverse 3D distances between predicted target scene coordinates and source points. A second cross-target attention block re-imposes consistency across target frames observing the same surface. FiLM conditioning from the detached temporal embedding lets the branch adapt to flow time. All output projections are zero-initialized, so the branch starts as identity.

Decoupled training

Training proceeds in two phases. Phase one trains only the backbone with the CFM loss. Phase two freezes it and trains the adapter and branch on detached outputs. LPIPS gradients are routed through a masked stop-gradient construction, $O_i$2, preventing regression signals from contaminating disoccluded regions. A subtle but consequential design choice: the refinement stage is supervised on clean-end estimates produced by unrolled multi-step trajectories rather than single-step interpolants, aligning the training distribution with inference-time behaviour where accumulated Euler error is present.

Quantitative results

In single-view extrapolation on RealEstate10K, GenRec reaches 17.05 dB PSNR globally versus 15.24 dB for Gen3C, with observed-region PSNR of 20.85 dB (+2.72 dB) and the best FID (33.09) and LPIPS (0.3220). On DL3DV-10K the margins widen: 19.80 vs. 18.47 dB PSNR, FID 32.36 vs. 44.54. On out-of-distribution Mip-NeRF 360, GenRec leads on every metric despite the domain shift. In two-view interpolation, GenRec attains 19.80 dB PSNR / 0.7043 SSIM on RE10K and leads all baselines on DL3DV-10K (21.99 dB), confirming effectiveness even in the nearly reconstruction-only regime.

Setting	Dataset	Metric	Best baseline	GenRec
Single-view extrap.	RE10K	PSNR (global)	15.24 (Gen3C)	17.05
Single-view extrap.	RE10K	PSNR (observed)	18.13 (Gen3C)	20.85
Single-view extrap.	DL3DV	FID	43.51 (SEVA)	32.36
Two-view interp.	RE10K	PSNR	17.64 (GLD)	19.80
Two-view interp.	DL3DV	PSNR	21.38 (SEVA)	21.99

Against dedicated deterministic reconstruction models in the interpolation setting, GenRec trails DepthSplat by only 0.12 dB PSNR on RE10K while leading on LPIPS on both datasets and leading DepthSplat by +2.20 dB on DL3DV-10K. The authors attribute the residual RE10K gap to squared-error regressors' tendency toward the conditional mean, which PSNR rewards and LPIPS penalizes—an honest framing of the perceptual-fidelity trade-off inherent to generative models.

Zero-shot scaling to four and six input views (never seen in training) yields 23.68 and 24.07 dB PSNR on RE10K, beating Gen3C's natively multi-view cache by 2.9–4.1 dB. More tellingly, GenRec gains 0.39 dB from the fourth to sixth view while Gen3C saturates at 0.06 dB—the decomposition scales with available evidence because more sources raise the observed fraction, exactly the regime the reconstruction branch exploits.

Ablations validating the core claim

The most important ablation isolates the generation–reconstruction split itself on RE10K. Four configurations with matched architecture and parameter count show:

• Applying mask-gated regression directly to the backbone (no branch): 16.51 dB PSNR, FID 36.84.
• Branch without mask gating: FID degrades sharply to 40.43—the weakest generative quality in the study—despite better observed-region fidelity.
• Branch with gating but gradients back-propagated into the backbone through the last sampling step: strictly dominated by the fully decoupled variant on every metric.
• Full decoupled design: 17.05 dB, FID 33.09.

These results support the paper's strongest claim: neither the gating nor the gradient isolation alone suffices, and routing regression gradients into the generative backbone actively harms both objectives even when given additional trainable capacity. A complementary ablation shows joint RGB + scene-coordinate denoising improves every metric over an RGB-only variant, attributed to cross-modal attention forcing 3D-consistent commitments at each denoising step.

Pose-consistency evaluation via VGGT-recovered camera trajectories reveals a dataset-dependent pattern worth noting: GenRec achieves the best ATE/RPE on DL3DV-10K (29–36% reductions), but Gen3C wins on RE10K's narrow-baseline sequences, where reliable forward warps directly favour warp-as-target formulations. The ordering tracks warp quality rather than architecture alone.

Limitations and open questions

The paper concedes several dependencies plainly. First, the entire pipeline inherits the accuracy of its monocular depth front-end: the observation mask and warped conditioning are only as good as DA3's predictions. The depth-corruption sweep quantifies this—affine disparity corruption costs the most (as expected, since it deforms exactly the evidence the reconstruction branch consumes), and in every corruption family the observed-region loss is two to five times the unobserved-region loss, meaning the reconstruction path absorbs depth error while the generative path stays comparatively insulated. When depth is substantially wrong, the mask can label genuinely unobserved pixels as observed, and the branch then confidently reconstructs mis-warped content instead of deferring to the prior. Second, computational constraints limit training to eight views per sample with at most two clean inputs; whether the decoupled design retains its advantage at much larger view counts remains untested beyond the zero-shot four/six-view probe. Third, under very low coverage the model reduces to its backbone, so the split contributes little; and strong view-dependent effects on observed surfaces mean source evidence mismatches the target appearance. An open question the results raise but do not resolve: whether the dataset-dependent pose-consistency ordering against Gen3C reflects a fundamental trade-off or merely the current warp-quality regime of the benchmarks.

Conclusion

GenRec demonstrates that treating generative NVS as a structured composition—per-pixel observation masks gating separate modules, losses, and gradient paths—yields measurable improvements on both sides of the reconstruction–generation divide simultaneously, rather than trading one for the other. The evidence is consistent across three benchmarks, two conditioning regimes, and stratified analyses by observation coverage, and the ablations attribute the gains specifically to architectural decoupling rather than added capacity. The approach's dependence on monocular depth quality is its clearest vulnerability, though the front-end is swappable at inference without retraining.