SyncFix: Fixing 3D Reconstructions via Multi-View Synchronization

Abstract: We present SyncFix, a framework that enforces cross-view consistency during the diffusion-based refinement of reconstructed scenes. SyncFix formulates refinement as a joint latent bridge matching problem, synchronizing distorted and clean representations across multiple views to fix the semantic and geometric inconsistencies. This means SyncFix learns a joint conditional over multiple views to enforce consistency throughout the denoising trajectory. Our training is done only on image pairs, but it generalizes naturally to an arbitrary number of views during inference. Moreover, reconstruction quality improves with additional views, with diminishing returns at higher view counts. Qualitative and quantitative results demonstrate that SyncFix consistently generates high-quality reconstructions and surpasses current state-of-the-art baselines, even in the absence of clean reference images. SyncFix achieves even higher fidelity when sparse references are available.

• The paper introduces a joint multi-view generative refinement method that synchronizes noisy 3D reconstructions for improved stability and fidelity.
• It employs latent bridge matching with permutation-invariant cross-view attention to enforce semantic and geometric consistency during refinement.
• Experimental results show significant gains in PSNR, LPIPS, and cross-view consistency across various sparse-view reconstruction scenarios.

SyncFix: Multi-View Consistency for 3D Reconstruction Refinement

Introduction

SyncFix addresses a core deficiency in prevailing 3D reconstruction refinement pipelines, particularly in scenarios with sparse or off-trajectory input views: most methods improve individual 2D renderings using diffusion-based priors but ignore cross-view geometric and semantic constraints intrinsic to the underlying 3D scene. This omission often leads to hallucinated features and geometric artifacts that are inconsistent across viewpoints, degrading the stability and fidelity of the distilled 3D model. SyncFix introduces a deterministic multi-view joint conditional generative modeling approach—implemented via latent bridge matching with permutation-invariant cross-view attention—that enforces semantic and geometric consistency during the refinement process itself, not after the fact.

Problem Context and Motivation

Reconstruction artifacts like floaters, discontinuous surfaces, and high-frequency texture smearing are ubiquitous when applying methods such as NeRF or 3D Gaussian Splatting (3DGS) in the sparse-view regime. Recent solutions have adapted 2D diffusion priors for per-view image refinement; however, these pipelines treat each rendered view in isolation, neglecting the couplings necessary for enforcing 3D-consistent corrections. This failure manifests especially in ambiguous or underconstrained regions, leading to spatially inconsistent corrections that destabilize subsequent 3D optimization or fusion steps. Prior attempts at incorporating cross-view information have typically involved only limited conditioning, or used iterative denoising-based frameworks with weak view-coupling mechanisms.

The main innovation in SyncFix is the formulation of refinement as joint generation over multiple views, explicitly modeling the joint conditional distribution $P(X_{GT}|X_D)$ in the latent space of a pre-trained autoencoder. This contrasts fundamentally with prior methods' marginal modeling approach, which decouples the refinement task into per-view conditionals. By enabling information exchange across views via multi-view cross-attention, SyncFix synchronizes both geometry and semantics prior to decoding and distillation.

Methodology

SyncFix encodes a set of $N$ distorted rendered images $X_D$ and (when available) sparse clean reference renderings $X_{GT}$ into a shared latent space using a frozen autoencoder. Refinement is then posed as learning a deterministic flow (via latent bridge matching) from the noisy input latent distribution $Z_D$ to the clean distribution $Z_{GT}$, with the velocity field conditioned on the set of all input views. Model supervision is applied both in the latent space (mean squared velocity regression) and image space (pixel and perceptual losses, including LPIPS and Gram matrix-based style losses).

Multi-view attention mechanisms in the Transformer architecture are modified such that each latent token attends globally across all view tokens, subject to appropriate positional encoding. This design ensures permutation invariance and scale-independence with respect to the number and order of input views. Notably, SyncFix is trained on view pairs (N=2) but generalizes seamlessly to sets of arbitrary cardinality at test time, due to this exchangeable structure.

A defining feature of SyncFix is that no explicit camera pose information, calibrated depth maps, or epipolar constraints are utilized during training or inference. The multi-view synchronization is induced solely through learned attention in the coupled latent space.

Experimental Results and Analysis

Quantitative evaluation is performed on DL3DV, Nerfbusters, and MipNeRF 360 datasets, with renderings produced by both optimization-based (3DGS) and feedforward (AnySplat) splatting pipelines. SyncFix is compared against 3DGS baselines, Fixer, and state-of-the-art per-view generative refinement methods such as Difix3D+.

Key Results

• On DL3DV, SyncFix achieves 0.77 dB higher PSNR and a 0.038 absolute reduction in LPIPS compared to Difix3D+, and reduces DreamSim scores by 27% and FID by over 4x relative to the raw 3DGS outputs.
• On Nerfbusters, SyncFix attains a PSNR gain of over 2 dB relative to Difix3D+ and yields the highest cross-view semantic consistency scores (CVSC). The improvement persists even under strong viewpoint splits and when no reference images are available.
• On AnySplat feedforward renderings (MipNeRF 360), SyncFix further increases CVSC from 0.589 (Difix3D+) to 0.686 and reduces LPIPS and DreamSim, demonstrating strong zero-shot generalization to alternative 3D pipelines and scene distributions.

Importantly, SyncFix maintains or improves cross-view semantic consistency in all experimental regimes, including scenarios where per-view refinements degrade 3D coherence by introducing view-dependent hallucinations. Consistency further improves with the number of views jointly provided at inference, plateauing as constraints saturate.

Ablations

• Simply supplying multiple views at inference to a single-view trained model does not yield significant cross-view consistency gain, highlighting the necessity of joint multi-view objectives during training.
• Removing cross-view attention restores view-specific errors, whereas the latent bridge matching framework alone (even in a single-view mode) raises quantitative fidelity relative to prior diffusion-denoising approaches.
• Incorporation of reference views further boosts fine-grained detail fidelity, with perceptual and consistency metrics continuing to improve when additional reference or degraded views are provided.

Implications and Limitations

Practically, SyncFix delivers a highly scalable and robust refinement pipeline for sparse-view 3D reconstruction scenarios, enabling consistent and plausible novel view synthesis and photorealistic scene completion even in ill-posed settings. The automatic enforcement of multi-view consistency directly in the generative process obviates the need for explicit geometric constraints, simplifying integration with arbitrary upstream rendering architectures.

Theoretically, SyncFix demonstrates that direct modeling of joint multi-view conditionals, rather than independent per-view marginals, is essential for reliable 3D scene understanding and generative completion in under-constrained or ambiguous visual domains. The permutation-invariant multi-view attention approach may generalize to other multi-modal or multi-perspective tasks in generative modeling and scene-level inference. Future directions include integrating more powerful or geometrically-aware conditioning, as well as principled development of perceptually and semantically aligned multi-view evaluation protocols.

Limitations include failure cases when view overlap is extremely sparse, as well as plausible hallucinations that may be penalized by pixel-wise metrics, and the influence of priors that may not fully reflect the geometry of the true scene in cases of severe underlying corruption. The method does not leverage explicit pose, depth, or epipolar geometry supervision, which could potentially further enhance generalization and robustness.

Conclusion

SyncFix introduces a significant methodological advancement for 3D reconstruction refinement by enforcing multi-view semantic and geometric consistency during generative denoising via latent bridge matching and cross-attention. This joint modeling approach yields consistent, stable, and high-fidelity multi-view outputs over a range of settings, outperforming previous state-of-the-art methods in both pixel-level and cross-view evaluation metrics. The findings encourage a shift from independent per-view generation protocols toward intrinsically coupled multi-view architectures for scene-level visual generation and understanding.

Reference: "SyncFix: Fixing 3D Reconstructions via Multi-View Synchronization" (2604.11797)