Multi-View Reconstruction (MVR)

• Multi-view reconstruction is a technique that estimates 3D geometry from multiple images or views by enforcing cross-view consistency.
• It integrates geometric, photometric, and learned cost volume methods to refine models and overcome ambiguities present in single observations.
• Applications span anomaly detection, non-rigid motion recovery, and panoramic scene reconstruction, each using specialized cues and optimization frameworks.

Multi-view reconstruction (MVR) denotes the estimation of 3D geometry or a view-consistent scene representation from multiple observations of the same object, room, scene, or point cloud. In the surveyed literature, the term covers calibrated multi-view stereo, variational shape-from-shading, inverse rendering, non-rigid deformation recovery, panorama-based layout estimation, transformer-based object reconstruction, and point-cloud-to-multi-view image conversion for anomaly detection. The common principle is that multiple views impose cross-view constraints that reduce the ambiguity of any single observation, whether the unknown is a depth map, occupancy grid, plane field, mesh, point cloud, or latent shape representation (Guo et al., 18 Mar 2025, Quéau et al., 2017, Wang et al., 2021, Sun et al., 29 Jul 2025).

1. Geometric foundations and camera models

A large part of MVR begins with calibrated imaging geometry. In the calibration-oriented formulation of Chaudhury et al., each camera matrix is recovered by a DLT+SVD routine, decomposed as $M = K[R \mid t]$, and then transferred from a local coordinate system into a common global coordinate system so that all cameras are registered in one world frame. Their global composition is written as $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$, and the paper emphasizes a calibration setting in which calibration-pattern points yield identical computed image coordinates across cameras after alignment (Chaudhury et al., 2010).

Classical dense stereo further relies on epipolar rectification so that correspondence search becomes one-dimensional. Elhashash and Qin reformulate rectification on a virtual sphere rather than a plane, using longitude–latitude coordinates

$$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$

followed by dense matching on spherical panoramas. On the Dortmund and Bordeaux aerial datasets, the spherical model improves completeness by up to $4.05\%$ and $3.70\%$, and improves accuracy by up to $10.23\%$ and $7.70\%$, respectively, relative to frame-based epipolar correction with LiDAR as ground truth (Elhashash et al., 2022).

Recent formulations also write MVR directly as a bidirectional mapping between 3D points and multi-view images. In the anomaly-detection method titled “Multi-View Reconstruction with Global Context for 3D Anomaly Detection,” a point cloud $P \in \mathbb{R}^{N \times 3}$ is projected into virtual views by

$$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$

and mapped back by

$$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$

This explicit forward/inverse projection is later used to fuse multi-view 2D features back onto 3D points (Sun et al., 29 Jul 2025).

2. Variational, photometric, and non-rigid formulations

A standard geometric MVR pipeline often begins with structure-from-motion and multi-view stereo, then refines geometry with additional image cues. “Polarimetric Multi-View Inverse Rendering” starts from a standard SfM stage on unpolarized RGB images, applies an MVS back end such as OpenMVS to produce a dense point cloud and an initial triangular mesh, optionally performs $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$0-subdivision, and then jointly optimizes photometric rendering error, polarimetric normal error, and smoothness priors over vertex positions, albedos, and illumination parameters (Zhao et al., 2022).

Qu et al. explicitly formulate “Dense Multi-view 3D-reconstruction Without Dense Correspondences,” replacing dense matching with a variational coupling of single-view shape-from-shading PDEs across views and color channels. Their energy combines least-squares single-view SFS terms with sparse multi-view geometric constraints and is optimized by ADMM. On synthetic binocular MVR, they report single-view to binocular improvements from MAE-N $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$1 to $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$2 in grey and from $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$3 to $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$4 in color, and they further show real reconstructions on “Sokrates” and “Figure” with fine detail using only sparse matches (Quéau et al., 2017).

NRMVS generalizes the setting to scenes with non-rigid motion observed from arbitrary, sparse, and wide-baseline RGB views. Its core objective is a weighted sum of a sparse 3D–3D correspondence term, a dense photometric consistency term, and an as-rigid-as-possible deformation-graph regularizer. The method assumes at least two canonical views with only small non-rigid change, enough static background for SfM camera recovery, and moderate deformations represented by a deformation graph. With $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$5 images and $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$6 graph nodes, the reported total preprocessing, deformation, and depth runtime is approximately $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$7 hours on a $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$8-core CPU plus GTX $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$9 Ti, with deformation dominating roughly $$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$0 of the CPU time (Innmann et al., 2019).

These formulations make clear that MVR is not restricted to a rigid, purely correspondence-driven stereo pipeline. In the surveyed work, multi-view coupling can arise from shading, sparse geometry, deformation priors, or joint inverse-rendering objectives rather than only from dense image matching.

3. Learned cost volumes, guided stereo, and panoramic reconstruction

Modern learned MVR frequently centers on cost-volume design. AACVP-MVSNet introduces a coarse-to-fine depth inference strategy in which the coarsest level estimates a full depth map and finer levels regress pixel-wise residuals inside narrowed search ranges. At each level, the network uses self-attention in the feature extractor and group-wise similarity rather than feature variance for cost-volume construction. On DTU, it reports Accuracy $$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$1 mm, Completeness $$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$2 mm, and OA $$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$3 mm, surpassing CasMVSNet’s $$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$4 mm OA; the best OA reported is $$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$5 mm when training on $$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$6 views and evaluating on $$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$7 views (Yu et al., 2020).

Multi-View Guided Multi-View Stereo modifies the plane-sweep volume itself by injecting sparse depth hints. The guidance is aggregated from multiple views onto the reference frame, filtered for occlusion, and used to modulate the cost volume at every stage of a backbone MVS network. The framework is reported with MVSNet, D$$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$8HC-RMVSNet, CAS-MVSNet, UCSNet, and PatchMatchNet. On DTU, for example, CAS-MVSNet improves from Acc. $$\theta = \arctan2(X_c,Z_c), \qquad \phi = \arctan2(Y_c,\sqrt{X_c^2 + Z_c^2}),$$9, Comp. $4.05\%$0, Avg. $4.05\%$1 to $4.05\%$2, $4.05\%$3, $4.05\%$4, while UCSNet improves from $4.05\%$5, $4.05\%$6, $4.05\%$7 to $4.05\%$8, $4.05\%$9, $3.70\%$0 after guidance (Poggi et al., 2022).

Panoramic MVR changes both feature extraction and sweeping geometry. 360Recon is an MVS algorithm for ERP images that uses SphereCNN-style tangent-plane sampling, spherical sweeping to build a 3D cost volume, an MLP to reduce channel dimension, a U-Net-style encoder–decoder for depth estimation, and ERP-adapted TSDF fusion. On Matterport3D, it reports Completeness $3.70\%$1 cm, Accuracy $3.70\%$2 cm, Chamfer $3.70\%$3 cm, and F-Score $3.70\%$4, improving substantially over BiFuse++, PanoFormer, FoVA-Depth, and 360-MVSNet; for depth estimation on the same dataset it reports MAE $3.70\%$5 cm, MRE $3.70\%$6, RMSE $3.70\%$7 cm, and $3.70\%$8 (Yan et al., 2024).

Taken together, these systems show that learned MVR is often a problem of how to parameterize the search space: coarse-to-fine residual intervals, sparse-depth-guided hypotheses, or spherical rays for panoramic cameras. This suggests that cost-volume engineering remains central even when the downstream regressor is fully learned.

4. Object-centric, generative, and monocular-guided reconstruction

Object-centric MVR has also been formulated around explicit multi-view representations. MVPNet represents a 3D surface as the union of $3.70\%$9 dense view-dependent point clouds embedded on regular $10.23\%$0 image-plane grids, called a multi-view point cloud (MVPC). Because each grid can be triangulated cell-wise, the representation is mesh-friendly as well as convolution-friendly. On ShapeNet-Chair it reports voxel-IoU $10.23\%$1, and on ShapeNet-13 it reports mean IoU $10.23\%$2 and Chamfer distance $10.23\%$3, with better thin structures and concavities than prior volumetric or point-only baselines (Wang et al., 2018).

The 3D Volume Transformer (VolT) recasts multi-view 3D reconstruction as a sequence-to-sequence problem in which a Transformer encoder jointly performs 2D-view feature extraction and view fusion, and a Transformer decoder maps spatial 3D queries to an occupancy grid. Its divergence-enhanced variant, EVolT, reports IoU $10.23\%$4 and F-score@$10.23\%$5 $10.23\%$6 on $10.23\%$7-view ShapeNet reconstruction, compared with IoU $10.23\%$8 and F-score $10.23\%$9 for Pix2Vox++/A. The reported parameter counts are $7.70\%$0M for VolT and $7.70\%$1M for EVolT, versus $7.70\%$2M for Pix2Vox++/A and $7.70\%$3M for Pix2Vox-A (Wang et al., 2021).

Wei et al. propose a different interpretation: single-view shape prediction should be conditional rather than deterministic because the back side is occluded. Their model samples shapes from $7.70\%$4, where $7.70\%$5 is a $7.70\%$6-D Gaussian latent, and then formulates multi-view reconstruction as the intersection of single-view shape spaces. Training uses a front constraint, a diversity constraint, and a latent-space GAN prior; test-time multi-view inference freezes $7.70\%$7 and optimizes per-view latents by minimizing pairwise consistency. On ShapeNet chairs with $7.70\%$8 views, the reported Chamfer Distance $7.70\%$9 is $P \in \mathbb{R}^{N \times 3}$0 for the conditional model, compared with $P \in \mathbb{R}^{N \times 3}$1 for PSGN, $P \in \mathbb{R}^{N \times 3}$2 for Lin et al., and $P \in \mathbb{R}^{N \times 3}$3 for 3D-R2N2, and the error drops steadily from $P \in \mathbb{R}^{N \times 3}$4 with $P \in \mathbb{R}^{N \times 3}$5 view to $P \in \mathbb{R}^{N \times 3}$6 with $P \in \mathbb{R}^{N \times 3}$7 views (Wei et al., 2019).

Several recent systems explicitly combine monocular priors with multi-view geometry. Murre replaces classical cost-volume MVS with SfM-guided monocular depth estimation: sparse SfM points are projected into each image, densified by $P \in \mathbb{R}^{N \times 3}$8-nearest-neighbors interpolation into a depth prior and distance map, and then used to condition a Stable Diffusion V2 depth model. The predicted depths are RANSAC-aligned to sparse SfM depths and fused by TSDF. Reported results include DTU Chamfer improving from MVSNet $P \in \mathbb{R}^{N \times 3}$9 mm to Murre $$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$0 mm, ScanNet F-score from $$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$1 to $$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$2, Replica F-score from $$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$3 to $$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$4, and Waymo RMSE from $$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$5 m to $$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$6 m (Guo et al., 18 Mar 2025).

Mono3R takes a matching-based multi-view reconstruction foundation model, DUSt3R, and adds a monocular branch based on MoGe plus a ConvGRU refinement module. The monocular pointmaps are first aligned to DUSt3R pointmaps by a weighted Sim(3) fit, then iteratively fused with pairwise features and confidences. On 7Scenes, the reported pose metric $$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$7 improves from $$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$8 to $$[ u_k^i , v_k^i , 1 ]^T = (1/z_i)\, K\, T_k\, [ x_i , y_i , z_i , 1 ]^T,$$9; on NeuralRGBD it improves from $$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$0 to $$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$1; on DTU, Comp-Mean improves from $$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$2 mm to $$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$3 mm (Li et al., 18 Apr 2025).

MVBoost uses yet another hybrid loop: a single-view input is expanded into multiple views by a multi-view diffusion model, converted into a consistent 3D Gaussian Splatting representation by a large 3D reconstruction model, rendered back to images, and refined into pseudo-ground-truth multi-view data for training a fast feed-forward reconstructor. On GSO, the reported novel-view fidelity is PSNR $$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$4, SSIM $$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$5, LPIPS $$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$6, and the geometry metrics are CD $$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$7 and F-Score $$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$8 (Liu et al., 2024).

5. Specialized cues and structured-scene reconstruction

Some MVR methods target strong scene structure rather than arbitrary geometry. PlaneMVS decouples plane reconstruction into a semantic plane detection branch and a plane MVS branch that replaces fronto-parallel depth hypotheses with slanted plane hypotheses $$P^i = Pos_k(I_k^i) = R_k^{-1}\!\left[z_i\,K^{-1}[u_k^i,v_k^i,1]^T - t_k\right].$$9. The association between instance masks and per-pixel plane parameters is handled by a soft-pooling loss. On ScanNet, PlaneMVS-final reports AbsRel $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$00, SqRel $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$01, RMSE $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$02, RMSE$M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$03 $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$04, and $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$05, together with improved plane detection AP over PlaneRCNN (Liu et al., 2022).

MVLayoutNet is specialized for indoor panoramas and room layout. It combines a monocular layout network with a layout-guided MVS module whose key construct is a “layout cost volume” that aggregates multi-view costs at the same depth layer into layout elements. An attention-based scheme further suppresses clutter and focuses estimation on structural regions. On 2D-3D-S it reports depth RMSE $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$06 m, scale error $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$07 m, and coherency $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$08 m; on ZInD it reports $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$09 m, $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$10 m, and $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$11 m, respectively (Hu et al., 2021).

Polarimetric MVIR extends MVR with angle of polarization (AoP) and degree of polarization (DoP). Because one AoP measurement yields four ambiguous azimuth candidates,

$M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$12

the method defines a polarimetric cost around the minimum angular residual and weights it by DoP as a reliability signal. On synthetic data, the reported geometric accuracy and completeness are approximately $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$13 and $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$14 for OpenMVS, $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$15 and $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$16 for MVIR without polarimetry, and $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$17 and $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$18 for Polarimetric MVIR, with qualitative improvements on real objects such as a toy car, an SLR camera, and an outdoor statue (Zhao et al., 2022).

Weakly supervised domain-specific MVR has also been developed for faces. DF-MVR uses three view-specific encoders, a shared decoder with attention gates, a face mask mechanism based on parsing, and a Basel Face Model parameter regressor. Without 3D annotation, it reports RMSE improvements of $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$19 on Pixel-Face and $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$20 on Bosphorus relative to existing weakly supervised MVRs (Zhao et al., 2022).

6. Scope, evaluation regimes, and recurring limitations

The scope of MVR has widened beyond 3D geometry estimation from RGB cameras. In industrial anomaly detection, “Multi-View Reconstruction with Global Context for 3D Anomaly Detection” converts high-resolution point clouds into $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$21 rendered views, processes them with a teacher–student Dinomaly-style ViT hierarchy, fuses features back onto 3D points, and uses a hard-mining cosine loss for anomaly scoring. On Real3D-AD, it reports $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$22 object-wise AU-ROC and $M_i^{(glob)} = K_i [R_i R \mid R_i T + t_i]$23 point-wise AU-ROC (Sun et al., 29 Jul 2025).

A different non-geometric branch is represented by “Robust image reconstruction from multi-view measurements,” where each observation is modeled as a transformed common background plus a view-specific foreground accounting for occlusions. The unknown images and transformation parameters are estimated jointly by a non-convex alternating descent scheme with convergence to a critical point under Kurdyka–Łojasiewicz assumptions. The method is demonstrated on robust alignment, compressed sensing, and multi-frame super-resolution rather than dense 3D geometry (Puy et al., 2012).

A common simplification is to equate MVR with dense, rigid, correspondence-based stereo on pinhole images. The literature surveyed here explicitly contradicts each part of that simplification: Qu et al. solve dense multi-view 3D reconstruction without dense correspondences (Quéau et al., 2017); NRMVS addresses non-rigid scenes from arbitrary, sparse, and wide-baseline views (Innmann et al., 2019); 360Recon and spherical epipolar rectification target ERP panoramas and oblique aerial imagery rather than conventional fronto-parallel pinhole views (Yan et al., 2024, Elhashash et al., 2022); and Murre, Mono3R, and Wei et al. integrate monocular or generative priors instead of relying solely on matching (Guo et al., 18 Mar 2025, Li et al., 18 Apr 2025, Wei et al., 2019). This suggests that MVR is better understood as a family of cross-view consistency mechanisms than as a single algorithmic template.

At the same time, recurring constraints remain visible across the literature. Polarimetric MVIR requires a good initial MVS mesh and assumes low-frequency lighting and sufficiently smooth surfaces (Zhao et al., 2022). MVLayoutNet relies on correct 2D layout extraction and does not handle complex non-Manhattan geometry or curved walls (Hu et al., 2021). VolT inherits quadratic self-attention scaling in the number of views and 3D queries (Wang et al., 2021). 360Recon notes that TSDF cannot hallucinate unseen regions and that larger scenes raise memory concerns (Yan et al., 2024). NRMVS assumes canonical views with only small non-rigid change and enough static background for SfM (Innmann et al., 2019). These limitations indicate that MVR remains highly representation-dependent: gains in one regime—planar structure, panoramic geometry, monocular priors, or polarimetric cues—usually come with assumptions that define where the reconstruction remains well posed.