---
title: 'MonoMVSNet: Hybrid Monocular & Multi-View Stereo'
url: https://www.emergentmind.com/topics/monomvsnet
type: topic
---

# MonoMVSNet: Hybrid Monocular & Multi-View Stereo

MonoMVSNet is a learning-based Multi-View Stereo network that incorporates monocular priors from a pre-trained monocular foundation model into a cascade-style MVSNet pipeline in order to improve depth prediction for calibrated image sequences in regions where feature matching is unreliable, including textureless regions, reflective/specular surfaces, depth-discontinuity edges, and occlusion-heavy regions. The method uses monocular features from the reference view, monocular-depth-guided dynamic depth sampling, and a relative consistency loss based on monocular relative depth; in the main setting, the monocular backbone is Depth Anything V2. The central premise is that monocular depth estimation does not rely on multi-view feature matching and can therefore provide robust relative depth and contextual features in precisely the regions where conventional MVS degrades, while metric reconstruction remains governed by stereo geometry rather than by direct use of monocular depth [2507.11333].

## 1. Problem setting and conceptual position

Multi-View Stereo reconstructs dense depth maps and point clouds from a sequence of calibrated images by matching features across views. Learning-based MVS has improved over traditional methods, but MonoMVSNet is explicitly motivated by failure modes in challenging regions: textureless regions, reflective/specular surfaces, depth-discontinuity edges, and occlusion-heavy regions. In such areas, feature matching becomes unreliable, which degrades cost-volume construction and subsequent depth inference [2507.11333].

MonoMVSNet addresses this by treating monocular estimation as a source of priors rather than as a substitute for geometric stereo. The paper states that modern monocular foundation models trained on large-scale data provide robust semantic/contextual features, strong generalization, and reliable relative depth even where matching fails. At the same time, monocular depth is scale-ambiguous, so it cannot be directly used as metric depth for MVS reconstruction. The method therefore injects monocular outputs into the MVS pipeline in three distinct roles: monocular features strengthen feature extraction and cross-view attention; monocular depth dynamically guides depth sampling, especially near edges; and a relative consistency loss uses monocular relative ordering to supervise predicted depths.

This framing distinguishes MonoMVSNet from two adjacent lines of work. Compared with prior learning-based MVS, it explicitly uses external monocular foundation priors rather than relying only on improved stereo feature extractors, attention modules, transformers, or sampling strategies. Compared with monocular depth methods, it does not use monocular depth as the final reconstruction output, but instead combines monocular robustness with the metric correctness of stereo geometry. A plausible implication is that the method is best understood as a hybridization of monocular priors and calibrated multi-view geometry rather than as a reformulation of either task alone.

## 2. Network architecture and cascade pipeline

MonoMVSNet integrates a pre-trained monocular foundation model into a standard cascade-style MVSNet pipeline. The inputs are \(N\) calibrated images \(\{\mathbf{I}_n\}_{n=0}^{N-1}\), with \(\mathbf{I}_0\) designated as the reference view and the remaining \(N-1\) images treated as source views. Processing is carried out across scales \(s\in\{0,1,2,3\}\) [2507.11333].

At each scale, the pipeline proceeds as follows. First, the network extracts reference features from both an FPN encoder/decoder and the monocular model output. Second, source-view features are enhanced by attention using the reference monocular prior and Cross-View Position Encoding (CVPE). Third, a cost volume is built from the enhanced features. Fourth, the cost volume is regularized with a lightweight 3D UNet. Fifth, depth is predicted from the probability volume. Sixth, monocular depth is aligned and used to update depth candidates, especially at edge pixels. Seventh, training uses cross-entropy at all scales together with a relative consistency loss at the final scale.

The depth prediction module follows the standard MVS pattern: source features are warped to the reference view at sampled depth planes, a correlation cost volume \(\mathbf{V}_s\) is constructed, the volume is regularized by a lightweight 3D UNet, softmax produces a probability volume \(\mathbf{P}_s\), and depth is then predicted with winners-take-all:
\[
\mathbf{D}_s[\mathbf{c}] = d_{i,s} \quad \text{s.t.} \quad i = \arg\max_i \mathbf{P}[i,\mathbf{c}].
\]

This architecture preserves the coarse-to-fine geometry-processing structure of cascade MVS while making monocular priors operational at feature level, candidate-sampling level, and supervision level. The design choice to keep the stereo pipeline intact is significant because it limits the role of monocular estimation to guidance and regularization, which is consistent with the paper’s emphasis on avoiding direct use of scale-ambiguous monocular depth as metric output.

## 3. Monocular feature fusion and Cross-View Position Encoding

The first major component is monocular feature integration. The reference image is passed through the monocular foundation model to produce a monocular feature
\[
\mathbf{F}_0^{mono}\in \mathbb{R}^{C' \times h \times w},
\]
while the same reference image is passed through the FPN encoder to obtain
\[
\mathbf{F}_{0}^{enc}\in \mathbb{R}^{C \times \frac{H}{8} \times \frac{W}{8}}.
\]
The monocular feature is aligned by convolution and bilinear upsampling, then added to the encoder feature:
\[
\mathbf{F}_0 = \mathbf{F}_{0}^{enc} \oplus {\rm Bilinear}({\rm Conv}(\mathbf{F}_{0}^{mono})).
\]
The fused \(\mathbf{F}_0\) is then decoded by the FPN to produce multi-scale reference features [2507.11333].

A notable design decision is that the monocular model is applied only to the reference view. The stated reason is to minimize overhead, and the ablations report that extracting monocular features for all images is less efficient and slightly worse than reference-only monocular fusion. In the reported comparison, “Ref. w/ CVPE” yields Overall 0.278, MAE 4.99, memory 2.01 GB, and time 0.25 s, whereas “All w/ CVPE” yields Overall 0.296, MAE 5.82, memory 5.72 GB, and time 0.56 s.

The second part of this component is CVPE. Standard positional encodings in attention are 2D-image based and do not encode the 3D geometric relation across views. MonoMVSNet introduces Cross-View Position Encoding to improve attention between reference and source features by embedding camera geometry. For the \(n\)-th source view at the coarsest scale \(s=0\), source features are warped into the reference frame using intrinsics and extrinsics:
\[
\mathbf{c}^{i}_{n\rightarrow0,0} = \mathbf{K}_0 \left( \mathbf{R}_{n\rightarrow0}\mathbf{K}_n^{-1}\mathbf{c}_{n,0}\,d_{i,0} + \mathbf{t}_{n\rightarrow0} \right).
\]
The warped feature satisfies
\[
\mathbf{F}_{n\rightarrow0,0}[:,i,\mathbf{p}^{i}_{n\rightarrow0,0}] = \mathbf{F}_{n,0}[:,\mathbf{p}_{n,0}].
\]
Similarly, the reference feature is warped into the source view, giving \(\hat{\mathbf{F}_{0\rightarrow n,0}}\).

The appendix-described embedding pipeline reshapes the warped feature from \(B\times C\times D\times H\times W\) to \(B\times CD\times H\times W\), processes it by Conv \(\rightarrow\) BN \(\rightarrow\) ReLU to \(B\times C\times H\times W\), reshapes camera parameters from \(B\times 4\times 4\) to \(B\times16\), applies BN + MLP to obtain \(B\times C\), broadcasts and adds that embedding to the warped feature, and then applies Squeeze-and-Excitation \(\rightarrow\) Conv layers. The resulting reference and source CVPE codes are added to corresponding features and fed into intra-view and inter-view attention blocks. The final attention output becomes the enhanced source feature at each scale, and this enhancement is propagated through the pyramid.

The intended effect is that source features absorb monocular priors from the reference view while attention remains conditioned on cross-view geometry rather than purely on 2D positional structure. This suggests that MonoMVSNet treats the reference-view monocular representation as a latent scene prior and CVPE as the mechanism that makes this prior geometrically communicable across views.

## 4. Monocular-depth-guided dynamic depth sampling

The second major component is the use of monocular depth to guide candidate sampling. The monocular depth map is denoted
\[
\mathbf{D}^{mono}\in \mathbb{R}^{H\times W}.
\]
Because monocular depth is scale-ambiguous, MonoMVSNet first aligns it to the current MVS prediction. At scales \(s\ge 1\), the previous-stage prediction \(\mathbf{D}_{s-1}\) is used for alignment after both maps are resized to the current resolution. To avoid unreliable coarse predictions, the top \(80\%\) highest-confidence pixels from the previous stage are retained as \(\mathcal{C}_s\). Alignment parameters \(a,b\) are then estimated by least squares on these reliable pixels, and the aligned monocular depth is written as
\[
\mathbf{D}_s^{align}=a\hat{\mathbf{D}^{mono}_s+b.
\]
At scale 0, the monocular depth is directly scaled to the predefined depth range [2507.11333].

Once aligned, monocular depth is used to modify inverse-depth candidates in edge regions. A lightweight edge detector produces an edge confidence map
\[
\mathbf{E}\in \mathbb{R}^{H\times W}.
\]
After resizing to scale \(s\), pixels with confidence above threshold \(\lambda\) define \(\mathcal{C}_s^{edge}\). For each edge pixel \(\mathbf{c}\in\mathcal{C}_s^{edge}\), the method computes absolute differences \(|d_{i,s} - \mathbf{D}_s^{align}[\mathbf{c}]|\), finds the candidate closest to the aligned monocular depth, and replaces that candidate with \(\mathbf{D}_s^{align}[\mathbf{c}]\). For non-edge pixels, the original shared candidate set is kept.

The paper’s interpretation is explicit: using monocular depth everywhere can be harmful, so the edge mask is important to avoid injecting erroneous candidates in non-critical regions. The resulting sampling strategy is described as more locally adaptive at depth discontinuities, more informed by robust monocular relative depth, and less blurry than standard inverse-depth sampling. In context, the edge restriction is central rather than ancillary: it is the mechanism that confines monocular intervention to the locations where conventional shared sampling is most likely to be inadequate.

## 5. Relative consistency supervision

The third major component is a relative consistency loss that enforces agreement between predicted depth ordering and monocular depth ordering. From the cost volume, the network predicts a probability volume \(\mathbf{P}_s\), from which the expected depth is computed as
\[
\mathbf{D}_s^{prob}[\mathbf{c}] = \sum_{i=0}^{D_s} \mathbf{P}[i,\mathbf{c}]\, d_{i,s}.
\]

Two random pixel sets of size \(M\), denoted \(\mathcal{C}_s^1\) and \(\mathcal{C}_s^2\), are sampled. For each pair \((\mathbf{c}_m^1,\mathbf{c}_m^2)\), the method defines
\[
e_m= \left( \mathbf{D}_s^{prob}[\mathbf{c}_m^1] - \mathbf{D}_s^{prob}[\mathbf{c}_m^2] \right) \cdot {\rm Sign}\!\left( \hat{\mathbf{D}^{mono}_s[\mathbf{c}_m^1] - \hat{\mathbf{D}^{mono}_s[\mathbf{c}_m^2] \right),
\]
and the loss
\[
\mathcal{L}^{rc}_s = \frac{1}{M}\sum_{m=0}^{M-1}\max(0,-e_m).
\]
If monocular depth indicates that one pixel should be deeper than another, the predicted depth should preserve that ordering; violations are penalized via hinge loss [2507.11333].

The total training objective is
\[
\mathcal{L}_{overall}=\sum_{s=0}^{3}\mathcal{L}^{ce}_s+\gamma \mathcal{L}^{rc}_3,
\]
where \(\mathcal{L}^{ce}_s\) is standard cross-entropy supervision on the probability volume at each scale, \(\mathcal{L}^{rc}_3\) is applied only at the final scale, and \(\gamma\) is a weighting factor.

This supervision strategy operationalizes a specific compromise between monocular and stereo cues. Monocular depth is not trusted for metric values, but its pairwise ordering is treated as informative. A plausible implication is that the loss regularizes geometry in regions where absolute alignment may remain imperfect yet ordinal structure is still reliable, which matches the broader logic of using monocular depth as a prior rather than as ground-truth replacement.

## 6. Training configuration, datasets, and quantitative results

MonoMVSNet is implemented in PyTorch with Adam. The main monocular backbone is Depth Anything V2. The depth range is 425 mm to 935 mm. The coarse-to-fine depth hypotheses are 8-8-4-4, the depth interval is 0.5-0.5-0.5-0.5, and the group correlations are 8-8-4-4 [2507.11333].

Training and evaluation use three datasets. DTU is described as an indoor object-centric benchmark, BlendedMVS is used for fine-tuning/generalization, and Tanks-and-Temples is an outdoor large-scale benchmark. For DTU training, the setup is 15 epochs, 5 views, resolution \(512\times640\), batch size 4, and learning rate 0.001 halved after epochs 10, 12, and 14. For BlendedMVS fine-tuning, the setup is 15 epochs, 9 views, resolution \(576\times768\), batch size 4, and learning rate 0.001 halved after epochs 6, 8, 10, and 12. Testing uses 5 views at \(832\times1152\) with dynamic fusion for DTU point clouds and 21 views at 2k resolution with dynamic fusion for Tanks-and-Temples.

The reported evaluation metrics are Accuracy, Completeness, and Overall for DTU point clouds; MAE and error ratios at 2 mm, 4 mm, and 8 mm for DTU depth; and official benchmark F-score for Tanks-and-Temples.

| Benchmark | Setting | Reported result |
|---|---|---|
| DTU point cloud | MonoMVSNet (N=5) | Overall 0.278, Acc. 0.313, Comp. 0.243 |
| DTU point cloud | MonoMVSNet (N=9) | Overall 0.275, Acc. 0.302, Comp. 0.248 |
| Tanks-and-Temples | Intermediate mean F-score | 68.63 |
| Tanks-and-Temples | Advanced mean F-score | 43.58 |

On DTU, the paper reports best overall point-cloud error, second-best accuracy, and strong memory efficiency. It also notes qualitative improvements in difficult scenes, especially scan48 and scan77. On Tanks-and-Temples, MonoMVSNet reports an Intermediate mean F-score of 68.63 and an Advanced mean F-score of 43.58, and is claimed to rank first on both the Intermediate and Advanced subsets, which the abstract also presents as state-of-the-art performance.

Efficiency is a recurrent result. The reported inference memory consumption is 2.01 GB, time is 0.25 s, and trainable parameters are 2.89 M. The paper states that this is substantially lower memory than recent transformer-based SOTA methods and that the trainable memory/parameter burden is much less than transformer-heavy methods like MVSFormer++. Within the scope of the reported experiments, the method is therefore positioned not only as accurate but also as comparatively lightweight.

## 7. Ablations, model variants, strengths, and limitations

The ablations are conducted on DTU and directly support the three-component design. For monocular feature components, the baseline yields Overall 0.303 and MAE 6.53; adding the reference monocular feature gives Overall 0.293 and MAE 5.86; adding cross-attention gives a modest improvement; adding CVPE gives a stronger improvement; and the full monocular-feature path reaches Overall 0.281 and MAE 4.99. The interpretation provided in the summary is that geometry-aware positional encoding matters [2507.11333].

For monocular depth components, adding monocular-depth-guided depth sampling helps depth metrics, but using monocular candidates too broadly can hurt. Adding the edge map improves point cloud and depth metrics, showing that edge restriction is beneficial, and adding the relative consistency loss further improves final performance. These observations reinforce the paper’s broader claim that monocular priors are useful when selectively integrated rather than globally imposed.

For foundation model variants, the reported results are as follows:

| Monocular backbone | Overall | MAE |
|---|---:|---:|
| Depth Pro | 0.286 | 5.78 |
| DINOv2 | 0.284 | 5.50 |
| Depth Anything V1 | 0.282 | 5.03 |
| Depth Anything V2 | 0.278 | 4.99 |

Depth Anything V2 performs best among the reported monocular backbones. The number-of-views study shows performance improving from 4 to 9 views and saturating around 9 views: 4 views give Overall 0.2825, 5 views give 0.2780, and 9 views give 0.2750.

The paper’s stated contributions are fourfold: monocular feature prior integration through reference-view fusion and CVPE; monocular-depth-guided sampling via alignment and edge-region candidate updates; relative consistency supervision through a pairwise ranking loss; and strong empirical results, including SOTA on DTU and Tanks-and-Temples, first place on Tanks Intermediate and Advanced, and better efficiency than many recent transformer-based methods.

The stated strengths are that the method uses monocular priors in a principled, low-overhead way; improves especially in edges, textureless regions, and reflective surfaces; is more efficient than methods that require monocular features for all views; works with different monocular foundation models; and generalizes strongly to Tanks-and-Temples. The stated limitations and caveats are that monocular depth still has scale ambiguity and therefore requires alignment, dynamic depth sampling depends on edge detection quality, the method still requires calibrated multi-view inputs and standard MVS compute, generalization beyond benchmark-centered depth reconstruction scenarios is not explicitly established, and some reported equations in the manuscript appear typographically inconsistent although the intended method is clear from context.

A recurrent misconception in this area is that adding monocular depth to MVS amounts to replacing geometric stereo with a single-image prior. MonoMVSNet is explicitly not formulated that way. Its defining claim is narrower and more technical: monocular outputs are useful directly as features, useful after alignment as depth priors, and useful as ordinal supervision, but they are not treated as direct metric reconstruction outputs.

Source: https://www.emergentmind.com/topics/monomvsnet