VFMRecon: Scale-Aligned Neural Reconstruction

• The paper introduces VFMRecon, a technique that resolves scale ambiguity in VFM depth priors to enable consistent, cross-domain scene reconstruction.
• It employs a two-stage pipeline with lightweight scale alignment via feature correspondences and VFM-augmented volumetric fusion to extract dense SDF volumes and meshes.
• Experimental evaluations show significant improvements, notably boosting Tanks and Temples F1 scores from 51.8 to 70.1 through robust scale alignment and depth reprojection.

Searching arXiv for the cited paper and closely related work to ground the article. VFMRecon, introduced as “VFM-Recon: Unlocking Cross-Domain Scene-Level Neural Reconstruction with Scale-Aligned Foundation Priors,” is a scene-level neural volumetric reconstruction method for monocular RGB videos with known camera poses and intrinsics. Given an image sequence $I=\{I_t\}_{t=1..N}$, poses $\mathcal{T}=\{T_t \in SE(3)\}$, and intrinsics $K \in \mathbb{R}^{3 \times 3}$, it reconstructs a dense signed distance function (SDF) volume $\mathcal{V}$ and extracts a mesh $\mathcal{M}$. The method addresses a specific incompatibility between transferable vision foundation model (VFM) priors and volumetric fusion: VFMs such as VGGT provide robust cross-domain geometric priors, but their predictions are scale-ambiguous, whereas scene-level volumetric fusion requires multi-view scale consistency. VFMRecon combines a lightweight scale alignment stage with VFM-augmented neural reconstruction through lightweight task-specific adapters, and it is presented, to the best of the authors’ knowledge, as the first attempt to systematically connect transferable VFM priors with the scale-consistency requirements of scene-level neural volumetric reconstruction under domain shifts (Ming et al., 13 Mar 2026).

1. Problem setting and motivation

Scene-level neural volumetric reconstruction from monocular videos remains difficult under severe domain shifts. The stated task is to reconstruct an entire scene consistently across views and then extract a mesh via iso-surface extraction. In this setting, learning-based neural volumetric methods trained on indoor datasets typically degrade under domain shift, including outdoor scenes, different trajectories, or different appearance statistics. Unlike classical multi-view stereo with geometric constraints, these data-driven volumetric methods rely on learned scene priors and are therefore sensitive to distribution shift (Ming et al., 13 Mar 2026).

The central motivation for VFMRecon is the tension between cross-domain transferable priors and volumetric consistency. VGGT depths are described as “coherent up to a shared relative scale” per batch or submap, but they lack global metric scale. Predictions are thus only defined up to an unknown scale. Without scale coherence, fusing such depths with known camera poses leads to drift and artifacts in the reconstructed volume and mesh.

This design target distinguishes VFMRecon from both purely feed-forward neural volumetric methods and direct use of VFM depth or point maps. The method does not attempt to re-optimize trajectories or dense volumes globally. Instead, it resolves global and local scale ambiguity through a compact optimization over per-submap scale factors, then uses the aligned geometry as conditioning for a learned reconstruction stage. A plausible implication is that the method treats scale alignment not as a post-processing refinement, but as a prerequisite for making foundation priors usable within scene-level fusion.

2. System architecture and pipeline

The pipeline assumes monocular RGB video frames $I=\{I_t\}$, known camera intrinsics $K$, and known poses $T_t \in SE(3)$. A sparse set of keyframes $I'=\{I_{k_1}, \dots, I_{k_P}\}$ is selected using the strategy of Duzceker et al. (DeepVideoMVS). The overall system has two stages: a lightweight scale alignment and depth reprojection stage, followed by a VFM-augmented neural volumetric reconstruction stage (Ming et al., 13 Mar 2026).

In Stage A, keyframes are divided into overlapping submaps $\mathcal{S}_m$ of consecutive frames, with overlap $\mathcal{T}=\{T_t \in SE(3)\}$0. The reported settings are $\mathcal{T}=\{T_t \in SE(3)\}$1 for ScanNet, $\mathcal{T}=\{T_t \in SE(3)\}$2 otherwise, and $\mathcal{T}=\{T_t \in SE(3)\}$3. VGGT is run independently on each submap to produce scale-ambiguous depths $\mathcal{T}=\{T_t \in SE(3)\}$4. An initial per-submap scale is estimated using feature correspondences, triangulation, and robust depth ratios. Relative scale constraints are then computed on overlapping submaps and optimized globally in log-scale. The aligned depths are fused into an initial SDF volume $\mathcal{T}=\{T_t \in SE(3)\}$5 using running-average volumetric fusion at $\mathcal{T}=\{T_t \in SE(3)\}$6 voxels. A temporary mesh $\mathcal{T}=\{T_t \in SE(3)\}$7 is extracted, and refined depth maps $\mathcal{T}=\{T_t \in SE(3)\}$8 are rendered from that mesh for all keyframes.

In Stage B, each keyframe is encoded twice: once by a CNN encoder and once by VGGT. Intermediate VGGT encoder features with a DPT head, denoted $\mathcal{T}=\{T_t \in SE(3)\}$9, are combined with CNN features $K \in \mathbb{R}^{3 \times 3}$0. VFMRecon inserts a lightweight bottleneck MLP adapter with dimensions $K \in \mathbb{R}^{3 \times 3}$1 into VGGT’s frame-attention transformer blocks, immediately after multi-head attention, while freezing the original VGGT weights. The fused 2D features are back-projected into a 3D feature volume, concatenated with $K \in \mathbb{R}^{3 \times 3}$2, and decoded by a 3D U-Net to obtain a refined representation $K \in \mathbb{R}^{3 \times 3}$3. For 3D query points, visible-view features are back-projected and average-pooled, then concatenated with volumetric features and passed to two MLP heads that predict SDF and occupancy. At inference, the final mesh is extracted by marching cubes.

The paper’s algorithmic summary is: keyframe selection, submap formation, VGGT per-submap depth, initial scales via triangulation, relative scale constraints on overlaps, log-scale Levenberg–Marquardt optimization, depth alignment, volumetric fusion and mesh rendering for refined depths, VFM-augmented reconstruction with adapters, and final mesh extraction.

3. Scale alignment methodology

The scale alignment stage is the method’s defining geometric component. For each submap $K \in \mathbb{R}^{3 \times 3}$4, feature correspondences are established between overlapping frames using SuperPoint and LightGlue. 3D points are triangulated using known poses and intrinsics, and correspondences are filtered by reprojection error and chirality. Let $K \in \mathbb{R}^{3 \times 3}$5 denote the triangulated depth at pixel $K \in \mathbb{R}^{3 \times 3}$6, and let $K \in \mathbb{R}^{3 \times 3}$7 denote VGGT’s predicted depth for the same pixel in submap $K \in \mathbb{R}^{3 \times 3}$8. The initial submap scale is

$K \in \mathbb{R}^{3 \times 3}$9

This initial estimate is then complemented by relative scale constraints between overlapping submaps. For adjacent submaps $\mathcal{V}$0 and $\mathcal{V}$1 with overlapping keyframes $\mathcal{V}$2, the method compares predictions on valid pixels $\mathcal{V}$3:

$\mathcal{V}$4

The robust median ratio for the edge $\mathcal{V}$5 is then

$\mathcal{V}$6

The global scale optimization is performed in log-space. Defining $\mathcal{V}$7 and $\mathcal{V}$8, VFMRecon solves

$\mathcal{V}$9

where $\mathcal{M}$0 denotes adjacency edges between overlapping submaps, $\mathcal{M}$1 weights each edge by overlap or quality, and $\mathcal{M}$2 regularizes the solution toward the initial scales. The solver is Levenberg–Marquardt, initialized by $\mathcal{M}$3, after which $\mathcal{M}$4.

Aligned depths are obtained by

$\mathcal{M}$5

After volumetric fusion and temporary mesh extraction, refined depths are rendered as

$\mathcal{M}$6

The paper characterizes the robustness strategy in three parts: robust medians for initial scales and relative constraints, a log-space formulation that converts multiplicative scales to additive constraints and yields a convex least-squares structure, and Levenberg–Marquardt for stable convergence. The ablation results make this stage central rather than auxiliary: on Tanks and Temples, removing scale alignment reduces F1 from $\mathcal{M}$7 to $\mathcal{M}$8, and removing both scale alignment and depth reprojection reduces F1 to $\mathcal{M}$9 (Ming et al., 13 Mar 2026).

4. VFM priors, adapters, and volumetric prediction

VFMRecon uses VGGT as its geometry-aware VFM. The integration strategy is deliberately narrow: the foundation model is frozen, and only lightweight adapters are trainable. The adapters are placed after multi-head attention in VGGT’s frame-attention transformer blocks, while global-attention blocks are not adapted because adapting only frame-attention layers is reported to work best (Ming et al., 13 Mar 2026).

Feature fusion proceeds by resizing VGGT features to the CNN feature resolution and summing them element-wise to obtain $I=\{I_t\}$0. These fused 2D features are back-projected into a 3D volume and averaged over visible views:

$I=\{I_t\}$1

where $I=\{I_t\}$2 if voxel $I=\{I_t\}$3 lies inside frame $I=\{I_t\}$4’s frustum and $I=\{I_t\}$5 otherwise, and $I=\{I_t\}$6 are projected image coordinates sampled bilinearly. The paper then concatenates the feature volume with the initial geometry:

$I=\{I_t\}$7

with $I=\{I_t\}$8 implemented as a 3D U-Net decoder.

The final field representation is an SDF volume with an auxiliary occupancy head. For query points, the model back-projects visible-view 2D fused features, aggregates them by average pooling, concatenates them with volumetric features, and predicts SDF logits and occupancy probability through two MLPs. Training uses

$I=\{I_t\}$9

The occupancy term is binary cross-entropy:

$K$0

and the SDF term is

$K$1

where SDF logits are mapped by $K$2 and $K$3. Supervision is prepared following FineRecon’s SDF fragment preparation.

The reported implementation uses an FPN-style architecture with EfficientNetV2-S, separate point CNN and voxel CNN trained independently, a 3D U-Net, Adam, batch size $K$4, initial learning rate $K$5 with decay factor $K$6, two NVIDIA A6000 GPUs with $K$7 each, and approximately one week of training.

5. Experimental evaluation and ablation evidence

Training is performed on the ScanNet official training split. Evaluation is reported on the ScanNet official test split with 100 scans, TUM RGB-D, and Tanks and Temples. The latter two are the cross-domain settings: TUM RGB-D is characterized as indoor with distinct motion and trajectory statistics, while Tanks and Temples is characterized as indoor/outdoor, large-scale, and high-variability (Ming et al., 13 Mar 2026).

On ScanNet mesh evaluation, VFM-Recon reports Acc. $K$8, Comp. $K$9, Chamfer $T_t \in SE(3)$0, Prec. $T_t \in SE(3)$1, Recall $T_t \in SE(3)$2, and F1 $T_t \in SE(3)$3. FineRecon reports Acc. $T_t \in SE(3)$4, Comp. $T_t \in SE(3)$5, Chamfer $T_t \in SE(3)$6, Prec. $T_t \in SE(3)$7, Recall $T_t \in SE(3)$8, and F1 $T_t \in SE(3)$9. DG-Recon reports Acc. $I'=\{I_{k_1}, \dots, I_{k_P}\}$0, Comp. $I'=\{I_{k_1}, \dots, I_{k_P}\}$1, Chamfer $I'=\{I_{k_1}, \dots, I_{k_P}\}$2, Prec. $I'=\{I_{k_1}, \dots, I_{k_P}\}$3, Recall $I'=\{I_{k_1}, \dots, I_{k_P}\}$4, and F1 $I'=\{I_{k_1}, \dots, I_{k_P}\}$5. In depth rendering metrics, VFM-Recon reports Abs.Rel. $I'=\{I_{k_1}, \dots, I_{k_P}\}$6, Abs.Diff. $I'=\{I_{k_1}, \dots, I_{k_P}\}$7, Sq.Rel. $I'=\{I_{k_1}, \dots, I_{k_P}\}$8, $I'=\{I_{k_1}, \dots, I_{k_P}\}$9 $\mathcal{S}_m$0, $\mathcal{S}_m$1 $\mathcal{S}_m$2, and Comp. $\mathcal{S}_m$3, compared with FineRecon’s Abs.Rel. $\mathcal{S}_m$4, Abs.Diff. $\mathcal{S}_m$5, Sq.Rel. $\mathcal{S}_m$6, $\mathcal{S}_m$7 $\mathcal{S}_m$8, $\mathcal{S}_m$9 $\mathcal{T}=\{T_t \in SE(3)\}$00, and Comp. $\mathcal{T}=\{T_t \in SE(3)\}$01. The paper notes slightly weaker Acc. and Sq.Rel., attributing them to sensitivity to a few outliers, while threshold accuracies improve.

On TUM RGB-D, VFM-Recon reports Prec./Recall/F1 of $\mathcal{T}=\{T_t \in SE(3)\}$02, compared with SimpleRecon’s $\mathcal{T}=\{T_t \in SE(3)\}$03 and GP-Recon’s $\mathcal{T}=\{T_t \in SE(3)\}$04. On Tanks and Temples, VFM-Recon reports $\mathcal{T}=\{T_t \in SE(3)\}$05, while VGGT reports $\mathcal{T}=\{T_t \in SE(3)\}$06, SimpleRecon reports $\mathcal{T}=\{T_t \in SE(3)\}$07, and FineRecon reports $\mathcal{T}=\{T_t \in SE(3)\}$08. The outdoor Tanks and Temples result is emphasized in the abstract: an F1 score of $\mathcal{T}=\{T_t \in SE(3)\}$09 in reconstructed mesh evaluation, substantially above the closest competitor VGGT at $\mathcal{T}=\{T_t \in SE(3)\}$10.

Dataset	VFM-Recon	Selected comparison
ScanNet F1	74.3	FineRecon 73.6
TUM RGB-D F1	46.7	SimpleRecon 43.5
Tanks and Temples F1	70.1	VGGT 51.8

The ablations isolate the contributions of scale alignment, depth reprojection, and adapted VGGT features. The default configuration, with VGGT depth, Scale Alignment (S.A.), Depth Reprojection (D.R.), and Adapted VGGT features (A.V.), reaches ScanNet Prec./Recall/F1 of $\mathcal{T}=\{T_t \in SE(3)\}$11 and Tanks and Temples Prec./Recall/F1 of $\mathcal{T}=\{T_t \in SE(3)\}$12. Removing S.A. lowers Tanks and Temples F1 to $\mathcal{T}=\{T_t \in SE(3)\}$13. Removing D.R. yields Tanks and Temples F1 $\mathcal{T}=\{T_t \in SE(3)\}$14, while slightly improving in-domain F1 to $\mathcal{T}=\{T_t \in SE(3)\}$15; the paper interprets this as evidence that D.R. promotes multi-view consistency under shift. Removing both S.A. and D.R. lowers Tanks and Temples F1 to $\mathcal{T}=\{T_t \in SE(3)\}$16. “Adapted VGGT Only,” without VGGT depth, S.A., and D.R., gives Tanks and Temples F1 $\mathcal{T}=\{T_t \in SE(3)\}$17, and “Naive VGGT Only” gives $\mathcal{T}=\{T_t \in SE(3)\}$18. The stated conclusion is that each component contributes, with scale alignment and reprojection crucial for cross-domain performance and adapters providing additional gains.

6. Interpretation, limitations, and relation to adjacent work

The paper frames the method’s strengths in three terms: cross-domain robustness, a decoupled design, and empirical performance. Cross-domain robustness arises because scale alignment restores multi-view coherence for VFM depths, enabling stable volumetric fusion, while task-adapted VFM features improve coverage and structural fidelity across domains. The decoupled design is the use of compact optimization over scales rather than expensive global bundle or dense-volume re-optimization. This suggests that VFMRecon is organized around a separation of duties: geometry is stabilized first, and reconstruction learning is asked to refine rather than repair fundamentally inconsistent depth.

Several limitations are explicit. The method requires known camera poses and does not optimize them. The paper also notes outlier sensitivity on ScanNet, visible as slightly weaker Acc. and Sq.Rel., attributed to a few spurious fragments or slight surface thickening in fine-detail regions when coverage increases. Depth reprojection may introduce small quantization errors; while it can be neutral or slightly negative in-domain, it significantly improves cross-domain robustness. These observations qualify any interpretation that the method is uniformly superior across all metrics or all operating points (Ming et al., 13 Mar 2026).

A common misconception addressed by the reported comparisons is that a strong geometry-aware VFM alone suffices for scene-level fusion. The paper explicitly contrasts VFMRecon with VGGT alone: VGGT produces plausible point maps but lacks scale consistency for fusion. Another misconception is that feature adaptation should encompass the entire VFM backbone. The reported design instead freezes VGGT and trains only small adapters in frame-attention blocks, with the stated purpose of preserving cross-domain robustness.

In adjacent work, “Reasmory: 3D Reconstruction as Explicit Memory for VLMs Spatial Reasoning” treats Vision Foundation Model-based reconstruction, denoted there as VFMRecon, as the backbone that converts multi-view images or monocular videos into an explicit, queryable 3D memory (He et al., 31 May 2026). In that context, reconstruction-based VFMs provide depth, poses, and fused geometry, while a domain-specific language and AST-based validator constrain how a VLM can query, transform, and render the resulting memory. This is not a reformulation of VFM-Recon’s reconstruction objective itself, but it illustrates a downstream use case in which reconstruction-based VFM pipelines function as explicit spatial memory. A plausible implication is that methods like VFMRecon are relevant not only for mesh reconstruction benchmarks but also for systems that require geometrically consistent scene representations for higher-level reasoning.

Taken together, VFMRecon occupies a specific place in scene reconstruction research: it targets scene-level volumetric SDF fusion with multi-view coherence and mesh extraction, differs from NeRF-style or object-level large reconstruction models, and attempts to preserve foundation-prior transferability while satisfying the metric consistency constraints of volumetric fusion.