GeoFuse-MV3D: Geometry Refinement for Robot Planning

• The paper introduces GeoFuse-MV3D as a multi-view reconstruction refinement module that mitigates hallucinated poses by verifying geometry against input masks.
• It employs a dual-branch approach, fusing an external geometry prior with provider-free axis corrections while preserving appearance attributes.
• Empirical results on the GSO-30 benchmark demonstrate modest yet consistent improvements in Chamfer Distance, PSNR, SSIM, and LPIPS for safer robot manipulation.

Searching arXiv for the cited papers to ground the article with fresh references. {"query":"GeneralVLA-2 Geometry-Aware Reconstruction and Governed Memory for Robot Planning arXiv (Wang et al., 16 Jun 2026)"} arXiv search query: "(Wang et al., 16 Jun 2026) GeneralVLA-2" GeoFuse-MV3D is the multi-view object reconstruction branch introduced in "GeneralVLA-2: Geometry-Aware Reconstruction and Governed Memory for Robot Planning" (Wang et al., 16 Jun 2026). Within that hierarchy, it provides planner-facing object-centric 3D evidence by refining an existing MV-SAM3D reconstruction when calibrated multi-view observations are available. Its stated motivation is that monocular SAM3D-style reconstruction can hallucinate object pose and unseen backside geometry, which is hazardous for manipulation because grasp points, clearance, collision margins, and occlusion structure all depend on geometry. GeoFuse-MV3D addresses this by using an external geometry estimate only as a prior provider, verifying geometry against input-view masks, applying soft visual-hull support, performing axis-wise refinement, and fusing only geometry while preserving appearance (Wang et al., 16 Jun 2026).

1. Placement within GeneralVLA-2

GeoFuse-MV3D is not a standalone robot system. In GeneralVLA-2, it is one perception module inside a larger hierarchy comprising affordance perception, 3D reasoning and planning through 3DAgent, execution through a low-level robot controller, and two new components: GeoFuse-MV3D for object geometry and a governed KnowledgeBank for memory (Wang et al., 16 Jun 2026). Its role is therefore narrowly defined but operationally important: to improve the quality of object-centric geometry supplied to the planner.

The paper frames the problem around a practical failure mode of monocular SAM3D-style methods. Single-image reconstruction can generate plausible 3D objects from one image and mask, but it suffers from monocular pose ambiguity, hallucinated unseen structure, and instability that is problematic for manipulation (Wang et al., 16 Jun 2026). GeoFuse-MV3D is presented as an improved MV-SAM3D branch rather than a full replacement. It is built on top of MV-SAM3D, keeps the same benchmark protocol and input setup, and changes the reconstruction refinement stage through conservative geometry fusion (Wang et al., 16 Jun 2026).

The planner-facing relevance is explicit. In GeneralVLA-2, 3DAgent plans a waypoint trajectory

$T_t = \{(x_e, y_e, z_e, g_e)\}_{e=1}^{E}.$

GeoFuse-MV3D is intended to make this downstream planning more reliable by stabilizing object extent, pose, and geometry under calibrated multi-view observations (Wang et al., 16 Jun 2026). The paper links improved geometry to better grasp selection, better clearance and collision estimation, and more reliable reasoning about openings, handles, and spatial relations.

2. Inputs, assumptions, and outputs

GeoFuse-MV3D operates when multi-view object observations are available. The input is defined as

$$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$

where $I_i$ is an RGB observation, $M_i$ an object mask, $K_i$ a camera intrinsic matrix, and $T_i$ a camera pose (Wang et al., 16 Jun 2026). The benchmark therefore uses five input views.

The branch also consumes an initial MV-SAM3D reconstruction,

$G_0 = \{(x_j,\theta_j)\}_{j=1}^{N},$

where $x_j \in \mathbb{R}^3$ is a Gaussian center and $\theta_j$ contains non-geometric rendering attributes, including opacity, scale, rotation, and spherical-harmonic appearance (Wang et al., 16 Jun 2026). In the implementation summarized in the paper, Source A additionally uses an external geometry prior provider instantiated with VGGT (Wang et al., 16 Jun 2026).

Several assumptions are explicit. The method assumes calibrated camera poses are accurate enough for projection-based support checks; masks are reasonably accurate and consistent across views; the observations correspond to the same object; the object is stable enough across views; and compatible Gaussian indexing is available for some fusion operations (Wang et al., 16 Jun 2026). The paper also makes a design assumption that conservative correction is preferable to aggressive editing because planning is sensitive to geometry errors.

The output remains in the same Gaussian/object-centric 3D representation family as MV-SAM3D. The final Gaussian set is written as

$$G_{\text{out}} = \{((1-\alpha)x_j^{\prime A} + \alpha x_j^{\prime B}, \theta_j^{A})\}_{j=1}^{N}.$$

This formulation makes the module’s central constraint explicit: only geometry is fused, while appearance attributes are preserved from a trusted source (Wang et al., 16 Jun 2026).

3. Reconstruction pipeline

The pipeline begins from the MV-SAM3D output $$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$0, which is not discarded but treated as the reference object model to be refined (Wang et al., 16 Jun 2026). The representation is a 3D Gaussian Splatting-style model in which geometry is encoded in Gaussian centers and non-geometry in rendering attributes. The paper also notes that a mesh is synchronized with Gaussian centers, indicating that geometric transforms are applied consistently to Gaussian centers and associated mesh vertices (Wang et al., 16 Jun 2026).

GeoFuse-MV3D then branches into two geometry sources. Source A is the geometry-prior main provider. It uses an external geometry-prior provider, instantiated with VGGT, but the paper is explicit that this prior is not treated as the final reconstruction. Instead, it is used as a prior provider whose geometry must be validated against observed masks (Wang et al., 16 Jun 2026). Source A also includes lightweight appearance affine calibration.

Source B is a provider-free axis compensation branch. It does not use the external geometry prior. Instead, it relies only on input masks, camera poses, and the existing reconstruction, and performs low-dimensional correction of Gaussian centers synchronized to the mesh vertices (Wang et al., 16 Jun 2026). The two branches are deliberately complementary: Source A injects external geometry cues, while Source B provides a provider-free conservative correction.

The validation stage projects 3D points into the input views and scores their consistency with the masks. For a point $$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$1, the projection into view $$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$2 is

$$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$3

and the mask-consistency score is

$$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$4

The paper interprets $$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$5 as the average support that a point receives from visible input masks: larger values indicate stronger silhouette consistency (Wang et al., 16 Jun 2026).

A key design choice is the use of soft visual-hull support rather than hard carving. Unsupported geometry is not deleted. Instead, it is moved inward toward the object center:

$$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$6

with bounded shrink strength

$$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$7

If support is low, $$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$8 increases and the point is shrunk inward; if support is stronger, the correction is smaller (Wang et al., 16 Jun 2026). The paper explicitly characterizes this as a conservative diagnostic and correction mechanism rather than a hard deletion rule.

Source A also performs appearance affine calibration using

$$D = \{(I_i, M_i, K_i, T_i)\}_{i \in V_{\text{in}}}, \qquad V_{\text{in}} = \{0,1,2,3,4\},$$9

with $I_i$0 (Wang et al., 16 Jun 2026). This is described as lightweight and intended to preserve the object’s original tone. It does not alter the final appearance-preserving rule of the module.

Source B applies axis-wise refinement:

$I_i$1

where $I_i$2 are axis scales and $I_i$3 is a small shift (Wang et al., 16 Jun 2026). The paper gives the corresponding objective as

$I_i$4

with

$I_i$5

This keeps the correction close to the original MV-SAM3D geometry while improving mask agreement (Wang et al., 16 Jun 2026).

The final fusion is conservative and geometry-only. In simple blended form,

$I_i$6

The more explicit residual form is

$I_i$7

with

$I_i$8

Thus geometry is blended, while color, opacity, scale, rotation, and spherical-harmonic appearance are preserved from the trusted source (Wang et al., 16 Jun 2026). The paper further states that if topology differs or mask support is weak, the system falls back to the trusted Source A result.

4. Design logic and mathematical character

The most distinctive feature of GeoFuse-MV3D is not merely the use of a geometry prior, but the combination of verification, bounded correction, and appearance preservation. The paper repeatedly emphasizes a conservative philosophy: external geometry is used as a guide rather than a replacement; mask support acts as a diagnostic, a shrink trigger, and a fusion confidence term; and unsupported geometry is corrected continuously rather than deleted (Wang et al., 16 Jun 2026).

This conservative logic is also visible in the average mask reprojection disagreement

$I_i$9

which the paper says is used as a diagnostic rather than a hard deletion criterion (Wang et al., 16 Jun 2026). That choice aligns the method with manipulation-oriented safety rather than aggressive geometric editing.

The representation further reinforces this positioning. GeoFuse-MV3D keeps the Gaussian/object-centric 3D format of MV-SAM3D, preserves non-geometric attributes from the trusted source, and changes only the refinement stage (Wang et al., 16 Jun 2026). The paper does not provide a full pseudocode algorithm box, nor does it specify step counts, optimizer choices, learning rates, or an additional learning procedure specific to GeoFuse-MV3D. In the evaluated setup, it is presented as effectively a training-free refinement module (Wang et al., 16 Jun 2026).

A plausible implication is that GeoFuse-MV3D should be read less as a general-purpose reconstruction architecture and more as a manipulation-oriented repair layer for multi-view object geometry. That interpretation is consistent with the paper’s emphasis on planner-facing 3D evidence, small continuous corrections, and explicit preservation of appearance fields.

5. Empirical evaluation and ablation evidence

GeoFuse-MV3D is evaluated on GSO-30, a single-object multi-view benchmark derived from Google Scanned Objects (Wang et al., 16 Jun 2026). The protocol keeps the same object list, the same five input views $M_i$0, the same masks, and the same camera poses as the reproduced MV-SAM3D baseline, and evaluates on held-out target views $M_i$1 (Wang et al., 16 Jun 2026). The reported metrics are Chamfer Distance (CD), PSNR, SSIM, and LPIPS.

Metric	MV-SAM3D baseline	GeoFuse-MV3D
CD $M_i$2	45.8876	44.8770
PSNR	13.2421	13.5547
SSIM	0.8051	0.8134
LPIPS	0.2795	0.2739

Relative to the MV-SAM3D baseline, GeoFuse-MV3D reduces CD by 2.20% and LPIPS by 2.02%, while increasing PSNR by 2.36% and SSIM by 1.03% (Wang et al., 16 Jun 2026). The paper characterizes these gains as modest but consistent.

The ablation study on GSO-30 full30 averaged over three seeds is more informative about internal structure. Source A: geometry prior + appaff achieves CD $M_i$3 $M_i$4, PSNR $M_i$5 $M_i$6, SSIM $M_i$7 $M_i$8, and LPIPS $M_i$9 $K_i$0 relative to baseline. A + softVH slightly improves CD and SSIM over A, but slightly lowers PSNR and LPIPS, which the paper attributes to the mask-support constraint emphasizing geometry over appearance fidelity. Source B: provider-free axis compensation gives the strongest single-source PSNR, SSIM, and LPIPS. A+B geometry fusion yields CD $K_i$1 $K_i$2, PSNR $K_i$3 $K_i$4, SSIM $K_i$5 $K_i$6, and LPIPS $K_i$7 $K_i$8, supporting the claim that the two branches are complementary (Wang et al., 16 Jun 2026).

The qualitative findings reported in the appendix are that GeoFuse-MV3D produces more complete and more geometrically consistent reconstructions on several challenging GSO-30 objects, with Figures 5 and 6 summarized as showing improved object completeness and pose consistency under the fixed input-view setup (Wang et al., 16 Jun 2026). The paper does not provide a dedicated numerical pose-accuracy metric, so these observations remain qualitative.

The limitations are explicit. GeoFuse-MV3D depends on reliable camera calibration, reliable masks, and reliable poses; failure cases include mask leakage, missing object regions, calibration drift, heavy occlusion, and deformable objects, which are outside the method’s current scope (Wang et al., 16 Jun 2026). The method is also intentionally conservative, and because the final output preserves appearance from the trusted source, it does not try to repair texture or lighting artifacts aggressively (Wang et al., 16 Jun 2026).

6. Relation to adjacent multi-view 3D work and nomenclature

GeoFuse-MV3D occupies a specific point in the broader multi-view 3D landscape. Relative to latent-space direct 3D extraction methods, it is markedly more projection-driven and verification-driven. For example, Fus3D decodes a dense SDF directly from intermediate VGGT latent features through a learned 3D extraction transformer, does not require externally supplied camera intrinsics or extrinsics at test time, and performs fusion in latent space before per-view geometric outputs are decoded (Fink et al., 26 Mar 2026). GeoFuse-MV3D instead uses calibrated cameras, explicit projection into observed masks, soft visual-hull support, and geometry-only fusion over an existing Gaussian/object-centric reconstruction (Wang et al., 16 Jun 2026).

Relative to direct joint multiview registration systems, GeoFuse-MV3D is also narrower in scope. FUSER directly predicts global poses for unordered partial point clouds in one feed-forward pass and then refines them through $K_i$9 diffusion, reframing multiview registration as joint global inference rather than pairwise estimation (Jiang et al., 10 Dec 2025). GeoFuse-MV3D does not solve global registration from scratch; it assumes calibrated cameras and focuses on conservative object reconstruction refinement (Wang et al., 16 Jun 2026).

Relative to systems-oriented online fusion, FUSE-Flow decouples geometry-aligned multi-view extrinsic calibration and stateless real-time multi-view point cloud fusion, with linear time and memory scaling for the fusion stage (Sun, 3 Jun 2026). GeoFuse-MV3D is not a real-time array-scale fusion system of that kind. Its concern is planner-facing object geometry for robot manipulation, not stateless large-scale point cloud fusion (Wang et al., 16 Jun 2026).

At the reasoning level, DR-MV3D is relevant because it decomposes multi-view 3D VQA into allocentric global map construction, question-conditioned trajectory planning, and egocentric grounding, using frozen VGGT and SAM3 to generate pseudo structural supervision (Choi et al., 22 Jun 2026). This suggests a broader pattern in current work: geometry-aware intermediate representations are increasingly treated as explicit interfaces between perception and downstream reasoning. GeoFuse-MV3D fits that pattern on the manipulation side by improving the quality of the object-centric 3D evidence passed to 3DAgent (Wang et al., 16 Jun 2026).

A separate nomenclature issue is important. GeoFuse-MV3D should not be conflated with the method called GeoFuse in "Road Maps as Free Geometric Priors: Weather-Invariant Drone Geo-Localization with GeoFuse" (Fang et al., 14 May 2026). That paper introduces GeoFuse for weather-invariant drone-view geo-localization through satellite-road map fusion and explicitly does not present a method called GeoFuse-MV3D (Fang et al., 14 May 2026). The shared prefix therefore reflects naming overlap rather than method identity.

In summary, GeoFuse-MV3D is best understood as a conservative multi-view geometry refinement branch for robot-facing object reconstruction. It starts from MV-SAM3D, uses an external geometry prior as a guide rather than a replacement, verifies geometry against input-view masks, applies soft visual-hull shrinkage instead of hard carving, performs provider-free axis-wise refinement, and fuses only geometry while preserving appearance (Wang et al., 16 Jun 2026). Its empirical contribution is not a wholesale reconstruction paradigm shift, but a consistent improvement in planner-facing object geometry under the same multi-view protocol, with the clearest technical novelty residing in the mask-verified, soft-support, axis-refined, geometry-only fusion mechanism.