GeoFusionLRM: Geometry-Aware 3D Refinement

• GeoFusionLRM is a geometry-aware refinement framework for single-image 3D reconstruction that integrates rendered depth and normal maps to correct structural errors.
• It introduces a dedicated transformer encoder (GeoFormer) and a lightweight fusion module (GeoFuser) to merge geometric cues with semantic tokens from InstantMesh.
• Empirical evaluations demonstrate that GeoFusionLRM significantly improves normal fidelity and mesh-image alignment without additional external supervision.

Searching arXiv for the named topic and closely related work to ground the article in current papers. GeoFusionLRM is a refinement framework for single-image 3D reconstruction with large reconstruction models (LRMs). It addresses a specific failure mode of modern LRMs: plausible rendered appearance can coexist with incorrect underlying geometry, including geometric inconsistencies, distorted or unreliable surface normals, misaligned details, and poor structural fidelity in thin structures, edges, holes, rims, and curved surfaces. The method introduces a geometry-aware self-correction mechanism in which an initial mesh is rendered into depth and normal maps, those geometric cues are encoded by a dedicated transformer, and the resulting features are fused back into the model’s semantic token stream to guide a refined reconstruction pass. In the published instantiation, GeoFusionLRM is built on top of InstantMesh and improves alignment between reconstructed meshes and conditioning images without additional supervision or external signals (Yildirim et al., 15 Feb 2026).

1. Concept and problem setting

GeoFusionLRM is situated in the recent LRM paradigm for image-to-3D reconstruction, where transformer-based models are trained on paired images and 3D assets and directly infer a 3D representation from a single input image. The paper places it alongside LRM, InstantMesh, SPAR3D, and LGM as part of a broader shift away from older category-specific reconstruction systems toward large-scale feed-forward reconstruction models (Yildirim et al., 15 Feb 2026).

The motivation is narrowly geometric. The paper argues that prior LRMs condition reconstruction only on image-derived semantic features, even when they use camera-aware encoders or synthesized multi-view images. This creates a gap between image plausibility and structural correctness. A reconstructed object may look acceptable in rendered RGB while remaining geometrically wrong, for example through softened edges, flattened high-frequency structures, absent holes, or missing topological details. The underlying diagnosis is that image features alone do not explicitly encode the errors present in the current reconstruction, and the model lacks an explicit mechanism to compare its reconstructed geometry against the conditioning view (Yildirim et al., 15 Feb 2026).

GeoFusionLRM changes the conditioning paradigm by adding self-predicted geometry to the reconstruction loop. Instead of reconstructing once from image tokens alone, it reconstructs an initial mesh, renders that mesh into depth and normal maps, encodes those geometry maps, and uses them to correct the next reconstruction. The central claim is therefore not that geometry supervision is added from outside the model, but that the model’s own intermediate geometry is turned into an internal corrective signal.

A common misconception arises from the name. Despite the prefix “Geo,” GeoFusionLRM is not a geospatial or GIS model. It is a geometry-aware 3D reconstruction framework. This distinguishes it from works such as GeoLRM, which target multi-view image-to-3D Gaussian generation with sparse 3D anchor tokens and deformable cross-attention (Zhang et al., 2024).

2. Architecture and reconstruction loop

The published implementation of GeoFusionLRM augments InstantMesh rather than replacing it wholesale (Yildirim et al., 15 Feb 2026). InstantMesh first takes a single input image $I$, uses Zero123++ to synthesize six views $\{I_k\}_{k=1}^{6}$ with associated camera parameters $\{C_k\}_{k=1}^{6}$, encodes each view with a transformer-based vision encoder initialized from DINO and conditioned on cameras through AdaLN, and then aggregates these tokens with a triplane decoder transformer to produce a triplane representation

$$\mathcal{T} \in \mathbb{R}^{3 \times R \times R \times d},$$

which is decoded into a mesh via differentiable isosurface extraction (Yildirim et al., 15 Feb 2026).

GeoFusionLRM preserves this backbone and inserts a geometry-aware refinement loop after an initial reconstruction. The first pass produces an initial mesh

$\mathcal{M}^{(0)}.$

From the current mesh $\mathcal{M}^{(t)}$, the method renders depth and normal maps

$D^{(t)},\; N^{(t)} = \Pi(\mathcal{M}^{(t)}),$

where $\Pi(\cdot)$ is differentiable rendering. These rendered geometry maps form the self-predicted cues that drive correction (Yildirim et al., 15 Feb 2026).

Two new modules implement the refinement. The first is GeoFormer, a geometry-aware encoder for rendered depth and normals. It is initialized as a copy of the ViT encoder used by InstantMesh, including camera-parameter conditioning via AdaLN. To process geometry rather than RGB, the input layer is changed from 3 channels to 4 channels, and the added weights are zero-initialized. GeoFormer produces geometry-aware token embeddings

$F^{\text{geo}} = E_{\text{geo}}(D^{(t)}, N^{(t)}).$

The second new module is GeoFuser, a lightweight fusion block that merges the original semantic tokens and the geometry-aware tokens through residual correction: $$F^{\text{fused}} = F^{\text{sem}} + f_\theta(F^{\text{sem}}, F^{\text{geo}}).$$ Here $\{I_k\}_{k=1}^{6}$0 is a two-layer feed-forward network with a hidden SiLU activation. The final linear layer of $\{I_k\}_{k=1}^{6}$1 is initialized with zero weights and zero bias, so the residual correction is initially disabled and the first iteration matches baseline behavior (Yildirim et al., 15 Feb 2026).

The refined tokens are then fed back into the LRM cross-attention layers, producing an updated triplane and a refined mesh $\{I_k\}_{k=1}^{6}$2. The resulting system is iterative in structure: reconstruct, render geometry, encode geometry, fuse tokens, reconstruct again. The paper describes this as geometry-aware self-correction because the model uses its own predicted mesh geometry, rather than an external depth or normal predictor, to improve itself (Yildirim et al., 15 Feb 2026).

3. Formalization and optimization

The baseline semantic token extraction inherited from InstantMesh is

$\{I_k\}_{k=1}^{6}$3

where $\{I_k\}_{k=1}^{6}$4 is synthesized view $\{I_k\}_{k=1}^{6}$5, $\{I_k\}_{k=1}^{6}$6 its camera parameters, and $\{I_k\}_{k=1}^{6}$7 the corresponding semantic token embeddings (Yildirim et al., 15 Feb 2026).

GeoFusionLRM adds the rendered-geometry branch

$\{I_k\}_{k=1}^{6}$8

followed by the residual token fusion rule

$\{I_k\}_{k=1}^{6}$9

This formalization is deliberately compact. The paper does not provide explicit self-attention or cross-attention equations for GeoFormer or the triplane decoder, nor does it specify a separate objective that directly penalizes disagreement between semantic and geometry tokens. The intervention is architectural rather than loss-centric (Yildirim et al., 15 Feb 2026).

Training uses the same loss composition as InstantMesh: $\{C_k\}_{k=1}^{6}$0 The paper identifies these terms as photometric MSE, perceptual LPIPS, mask consistency, depth alignment, normal similarity, and FlexiCubes regularization. “Without additional supervision or external signals” means that GeoFusionLRM does not introduce an external monocular depth estimator, an external normal prediction network, or new geometry labels beyond the supervision already used in InstantMesh. The refinement branch is trained under the existing reconstruction losses (Yildirim et al., 15 Feb 2026).

A notable optimization choice is that the baseline InstantMesh backbone is frozen during fine-tuning. Only GeoFormer and GeoFuser are trained. This makes GeoFusionLRM a targeted augmentation rather than a full end-to-end retraining of the underlying LRM (Yildirim et al., 15 Feb 2026).

4. Training setup and empirical evaluation

GeoFusionLRM is trained on Objaverse-1.0, with assets having rendered alpha coverage $\{C_k\}_{k=1}^{6}$1 removed, leaving about 168k objects for training. Fine-tuning is performed for 168k steps on 4 × A100 GPUs using AdamW with learning rate $\{C_k\}_{k=1}^{6}$2, $\{C_k\}_{k=1}^{6}$3, $\{C_k\}_{k=1}^{6}$4, and weight decay $\{C_k\}_{k=1}^{6}$5, with cosine decay to 0 over 100k steps. Supervision uses 32 views per object with random camera poses on a viewing sphere and radius sampled from

$\{C_k\}_{k=1}^{6}$6

Orientations are uniformly distributed (Yildirim et al., 15 Feb 2026).

Evaluation is conducted on OmniObject3D and Google Scanned Objects (GSO). For OmniObject3D, the protocol uses 5 objects per category across 100 categories, yielding 500 objects. For GSO, 500 uniformly sampled objects are used. The evaluation includes a fixed uniform view grid with elevations $\{C_k\}_{k=1}^{6}$7 crossed with six azimuths, as well as the standard OmniObject3D benchmark views with 16 benchmark views per object. RGB quality is measured with PSNR, SSIM, and LPIPS. Geometry quality is assessed by rendering normal maps from generated meshes in Blender on the same camera grid and comparing predicted and ground-truth normals using PSNR, SSIM, and LPIPS, averaged over views and objects (Yildirim et al., 15 Feb 2026).

The reported comparisons show that GeoFusionLRM outperforms InstantMesh on both appearance and geometry metrics, with the clearest gains in normal quality.

Setting	InstantMesh	GeoFusionLRM
GSO RGB (PSNR / SSIM / LPIPS)	20.31 / 0.920 / 0.0832	20.35 / 0.921 / 0.0831
GSO Normals (PSNR / SSIM / LPIPS)	25.83 / 0.947 / 0.0625	26.39 / 0.950 / 0.0592
OmniObject3D uniform RGB	21.94 / 0.915 / 0.0798	23.05 / 0.921 / 0.0722
OmniObject3D uniform Normals	24.68 / 0.918 / 0.0781	26.16 / 0.927 / 0.0658
OmniObject3D benchmark RGB	21.85 / 0.913 / 0.0805	22.75 / 0.916 / 0.0741
OmniObject3D benchmark Normals	24.24 / 0.918 / 0.0769	25.76 / 0.926 / 0.0648

The main empirical pattern is consistent with the method’s design. RGB gains on GSO are modest, but gains in normal-map quality are clearer. On OmniObject3D, both RGB and normal metrics improve more visibly, again with stronger absolute improvements on the geometry-oriented evaluation. The paper accordingly emphasizes that GeoFusionLRM’s primary contribution is not merely better image realism, but better structural fidelity (Yildirim et al., 15 Feb 2026).

5. Ablations and mechanism analysis

The ablation studies isolate three design choices: the geometry signals used, the fusion strategy, and the initialization of the geometry encoder (Yildirim et al., 15 Feb 2026).

First, the paper compares depth only, normal only, and the combination of both. Using both signals gives the best results. On OmniObject3D, “Normal Only” yields RGB SSIM 0.916 and LPIPS 0.0738, with normal SSIM 0.926 and LPIPS 0.0662. “Depth Only” produces nearly identical results: RGB SSIM 0.916, LPIPS 0.0738; normal SSIM 0.926, LPIPS 0.0661. The proposed joint design improves these to RGB SSIM 0.920, LPIPS 0.0722; normal SSIM 0.927, LPIPS 0.0658. The stated interpretation is that depth and normals provide complementary geometric cues (Yildirim et al., 15 Feb 2026).

Second, the fusion rule matters. Replacing the residual corrective fusion with simple token concatenation degrades performance. “Token Concat” yields RGB SSIM 0.916, LPIPS 0.0739; normal SSIM 0.926, LPIPS 0.0663, whereas the proposed GeoFuser achieves RGB SSIM 0.920, LPIPS 0.0722; normal SSIM 0.927, LPIPS 0.0658. This indicates that geometry is more effective as a correction term applied to semantic features than as a naively appended token stream (Yildirim et al., 15 Feb 2026).

Third, initialization of GeoFormer is nontrivial. Random initialization is worse than reusing the pretrained ViT encoder. “Random Init” gives RGB SSIM 0.914 and LPIPS 0.0755, with normal SSIM 0.924 and LPIPS 0.0680; the proposed pretrained initialization improves these to RGB SSIM 0.920 and LPIPS 0.0722, with normal SSIM 0.927 and LPIPS 0.0658. The paper attributes this to the benefit of reusing a DINO-based, camera-aware encoder as the starting point for geometry-aware feature learning (Yildirim et al., 15 Feb 2026).

The iterative nature of the method is also studied. Training is unrolled for $\{C_k\}_{k=1}^{6}$8 steps, but the paper reports that the first geometry-aware refinement pass provides the substantial gain and further iterations quickly plateau. The chosen inference setup is therefore one baseline pass followed by one refinement pass. This supports a specific interpretation of the mechanism: GeoFusionLRM is not primarily an open-ended recurrent correction process, but a compact second-pass repair stage that addresses the most salient structural errors left by the baseline reconstruction (Yildirim et al., 15 Feb 2026).

6. Qualitative behavior, limitations, and relation to adjacent work

Qualitatively, the paper emphasizes objects for which plausible RGB renderings mask structural mistakes. Reported examples include a turtle teapot whose shell details are flattened by baseline methods, a bowl whose rim becomes rounded off, a decorative planter whose cut-out holes degrade into shallow indentations, and a hat with a dark band where SPAR3D hallucinates a large hole while GeoFusionLRM reconstructs a coherent continuous surface. The claimed improvements are sharper geometry, more consistent normals, better preservation of holes, rims, edges, and fine structures, and better mesh-image alignment in ambiguous dark or low-texture regions (Yildirim et al., 15 Feb 2026).

These improvements come with a measurable computational cost. On an NVIDIA RTX 3090, InstantMesh is reported at 3.878 TFLOPs and 0.989 s inference time, whereas GeoFusionLRM requires 8.687 TFLOPs and 3.854 s inference time. The paper explicitly frames this as a quality-efficiency trade-off caused by the additional forward passes and the geometry branch (Yildirim et al., 15 Feb 2026).

The main limitation identified in the paper is thin structure reconstruction. GeoFusionLRM improves coarse branch continuity and fixes gaps and discontinuities relative to InstantMesh, but extremely fine structures such as tiny roots or very thin branches can still be missing. The stated cause is not primarily the fusion mechanism itself, but the low-resolution triplane representation in the InstantMesh backbone, which limits recoverable detail (Yildirim et al., 15 Feb 2026).

In the broader literature, GeoFusionLRM is best understood as a geometry-feedback refinement layer for LRMs rather than a new base 3D representation. Unlike GeoLRM, which uses sparse 3D anchor tokens, occupancy proposal, and deformable cross-attention to map multi-view image evidence into 3D Gaussian assets (Zhang et al., 2024), GeoFusionLRM is a single-image mesh-refinement method built around self-rendered depth and normals. A plausible implication is that the two papers occupy adjacent but distinct positions in the 3D reconstruction design space: GeoLRM emphasizes sparse geometry-aware 2D–3D fusion during reconstruction generation, whereas GeoFusionLRM emphasizes geometry-aware self-correction after an initial reconstruction has already been produced.

Taken on its own terms, GeoFusionLRM’s significance lies in reframing LRM conditioning. The method suggests that image-only semantic tokens are not always sufficient for high-fidelity 3D structure, and that self-predicted geometry can function as an internal corrective representation. Within the InstantMesh setting studied in the paper, that change yields more consistent normals and better structural fidelity without external geometry predictors or new supervision (Yildirim et al., 15 Feb 2026).