GeoWeaver: Research Systems for VGGT and 3D Reconstruction

Updated 25 August 2026

GeoWeaver refers to two distinct systems: one for multimodal large language models(MLLMs) to enhance spatial reasoning with a pre-reasoning geometric-grounding framework via VGGT geometry encoder, and another for monocular 3D reconstruction via hierarchical test-time adaptation.
GeoWeaver systems incorporate geometry as an explicitly optimized substrate after a frozen VGGT geometry encoder to enhance spatial consistency.
GeoWeaver for multimodal MLLMs improves robustly across multiple spatial reasoning benchmark datasets, including 72.8% accuracy on VSI-Bench for GeoWeaver-5B and 64.8% for Video-MME.

GeoWeaver is the name of two distinct 2026 research systems: a pre-reasoning geometric-grounding framework for multimodal LLMs (MLLMs), and a hierarchical framework for long-sequence monocular 3D reconstruction. The former grounds visual tokens with token-specific evidence drawn from multiple layers of a frozen geometry encoder before language-model reasoning; the latter assembles chunk-wise geometric predictions into globally consistent reconstructions through correspondence-driven test-time adaptation (TTA). Although they address different computational problems, both systems use geometry as an explicit representational or optimization substrate rather than leaving spatial consistency to unconstrained downstream reasoning (Miao et al., 21 May 2026, Jiang et al., 18 Aug 2026).

1. Name and conceptual scope

The two systems are technically unrelated beyond their shared emphasis on geometric structure.

GeoWeaver for multimodal reasoning addresses the problem that MLLMs can encode visual appearance and object semantics while remaining unreliable on depth, distance, direction, layout, viewpoint, routes, object relations, and temporal spatial changes. It constructs a multi-level geometry bank from hidden states of a frozen VGGT encoder, routes geometry evidence separately for each visual token, and injects the selected evidence into visual tokens before concatenation with text embeddings and processing by the LLM (Miao et al., 21 May 2026).

GeoWeaver for 3D reconstruction addresses long-sequence reconstruction from monocular RGB video. It processes short temporal chunks with a Geometric Prior Model (GPM), then uses a hierarchical TTA backend to estimate chunk transformations, correct accumulated drift, refine frame-level poses and depths, and reconcile camera calibration. Its objective is a globally coherent camera trajectory and dense 3D scene representation without requiring feed-forward inference over the complete video (Jiang et al., 18 Aug 2026).

The shared name is therefore best understood as a naming coincidence rather than evidence of a common software framework, research group, architecture, or dataset. The two systems operate at different levels:

System	Primary input	Principal output	Core mechanism
Multimodal GeoWeaver	Images or video frames plus language	Geometry-grounded visual tokens and spatial reasoning answers	Token-adaptive geometric evidence routing
Reconstruction GeoWeaver	Long monocular RGB sequences	Globally consistent poses, depths, and point clouds	Hierarchical correspondence-based TTA

Both systems reject uniform treatment of geometry. The multimodal system argues that visual tokens require different abstraction levels; the reconstruction system argues that local geometry, chunk alignment, and frame refinement should be optimized hierarchically rather than jointly from an unstable initialization.

2. GeoWeaver for pre-reasoning geometric grounding

The multimodal system is motivated by a representational distinction between semantic visual features and geometric evidence. Existing geometry-aware MLLMs are described as using structural branches, 3D-aware supervision, reasoning-stage fusion, or visual reasoning artifacts. These methods commonly expose a shared geometric signal to all tokens or inject geometry after the visual representation has already been formed. GeoWeaver instead treats geometric grounding as a prerequisite for language reasoning.

Given $K$ frames $\{\mathbf I_k\}_{k=1}^{K}$ and an instruction $q$ , the base vision encoder produces merged semantic visual tokens

$\mathbf V\in\mathbb R^{(KN_p)\times D},$

where $N_p$ is the number of merged tokens per frame and $D$ is the hidden dimension expected by the LLM. The $i$ -th token is denoted $\mathbf v_i\in\mathbb R^D$ .

The same frames are passed through a frozen VGGT geometry encoder. Hidden patch-token states are extracted from a selected layer set $\mathcal S$ ; non-patch tokens such as camera or register tokens are discarded. For layer $l$ ,

$\{\mathbf I_k\}_{k=1}^{K}$ 0

where $\{\mathbf I_k\}_{k=1}^{K}$ 1 is the number of raw geometry tokens per frame and $\{\mathbf I_k\}_{k=1}^{K}$ 2 is VGGT’s feature dimension. The collection $\{\mathbf I_k\}_{k=1}^{K}$ 3 constitutes the multi-level geometry bank.

The default configuration selects the latter twelve layers of a 24-layer VGGT:

$\{\mathbf I_k\}_{k=1}^{K}$ 4

The intended interpretation is hierarchical. Earlier selected layers respond more locally to fine structures and boundaries; intermediate layers represent object-level geometry and nearby context; later layers encode broader scene layout and global spatial organization.

Each selected layer is normalized independently:

$\{\mathbf I_k\}_{k=1}^{K}$ 5

A $\{\mathbf I_k\}_{k=1}^{K}$ 6 spatial merge aligns the geometry-token resolution with the semantic visual-token resolution. A shared MLP projector $\{\mathbf I_k\}_{k=1}^{K}$ 7 maps the resulting features into the language-model space:

$\{\mathbf I_k\}_{k=1}^{K}$ 8

The same projector is used for all geometry layers.

Token-adaptive routing

For each semantic visual token, a lightweight router produces a vector of layer-selection logits:

$\{\mathbf I_k\}_{k=1}^{K}$ 9

The router is not described as conventional query–key cross-attention. It predicts token-conditioned preferences over the geometry layers. Only the top $q$ 0 logits are retained, and a softmax is applied over the selected entries:

$q$ 1

The default is $q$ 2. Unselected layers receive zero weight, while the selected weights sum to one. The geometry evidence for token $q$ 3 is

$q$ 4

Unlike uniform averaging or a single global layer mixture, this permits $q$ 5 for different visual tokens. A boundary token may favor local geometry, whereas a token representing room layout or an object relation may favor deeper scene-level features.

Residual grounding

Selected geometry evidence is added to the original semantic token through

$q$ 6

where $q$ 7. The output projection is initialized to zero, so the initial grounding module is an identity mapping:

$q$ 8

This initialization preserves the pretrained visual representation at the beginning of fine-tuning. After grounding, the visual and text embeddings are concatenated:

$q$ 9

and passed through the ordinary LLM decoder. The LLM therefore receives geometry-grounded visual tokens rather than a late auxiliary geometry stream.

3. Training, evaluation, and evidence for the multimodal framework

GeoWeaver uses autoregressive instruction tuning with cross-entropy loss:

$\mathbf V\in\mathbb R^{(KN_p)\times D},$ 0

No auxiliary depth, reconstruction, contrastive, router-supervision, or layer-selection loss is used. The router learns layer preferences indirectly through answer prediction.

VGGT remains frozen:

$\mathbf V\in\mathbb R^{(KN_p)\times D},$ 1

The grounding module is optimized, while the base MLLM is trained differently in the two reported variants. GeoWeaver-5B uses full-parameter fine-tuning; GeoWeaver-10B uses LoRA with rank $\mathbf V\in\mathbb R^{(KN_p)\times D},$ 2, scaling parameter $\mathbf V\in\mathbb R^{(KN_p)\times D},$ 3, and dropout $\mathbf V\in\mathbb R^{(KN_p)\times D},$ 4. Training uses a learning rate of $\mathbf V\in\mathbb R^{(KN_p)\times D},$ 5, global batch size 64, eight A100 GPUs with 80 GB each, and eight frames for both variants. The reported naming contains an inconsistency: the main setup refers to Qwen3.5 backbones at 4B- and 9B-scale, whereas the evaluated systems are called GeoWeaver-5B and GeoWeaver-10B.

Evaluation covers VSI-Bench, ReVSI, SPAR-Bench, ViewSpatial, BLINK, 3DSRBench, EmbSpatial, and, in the appendix, Dyn-Bench. General multimodal capability is assessed with MMBench, Video-MME, BLINK, and TempCompass.

On VSI-Bench, the reported average scores are:

Method	VSI-Bench average
SpatialStack-5B	67.5
GeoWeaver-5B	72.8
GeoWeaver-10B	75.05

GeoWeaver-5B obtains category scores of 83.2 for Object Count, 61.8 for Absolute Distance, 76.8 for Object Size, 62.4 for Room Size, 72.8 for Relative Distance, 83.5 for Relative Direction, 56.3 for Route Plan, and 84.7 for Approaching Order. GeoWeaver-10B obtains 84.3, 58.8, 83.8, 68.8, 73.5, 92.0, 55.0, and 84.3 on the same categories.

Across six spatial benchmarks, GeoWeaver-5B is reported with an overall score of 64.8:

Method	VSI-Bench	SPAR-Bench	ViewSpatial	BLINK	3DSRBench	EmbSpatial	Overall
VLM-3R-7B	60.7	42.4	40.5	52.3	51.5	68.2	52.6
Cambrian-S-7B	62.9	37.9	41.3	37.9	45.0	72.8	49.6
GeoWeaver-5B	72.8	69.2	52.8	58.1	58.6	77.5	64.8

On ReVSI, GeoWeaver-5B averages 54.2 with 16 frames, 54.9 with 32 frames, and 54.1 with all frames. GeoWeaver-10B averages 57.4, 57.5, and 57.3 under the same frame budgets. The best performance is therefore associated with 32 frames rather than the largest available frame set.

The ablations directly evaluate the central architectural claims. For geometry-bank construction, the latter twelve layers obtain 72.80, compared with 67.40 for the first twelve and 69.65 for uniformly sampled twelve layers. For bank size, twelve layers outperform four, eight, sixteen, and all layers. For per-token evidence budgets, selecting two layers produces 72.80, compared with 71.69 for one, 72.54 for three, 72.09 for four, and 71.28 for all layers. For allocation strategy, token-adaptive routing reaches 72.80, compared with 71.28 for global layer weights and 70.35 for uniform averaging.

Grounding position produces the following VSI-Bench averages:

Fusion location	Average
Input-level fusion	68.10
Decoder-side fusion	70.45
Pre-reasoning grounding	72.80

These results support the narrower empirical claim that, for the tested models and benchmarks, geometry is more effective when it is selectively incorporated into visual tokens before language-model reasoning.

The framework retains general multimodal performance in the reported evaluation. GeoWeaver-5B obtains an overall score of 68.98 on MMBench, Video-MME, BLINK, and TempCompass, compared with 68.41 for Qwen3.5-4B and 68.00 for SpatialStack-5B. MMBench is 83.33 for GeoWeaver-5B, compared with 83.25 for Qwen3.5-4B and 83.42 for SpatialStack-5B. These results indicate no obvious degradation on the tested general-capability benchmarks, but they do not establish universal preservation of multimodal capability.

The principal limitations are dependence on the external VGGT geometry encoder, use of hidden geometric features rather than explicit depth maps, point maps, camera poses, or metric-scale states, absence of explicit geometric supervision, and lack of a detailed computational cost analysis. The top- $\mathbf V\in\mathbb R^{(KN_p)\times D},$ 6 router is a layer-preference mechanism rather than a full content-based geometry–token attention module. The framework may also be affected by domain shift, dynamic scenes, unusual viewpoints, sparse or blurry inputs, and redundant temporal evidence.

4. GeoWeaver for long-sequence 3D reconstruction

The second GeoWeaver addresses long-sequence monocular RGB reconstruction. Its problem formulation combines two requirements that are difficult to satisfy simultaneously: accurate local geometry and globally consistent camera motion.

Feed-forward models such as DUSt3R, MASt3R, VGGT, and Depth Anything 3 provide strong local estimates but become difficult to apply jointly to hundreds or thousands of frames because attention and activation-memory costs grow with sequence length. Independent chunk processing reduces memory requirements but produces separate coordinate frames, scale drift, pose errors, point-cloud discontinuities, and inconsistent calibration.

GeoWeaver divides the sequence into contiguous chunks

$\mathbf V\in\mathbb R^{(KN_p)\times D},$ 7

with one shared frame between adjacent chunks. Its two principal components are:

Geometric Prior Model (GPM): predicts local depth, confidence, camera translation, camera rotation, and field-of-view information for each chunk.
Test-Time Adaptation (TTA): assembles and adjusts the chunk predictions using a sparse correspondence graph and a sequence of increasingly flexible optimization stages.

The GPM is built on Depth Anything 3 and contains a ViT image encoder, a DPT-style depth-confidence head, and a transformer camera head. For observation $\mathbf V\in\mathbb R^{(KN_p)\times D},$ 8, it predicts

$\mathbf V\in\mathbb R^{(KN_p)\times D},$ 9

with

$N_p$ 0

Depth is back-projected into camera coordinates:

$N_p$ 1

The model does not require a separate point-map head. Predicted depth and camera intrinsics generate 3D points.

The GPM is trained in two stages: fixed four-view training for 100,000 iterations, followed by variable-length training for another 100,000 iterations using clips of two to sixteen frames. Optimization uses AdamW, cosine learning-rate decay, encoder learning rate $N_p$ 2, and task-head learning rate $N_p$ 3. Training data include ScanNet, ScanNet++, HM3D, Hypersim, ASE, MegaSynth, MVSSynth, Unreal4K, BlendedMVS, DynamicStereo, and TartanAir.

The GPM loss combines local geometry, normalized global chunk geometry, and relative camera supervision. Confidence-aware geometric losses weight point errors using predicted confidence while penalizing universally low confidence. Relative camera supervision removes ambiguity in the global coordinate frame and accounts for monocular translation scale ambiguity.

Correspondence graph

TTA uses a sparse view graph

$N_p$ 4

Edges include temporal neighbors, adjacent cross-chunk relationships, non-local edges retrieved by SALAD, and long-range edges selected by GEP. The method avoids matching every frame pair, which would require $N_p$ 5 candidate pairs.

For each selected edge $N_p$ 6, RoMa provides dense pixel correspondences

$N_p$ 7

and a matching confidence $N_p$ 8. GPM confidence at the matched pixels is

$N_p$ 9

The correspondence weight is

$D$ 0

Confidence- and consistency-aware sampling uses a confidence threshold of 0.5, a depth-consistency threshold of 0.2, and at most 10,000 correspondences per image pair. If strict filtering disconnects the graph, the depth-consistency threshold is progressively relaxed.

Correspondences are introduced hierarchically: adjacent cross-chunk correspondences initialize neighboring chunk transformations; adjacent and long-range cross-chunk correspondences optimize all chunk transformations; and the complete selected graph refines frame poses, depths, and intrinsics.

Three-stage test-time adaptation

Stage 1: sequential initialization. Matched pixels from neighboring chunks are back-projected using predicted depths. Robust, confidence-weighted pairwise Sim(3) transformations are estimated from the top 50% most confident correspondences and sequentially composed. Within-chunk poses, depths, and intrinsics remain fixed. This stage provides connectivity but may accumulate drift.

Stage 2: global chunk-level Sim(3) alignment. Each chunk receives a learnable similarity transformation

$D$ 1

The exponential parameterization ensures positive scale. The first chunk is fixed to remove global gauge freedom. Chunk transformations are optimized using dense 2D and 3D objectives, while internal GPM poses, depths, and intrinsics remain fixed.

Stage 2 uses Adam with learning rate $D$ 2, a maximum of 5,000 iterations, a 3D residual maximum distance of 0.8, and 3D residual weight 2.0. Adjacent correspondences impose local continuity, while long-range correspondences correct accumulated translation, rotation, and scale drift.

Stage 3: coarse-to-fine frame refinement. Chunk similarities are converted into frame-level $D$ 3 initializations. A frame-level parameter set is

$D$ 4

Depth is corrected with a low-dimensional affine model:

$D$ 5

For uncalibrated sequences, focal-length corrections are shared among frames from the same physical camera:

$D$ 6

Corrections are regularized toward the GPM prior through penalties on $D$ 7, $D$ 8, and focal corrections. Calibrated sequences retain fixed initial focal lengths.

The coarse phase uses a 15-pixel CDF range, 250 bins, and smoothing bandwidth 2. Fine refinement uses Adam with learning rate $D$ 9, 3D residual range 0.1, and 3D residual weight 1.0. Adaptation stops when the median reprojection error stabilizes.

Robust CDF-style objective

Rather than minimizing the mean reprojection or 3D residual, the method optimizes a confidence- and validity-weighted soft empirical cumulative distribution function. For residual type $i$ 0, residuals are mapped through a stage-dependent function and evaluated against a set of thresholds. A logistic function approximates hard inlier thresholds. The resulting loss increases the weighted fraction of reliable correspondences with small residuals over a range of thresholds.

The complete objective combines the 2D CDF loss, a weighted 3D CDF loss, and, in Stage 3, regularization of affine-depth and focal corrections. This distribution-level formulation is intended to be more robust than a mean residual in the presence of structured outliers and to avoid dependence on a single hard threshold.

5. Reconstruction experiments, results, and limitations

The reconstruction system is evaluated on Tanks and Temples, Mip-NeRF 360, Virtual KITTI 2, and Oxford Spires. Metrics include absolute trajectory error (ATE), mean rotation registration error (RRE), AUC@3°, AbsRel, SqRel, and camera AUC at $i$ 1 and $i$ 2. Sequence-level Sim(3) alignment is used for evaluation, so long-range drift remains visible.

The GPM obtains AbsRel 0.082, SqRel 0.069, AUC@5° 49.9, and AUC@30° 77.8. In the main long-sequence comparison, GeoWeaver obtains:

Dataset	AUC	ATE or RRE
Tanks and Temples	72.9	ATE 0.023 m
Mip-NeRF 360	66.4	ATE 0.036 m
Virtual KITTI 2	64.6	ATE 0.627 m
Oxford Spires	—	RRE 0.575°, ATE 4.372 m

The paper reports the lowest average ATE on Tanks and Temples and Mip-NeRF 360, the second-lowest ATE on Virtual KITTI 2, and the lowest reported Oxford Spires RRE and ATE among the corresponding comparisons. AMB3R obtains higher average AUC on some datasets, particularly Tanks and Temples and Virtual KITTI 2; GeoWeaver is therefore not uniformly best on every metric.

The TTA ablation on Tanks and Temples demonstrates the contribution of the hierarchy:

Sequential initialization	Chunk alignment	Frame refinement	ATE	AUC@3°
Yes	No	No	0.674	24.4
Yes	Yes	No	0.075	43.8
Yes	Yes	Yes	0.023	72.9

The largest ATE reduction occurs when global chunk alignment is added. Frame refinement then reduces residual errors and substantially improves AUC.

The TTA procedure also improves other geometric prior models. For Scal3R, ATE changes from 0.338 to 0.033 and AUC@3° from 58.1 to 63.6. For DA3, ATE changes from 0.358 to 0.033 and AUC@3° from 63.9 to 68.5. This supports the interpretation that the TTA backend is not tied to one GPM architecture, although its performance remains dependent on the quality of the prior geometry and correspondences.

The method is designed for mostly static scenes, monocular RGB input, fixed physical-camera groups, and sufficiently reliable correspondences. Limitations include dynamic scenes, rolling shutter, arbitrary zoom or changing intrinsics, latency from matching and optimization, correspondence failures under weak texture or severe blur, dependence on a connected view graph, inability of affine depth correction to repair severe local topology errors, and the absence of a complete complexity or timing analysis.

The authors also identify future directions including direct GPM training with the TTA objective, stronger or domain-adaptive geometry encoders, explicit 3D representations, improved robustness to dynamic scenes and changing camera conditions, and reduced test-time optimization.

The multimodal and reconstruction systems instantiate different uses of geometric evidence.

The multimodal framework treats geometry as a feature-selection and representation problem. A frozen geometry encoder supplies a hierarchy of spatial abstractions. A router chooses the relevant layers for each visual token, and a residual projection modifies the visual representation before language reasoning. Its central variables are visual tokens, geometry layers, router weights, and language-model hidden states.

The reconstruction framework treats geometry as a global consistency and optimization problem. A learned model produces local priors; image correspondences connect frames and chunks; Sim(3) transformations reconcile coordinate systems; affine depth corrections and focal adjustments refine the reconstruction. Its central variables are camera poses, chunk similarities, depths, intrinsics, correspondences, and robust residual distributions.

Both systems use a staged approach to avoid forcing a single mechanism to solve incompatible problems simultaneously. The multimodal model separates visual–geometric alignment from high-level linguistic reasoning. The reconstruction model separates local prediction from global assembly and then from frame-level refinement.

Both also use sparsity selectively. The multimodal system keeps only two geometry layers per visual token by default. The reconstruction system uses a compact connected correspondence graph and limits retained correspondences to 10,000 per image pair. In each case, the design rejects indiscriminate aggregation: all layers or all frame pairs are treated as potentially redundant, noisy, or computationally inefficient.

The two systems should not be conflated with earlier geographic and geometric platforms that also use the name “GeoWeaver” in conceptual comparisons. GNOSIS proposed a time-aware encyclopedic globe with thematic “skin-maps” (Oliva, 2012); GeoTR proposed Turkish geographic question answering through ontology-guided SPARQL generation (Tasar et al., 2023); GeoGLUE proposed a benchmark for geographic language understanding (Li et al., 2023); KnowWhereGraph proposed a large-scale RDF geospatial knowledge graph (Zhu et al., 19 Feb 2025); and GeoWarp proposed a nonstationary Gaussian-process model for subsea sediment properties (Bertolacci et al., 14 Jan 2025). These works are relevant to the broader technical vocabulary of spatial representation, but they are not components of either 2026 GeoWeaver system.

The shared term consequently identifies multiple research objects:

Token-level geometric grounding for MLLMs (Miao et al., 21 May 2026).
Hierarchical geometric assembly for long-sequence 3D reconstruction (Jiang et al., 18 Aug 2026).
Earlier conceptual or comparative references to geographic knowledge systems and spatial modeling that do not establish a common lineage.

The most precise technical distinction is that the first GeoWeaver modifies the representation consumed by a LLM, whereas the second modifies test-time geometric parameters and coordinate systems to assemble a coherent 3D reconstruction.