GeoWeaver: Accurate Long-Sequence 3D Reconstruction via Hierarchical Geometric Assembly

Abstract: Long-sequence 3D reconstruction from RGB videos requires both accurate local geometry and globally consistent camera motion. Feed-forward models provide strong depth and pose predictions, but their memory cost prevents joint inference over long sequences. Chunk-wise processing improves scalability, yet independently predicted chunks often exhibit scale drift, pose errors, and point-cloud misalignment. We present GeoWeaver, a unified framework comprising a Geometric Prior Model (GPM) and Test-Time Adaptation (TTA). The GPM predicts chunk-wise depth, confidence, and camera parameters as adjustable geometric priors. TTA then performs sequential initialization, global chunk-level Sim(3) alignment, and coarse-to-fine refinement of camera poses, affine depth corrections, and intrinsics. Dense correspondences provide adjacent, cross-chunk, and long-range constraints, while a robust CDF-style objective jointly optimizes weighted 2D reprojection and 3D consistency residuals. This design preserves local geometric accuracy while correcting accumulated pose, scale, depth, and calibration errors. Experiments across diverse long-sequence benchmarks demonstrate improved camera accuracy, global consistency, and point-cloud quality. Ablations verify the contribution of each adaptation stage, and applying the same TTA procedure to different geometric prior models consistently improves their trajectory estimates, demonstrating that GeoWeaver is not tied to a specific GPM.

• The paper introduces GeoWeaver, which assembles 60-frame geometric predictions through sequential Sim(3) initialization, global chunk alignment, and frame-level pose, depth, and focal refinement.
• The method achieves leading accuracy on several benchmarks, including 0.023 m ATE on Tanks and Temples and 0.575° relative rotation error on Oxford Spires.
• The paper shows that the same test-time adaptation substantially improves other frontends, reducing Tanks and Temples ATE to 0.033 m for Scal3R and 0.033 m for DA3 while avoiding quadratic all-pairs matching.

Overview

GeoWeaver addresses long-sequence 3D reconstruction from RGB video by combining a learned feed-forward prior with sequence-specific optimization (2608.17389). The system comprises two components: a Geometric Prior Model (GPM) that predicts chunk-wise depth, confidence, and camera parameters, and a Test-Time Adaptation (TTA) procedure that assembles independently predicted chunks into a globally consistent reconstruction. The central observation motivating the design is that feed-forward geometry models such as DUSt3R, MASt3R, VGGT, and Depth Anything 3 provide accurate local predictions but cannot jointly attend over hundreds or thousands of frames, while chunk-wise processing introduces scale drift, pose errors, and point-cloud misalignment across chunk boundaries. GeoWeaver's response is to treat the chunk predictions as adjustable priors rather than fixed outputs.

Method: hierarchical geometric assembly

The pipeline splits a long RGB sequence into contiguous 60-frame chunks sharing only one frame between neighbors — in contrast to baselines using 30-frame overlap. This reduces redundant inference by increasing the chunk stride from 30 to 59 while still providing a geometric anchor for alignment. The GPM, built on Depth Anything 3 with a ViT encoder, DPT-style depth-confidence head, and transformer camera head, is trained on mixed indoor, outdoor, and synthetic data over two stages (fixed four-view clips for 100K iterations, then variable-length 2–16-view clips for another 100K), with joint local-geometry, normalized chunk-geometry, and relative-pose supervision using geodesic rotation error and Huber translation penalties.

TTA proceeds through three stages of increasing degrees of freedom:

1. Sequential initialization estimates pairwise Sim(3) transformations between adjacent chunks via confidence-filtered back-projected correspondences, composing them into an initial layout.
2. Global chunk alignment optimizes one learnable Sim(3) transformation per chunk under dense correspondence constraints, fixing the first chunk to remove gauge freedom. Long-range correspondences act as loop-like constraints correcting accumulated scale and pose drift.
3. Coarse-to-fine refinement unlocks frame-level SE(3) poses, per-frame affine depth corrections $$\tilde{d}_i(\mathbf{u}) = a_i d_i^0(\mathbf{u}) + b_i$$, and focal corrections shared across frames from the same physical camera group, following the subgraph-then-global schedule of Marginalized Bundle Adjustment.

A notable objective design choice is the robust CDF-style loss: rather than minimizing mean residuals or applying a single hard inlier threshold, it maximizes the confidence-weighted empirical fraction of correspondences below multiple smoothed thresholds, treating residuals as samples from an error distribution. This makes the objective insensitive to structured outliers in wide-baseline dense matches. Correspondences are obtained via RoMa over a view graph pruned by SALAD retrieval and GEP edge prioritization, avoiding quadratic all-pairs matching. Confidence weighting combines matcher confidence multiplicatively with GPM depth confidence.

Results

Across four benchmarks spanning object-, room-, road-, and urban-scale scenes (Tanks and Temples, Mip-NeRF 360, Virtual KITTI 2, Oxford Spires), GeoWeaver achieves the lowest average ATE on Tanks and Temples (0.023 m vs. 0.053 m for MBA and 0.261 m for AMB3R) and Mip-NeRF 360 (0.036 m), second-lowest ATE on Virtual KITTI 2 (0.627 m), and lowest RRE on Oxford Spires (0.575°, roughly half the best baseline's 1.335°). On Tanks and Temples it attains AUC@3° of 72.9%, second only to AMB3R's 76.9%. The standalone GPM achieves the best SqRel (0.069) among compared frontends, indicating strong depth priors independent of adaptation.

Benchmark	Metric	GeoWeaver	Best competing
Tanks and Temples	ATE (m)	0.023	0.053 (MBA)
Mip-NeRF 360	ATE (m)	0.036	0.037 (DA3)
VKITTI 2	ATE (m)	0.627	0.608 (Scal3R)
Oxford Spires	RRE (deg)	0.575	1.335 (LingBot-Map)

The ablation isolates each TTA stage on Tanks and Temples: initialization alone yields ATE 0.674 / AUC 24.4; adding chunk alignment drops ATE to 0.075 / AUC 43.8; full frame refinement reaches ATE 0.023 / AUC 72.9. Chunk-level alignment thus accounts for most of the drift correction, while frame-level refinement contributes a decisive AUC gain of nearly 29 points. Convergence is fast: the objective falls by 50% within the first 100 chunk-alignment iterations and saturates near 440 iterations, with median-reprojection-error-based stopping limiting wasted computation.

The strongest claim in the paper concerns model agnosticism: applying the identical TTA configuration to Scal3R and DA3 priors reduces their Tanks-and-Temples ATE from 0.338 and 0.358 to 0.033 respectively, without source-specific tuning. This indicates that the gains derive from the adaptation machinery exploiting shared geometric information rather than from a bespoke frontend, though the frontend does matter — GeoWeaver-GPM + TTA still outperforms Scal3R + TTA and DA3 + TTA (ATE 0.025, AUC 71.3 vs. 68.5).

Limitations and open questions

The paper concedes several constraints. TTA adds matching and optimization cost beyond pure feed-forward inference, and although convergence is rapid, per-sequence test-time optimization remains fundamentally slower than a single forward pass; whether the procedure can reach real-time operation is left open. The method assumes static scenes and fixed per-camera calibration — dynamic content, rolling shutter, and zoom are not handled. Focal corrections are enabled only for uncalibrated sequences, so calibrated inputs do not benefit from this degree of freedom. Evaluation uses a single sequence-level Sim(3) alignment with no per-frame realignment, so reported ATE genuinely reflects drift, but all benchmarks are relatively well-textured; behavior on extremely weak-overlap or textureless trajectories remains untested. Finally, post-training the GPM jointly with the TTA objective is proposed but not evaluated, leaving open whether the test-time optimization burden can be absorbed into training.

Conclusion

GeoWeaver demonstrates that chunk-wise feed-forward geometry predictions, treated as adjustable priors and assembled through staged Sim(3) alignment followed by joint pose-depth-focal refinement under a robust distributional objective, can match or exceed both classical bundle adjustment and hybrid feed-forward backends on long sequences — while requiring only one shared frame between chunks. Its consistent improvement of multiple source models under identical adaptation settings supports the generality of the approach within its stated static-scene assumptions.