HiP3R: Hierarchical Pose Graphs for Long-Horizon 3D Reconstruction

Abstract: Feed-forward models accurately estimate camera motion and 3D geometry from short video windows. However, computational and memory costs prevent direct inference on sequences containing thousands of frames. Long videos must therefore be processed as overlapping chunks. This division produces independent local reconstructions with inconsistent coordinate frames and duplicate estimates of shared cameras. Achieving global consistency requires propagating constraints from adjacent overlaps and distant revisits across the sequence. This requirement naturally motivates pose graph optimization. However, the choice of graph level creates a trade-off. Chunk-level graphs efficiently resolve scale drift and close large loops. Yet, their rigid transformations cannot correct internal errors or conflicting frame predictions. Conversely, frame-level optimization offers local flexibility, but scaling it globally is computationally expensive and can break metric scale. HiP3R resolves this trade-off using a hierarchical pose graph. HiP3R first optimizes a robust chunk-level $\mathrm{Sim}(3)$ graph to establish the global layout and scale. It then holds this layout fixed and estimates one shared $\mathrm{SE}(3)$ camera per frame, turning overlapping window predictions into a single trajectory. Experiments across multiple benchmarks demonstrate the effectiveness of HiP3R in improving camera trajectories and 3D reconstruction. Our method applies to both chunk-based and streaming backends without additional training or fine-tuning. The codes are available at https://kkakkkka.github.io/HiP3R.

• the paper introduces the Hierarchical Pose Graphs (HiP3R) to solve long-window 3D reconstruction problems by distinguishing between aligning chunks for global layout and optimizing frames for shared placement and motion.
• After optimizing chunks in a common global world frame using $\operatorname{Sim}(3)$ transform, the method refines frame-level camera predictions with $\operatorname{SE}(3)$
• On public benchmarks and indoor/outdoor sequences, HiP3R achieves large reductions in Absolute Trajectory Error (ATE) and Relative Pose Errors (RPE), making it applicable to various setups and achieving competitive results with current dense global pose SLAM methods.

Problem setting and central contribution

HiP3R addresses the failure of short-window feed-forward 3D reconstruction systems when applied to long video sequences. Models such as VGGT-based architectures produce accurate local camera poses and dense pointmaps, but their attention and memory costs limit inference to short windows. Long sequences must therefore be partitioned into overlapping chunks. Each chunk is reconstructed in an independent coordinate system, and the resulting local trajectories may disagree in scale, orientation, translation, and even the poses assigned to frames that occur in multiple windows.

The paper’s central claim is that chunk-level alignment alone is insufficient for globally consistent reconstruction. A single $\operatorname{Sim}(3)$ transformation per chunk can correct the placement and scale of an entire local reconstruction, but it cannot alter the internal camera configuration predicted within that chunk. Conversely, optimizing every frame globally offers greater flexibility but is computationally expensive and can disturb the global metric layout. HiP3R resolves this tension through a hierarchical pose graph: a chunk-level $\operatorname{Sim}(3)$ optimization first establishes global layout and scale, followed by a frame-level $\operatorname{SE}(3)$ optimization that assigns exactly one shared camera pose to every physical frame (2610.31635).

This decomposition is important because it separates two distinct error sources. Long-range drift and loop inconsistency are treated as a global similarity-layout problem, whereas duplicate and mutually inconsistent estimates of overlapping frames are treated as a shared-camera trajectory problem. The method is backend-agnostic and operates on frozen predictions, requiring neither retraining nor test-time fine-tuning.

Hierarchical reconstruction formulation

Given a sequence of $T$ frames, HiP3R divides it into $K$ overlapping chunks. For each chunk, the feed-forward backend provides dense pointmaps, confidence estimates, and local camera poses. The objective is to recover a globally consistent trajectory containing one camera pose per frame while retaining the backend’s predicted scene geometry.

Coarse chunk-level layout

The first stage constructs a pose graph whose nodes are chunks and whose variables are similarity transformations

$A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$

These transformations map each chunk’s local coordinate system into a common world frame. Consecutive chunks contribute overlap edges, while visually retrieved revisits contribute loop-closure edges. Each edge supplies a measured relative similarity transformation obtained by aligning predicted 3D geometry.

The alignment procedure is confidence-weighted and robust. Rather than using every dense point, HiP3R samples a bounded number of point correspondences and estimates relative transformations with iteratively reweighted least squares. Weighted Umeyama alignment is repeatedly combined with Huber reweighting, allowing spatially inconsistent or low-confidence predictions to exert less influence. Loop edges are assigned bounded weights based on endpoint registration residuals. This design makes the graph insensitive to individual unreliable loop measurements, although it remains dependent on the quality and coverage of retrieved revisits.

The chunk graph optimizes rotation, translation, and log-scale residuals under a robust Huber loss. The first chunk is fixed as the world reference, resolving the global gauge freedom. A key design choice is progressive loop scheduling. Sequential overlap constraints first initialize a chain-like layout; loop closures are then introduced in increasing temporal-span order. This continuation strategy avoids simultaneously imposing many long-range constraints on a poorly initialized non-convex problem. Bounded trust-region optimization and scale bounds further constrain unstable updates.

The coarse stage is responsible for most of the reduction in absolute trajectory error. On KITTI, it reduces ATE from $77.42$ m to $9.750$ m, an $87\%$ reduction. On Oxford, the reduction is from $9.861$ m to $\operatorname{Sim}(3)$0 m, while on VBR it is from $\operatorname{Sim}(3)$1 m to $\operatorname{Sim}(3)$2 m (2610.31635). These results support the paper’s attribution of long-horizon ATE improvement primarily to global layout recovery and loop closure rather than to frame-level smoothing.

Fine shared-camera trajectory refinement

After obtaining the optimized chunk transformations $\operatorname{Sim}(3)$3, HiP3R places every local camera prediction in the recovered world frame. A separate shared pose variable $\operatorname{Sim}(3)$4 is then introduced for each physical frame $\operatorname{Sim}(3)$5. If a frame appears in multiple chunks, all of its placed predictions constrain the same $\operatorname{Sim}(3)$6 rather than producing independent camera estimates.

The fine graph contains two complementary factor types. Placement priors penalize deviations between a shared camera and every chunk-derived prediction of that frame. Relative-motion factors preserve the within-chunk motion relationships between consecutive retained frames. The resulting objective balances global placement against local trajectory coherence. This balance is essential: motion-only optimization can produce locally smooth trajectories while moving away from the globally corrected layout, whereas placement-only averaging does not explicitly preserve within-window motion.

The ablation confirms this interaction. On VGGT-Omega, the motion-only variant reduces relative errors but increases ATE to $\operatorname{Sim}(3)$7 m, compared with $\operatorname{Sim}(3)$8 m for the complete shared formulation. The placement priors therefore prevent the fine stage from undoing the coarse global solution, while motion factors reduce local discontinuities. On KITTI, shared-camera refinement reduces rotational RPE from $\operatorname{Sim}(3)$9 to $\operatorname{SE}(3)$0 and translational RPE from $\operatorname{SE}(3)$1 m to $\operatorname{SE}(3)$2 m after coarse layout optimization, with essentially unchanged ATE.

For sequences longer than 2,000 frames, HiP3R solves the fine graph on a keyframe subgraph and interpolates pose corrections for the remaining frames. The interpolation is applied to corrections relative to the coarse trajectory, using spherical linear interpolation for rotations and linear interpolation for translations. This preserves the coarse placement while avoiding a full optimization over tens of thousands of camera variables. On KITTI, stride-5 keyframe optimization produces nearly the same ATE as a dense frame graph: $\operatorname{SE}(3)$3 m versus $\operatorname{SE}(3)$4 m. The dense graph retains a modest advantage in relative errors, but the reported fine solve remains very small compared with inference and coarse graph construction.

Robustness and computational scaling

HiP3R combines several engineering choices to make graph construction and nonlinear optimization tractable. Bounded point sampling has the clearest isolated effect. In the reported extended-sequence ablation, limiting the sampled point pairs reduces graph construction and solve time from $\operatorname{SE}(3)$5 s to $\operatorname{SE}(3)$6 s, an $\operatorname{SE}(3)$7 reduction, while ATE changes negligibly from $\operatorname{SE}(3)$8 m to $\operatorname{SE}(3)$9 m. The result indicates that dense all-pixel alignment is computationally wasteful for the particular backend predictions and benchmarks considered.

Progressive loop scheduling also improves practical optimization. In the cumulative solver ablation, graph build and solve time decreases from $T$0 s to $T$1 s when progressive loop activation is added, while the final solve time falls from $T$2 s to $T$3 s. The complete configuration, which additionally incorporates continuation, robustification, and bounded restarts, reaches an ATE of $T$4 m with $T$5 s of total graph build and solve time and only $T$6 s for the final solve. Under perturbed loop measurements, where $T$7 of loop edges receive synthetic rotation, translation, and scale errors, the complete solver still obtains $T$8 m ATE compared with $T$9 m for the baseline solver.

The sparse graph structure is exploited explicitly in the fine stage. Each relative-motion residual depends on only two frame poses, and each unary prior depends on one. HiP3R uses closed-form $K$0 Jacobian blocks assembled in CSR format, while the smaller coarse $K$1 problem uses automatic differentiation. This avoids repeated residual sweeps for numerical differentiation. The appendix reports that graph coloring reduces post-processing time on a 735-frame scene from $K$2 s to $K$3 s relative to unstructured finite differences, while analytic derivatives eliminate these colored sweeps for the fine graph (2610.31635). These implementation details are not incidental: the hierarchical separation reduces the dimensionality of the expensive flexible optimization and permits an analytically sparse solver for the remaining frame variables.

The full pipeline remains dominated by backend inference and, for very long sequences, coarse graph processing. On KITTI, HiP3R reports approximately $K$4 s for local chunk inference, $K$5 s for loop-window inference, and $K$6 s for CPU graph solves. On VBR, with sequences averaging approximately 12,383 frames, the corresponding values are $K$7 s, $K$8 s, and $K$9 s. Thus, the method’s principal computational advantage is over pairwise alignment-heavy alternatives rather than over feed-forward inference itself. The comparison reports a component sum of $A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$0 s for HiP3R on VBR, versus $A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$1 s for VGGT-Long with loops.

Experimental evaluation

The evaluation covers seven benchmarks and 61 sequences, including indoor RGB-D environments, multi-scale reconstruction scenes, campus traversals, KITTI driving, and city-scale VBR sequences. The diversity of sequence lengths is useful because the method is intended specifically for long-horizon consistency, although the macro-averaging protocol gives each dataset equal weight rather than weighting sequences by duration.

With VGGT-Omega as the backend, HiP3R reduces macro-averaged ATE from $A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$2 m to $A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$3 m, a $A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$4 reduction. Rotational RPE decreases from $A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$5 to $A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$6, and translational RPE decreases from $A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$7 m to $A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$8 m, a $A_k=(s_k,R_k,t_k)\in\operatorname{Sim}(3).$9 reduction. With LingBot-MAP, the method reduces ATE from $77.42$0 m to $77.42$1 m, a $77.42$2 reduction, without modifying the streaming backend.

The strongest gains occur on long outdoor traversals. In the controlled comparison using identical VGGT-Omega predictions, loop-enabled VGGT-Long obtains ATE values of $77.42$3 m on KITTI and $77.42$4 m on VBR. HiP3R reduces these to $77.42$5 m and $77.42$6 m, respectively. The VBR result is particularly significant because it nearly halves the residual error after loop-enabled chunk alignment. On Oxford, the ATE improves from $77.42$7 m for loop-enabled VGGT-Long to $77.42$8 m. These results indicate that hierarchical optimization is most valuable when cumulative drift and distant revisits dominate the error budget.

The method also improves dense geometry, not only trajectory metrics. With VGGT-Omega, Chamfer distance decreases from $77.42$9 m to $9.750$0 m, a $9.750$1 reduction, while F1 increases from $9.750$2 to $9.750$3. The gains are especially large on Neural RGB-D, where F1 increases from $9.750$4 to $9.750$5, and on 7-Scenes, where it increases from $9.750$6 to $9.750$7. The stage-wise results show that coarse layout accounts for nearly all of the Chamfer improvement: VGGT-Omega reaches $9.750$8 m after layout optimization and $9.750$9 m after full refinement. Shared-camera refinement nevertheless provides consistent, smaller improvements, particularly by reconciling duplicated or discontinuous surfaces caused by conflicting camera poses.

The cross-backend results strengthen the modularity claim. The same graph formulation improves LingBot-MAP’s Chamfer distance from $87\%$0 m to $87\%$1 m and F1 from $87\%$2 to $74.60. On the dense-geometry benchmark, LingBot-MAP shows no change on ETH3D, where the native result is already strong, but improves substantially on Neural RGB-D and 7-Scenes. This pattern is consistent with HiP3R being primarily a long-sequence consistency module rather than a universal improvement to already coherent short-window predictions.

Limitations and open questions

HiP3R assumes that useful loop closures can be retrieved and that the corresponding backend reconstructions contain enough geometric overlap for reliable similarity alignment. The paper introduces robust weighting, Huber losses, progressive scheduling, and bounded restarts, and it reports robustness to synthetic loop corruption. Nevertheless, the method does not eliminate the dependence on visual place recognition or on the geometric validity of retrieved loop pairs. The behavior under systematic perceptual aliasing, repeated architectural structure, severe illumination change, or loop-retrieval failure is not fully characterized.

The evaluation also uses per-sequence $87\%$3 alignment for ATE and relative metrics. This is standard for scale-ambiguous reconstruction, but it means that the reported ATE does not directly measure absolute metric-scale accuracy. The coarse graph explicitly estimates relative scale and uses the first chunk as a scale reference, yet the evaluation protocol can absorb residual global scale discrepancies. A separate metric-scale evaluation would be necessary to determine how reliably HiP3R preserves an externally meaningful metric scale.

The frame-level refinement is not a full bundle-adjustment procedure. Pointmaps are transformed rigidly after camera optimization; their internal depth structure is not jointly optimized with camera poses. This design preserves the feed-forward geometry and avoids the cost of track-based bundle adjustment, but it leaves unresolved errors in depth, nonrigid local geometry, and camera-dependent pointmap prediction. The dense-geometry gains therefore arise mainly from correcting global placement and camera consistency, not from refining the underlying 3D structure.

Finally, the method is presented primarily as offline post-processing. The conclusion identifies online updates as an open direction, and the current progressive loop schedule, global graph reconstruction, and restart logic are not formulated as an incremental estimator. The paper also leaves open how to choose loop schedules and keyframes adaptively under changing motion, uncertainty, or backend quality. In particular, the default stride-5 keyframe policy works well in the reported experiments, but the evidence does not establish that it is optimal for highly irregular camera motion or substantially different frame rates.

Conclusion

HiP3R presents a principled separation of global and local consistency for long-horizon 3D reconstruction. Its chunk-level $87\%$4 graph resolves drift, scale, and loop placement, while its shared-camera $87\%$5 graph reconciles duplicate frame predictions without abandoning the recovered global layout. Across seven benchmarks, the method substantially improves trajectory and surface metrics for both VGGT-Omega and LingBot-MAP, with especially large gains on long outdoor sequences. The ablations show that bounded point sampling, progressive loop scheduling, robust weighting, and sparse analytic Jacobians are central to making the hierarchy computationally practical. The remaining technical questions concern robustness to incorrect or missing loop closures, absolute metric-scale evaluation, joint geometric refinement, and incremental operation.