HorizonStream: Long-Horizon Attention for Streaming 3D Reconstruction

Abstract: Online 3D reconstruction requires estimating camera pose and scene geometry under strict causal and bounded-memory constraints. Existing methods often suffer from drift, jitter, or collapse on long sequences. We trace these failures to a fundamental mismatch. Streaming geometry is inherently temporally heterogeneous, with evidence ranging from short-lived correspondences to persistent global scale. However, current architectures impose uniform and pathological influence patterns. For example, sliding windows enforce hard cutoffs, while ungated recurrence and causal attention cause cache saturation and spike-like attention sinks. To resolve this, we formalize geometric propagation as an \emph{evidence influence kernel} and propose HorizonStream, a long-horizon Transformer that explicitly factorizes this kernel. For the long-range temporal factor, Geometric Linear Attention learns channel-wise decay rates to enable bounded, multi-timescale propagation of geometric evidence. For the short-range spatial factor, Geometric Local Attention with Spatiotemporal RoPE performs reliable 3D matching while suppressing attention sinks. Finally, Metric Readout Tokens recover stable scale and rigid pose directly from the persistent geometric state. Extensive experiments show that HorizonStream, trained on only 48-frame clips, generalizes stably to sequences exceeding 10,000\ frames with constant memory and linear time, achieving state-of-the-art streaming 3D reconstruction performance. Project Page: https://3dagentworld.github.io/horizonstream/

• The paper introduces a factorized architecture that combines channel-wise gated linear attention for bounded, multi-timescale memory with gated local attention and spatiotemporal RoPE for reliable geometric matching.
• HorizonStream trained on 48-frame clips generalizes to sequences exceeding 10,000 frames with constant memory and linear-time inference, achieving 19.75 m KITTI ATE and 25.30 m average ATE on long VBR sequences without loop closure.
• The results show that learned retention spectra, metric readout tokens, and local relative-pose fusion reduce drift and state contamination, while remaining limitations include detail loss during dense revisits and corruption from dynamic objects.

HorizonStream addresses a persistent failure mode in online 3D reconstruction: streaming methods that perform well on short clips degrade, jitter, or collapse when run over long sequences. The authors attribute this to a mismatch between the temporal heterogeneity of geometric evidence and the uniform retention policies imposed by existing architectures, and they propose an architecture that factorizes evidence propagation into explicit long-range and short-range components (2605.23889).

The evidence influence kernel formulation

The paper's central conceptual move is to formalize streaming reconstruction as aggregation of geometric evidence through a spatio-temporal influence kernel $K(t,i)$, which weights the contribution of past frame $i$ to the current state at time $t$. Under this lens, common long-sequence failures become pathological kernel shapes: sliding-window methods (STream3R, StreamVGGT) impose hard-cutoff box kernels that discard still-valid evidence; periodic refresh mechanisms (LongStream) create blockwise discontinuities; causal softmax attention degenerates into spike-like attention sinks; and ungated recurrence (CUT3R, TTT3R) produces heavy-tailed kernels with unbounded error accumulation.

The appendix formalizes two of these claims. A "geometric attention dilution" proposition shows that under bounded logits, the fraction of softmax attention mass on co-visible frames decays as $O(W_{\mathrm{geo}} e^{2M}/t)$, so beyond roughly $W_{\mathrm{geo}}(1+e^{2M})$ frames more than half of attention falls on irrelevant frames — even with perfect score discrimination. Separately, a "zero-forgetting contamination" proposition shows that ungated recurrence ($\gamma \equiv 1$) leaves the initial state contributing to every output with undiminished magnitude, permanently anchoring predictions to initialization. These results imply that neither extreme of the retention spectrum — immediate forgetting or permanent retention — is viable; what is needed is a bounded, learnable decay profile.

Geometric Linear Attention

The long-range temporal factor is instantiated as gated linear attention derived from a discounted state-estimation objective. The recurrent state $S_t \in \mathbb{R}^{d\times d}$ is updated as $$S_t = \mathrm{diag}(\gamma_t) S_{t-1} + \phi(k_t)\tilde{v}_t^\top$$, where $$\gamma_t = \sigma(W_\gamma x_t + b_\gamma) \in (0,1)^d$$ is a channel-wise retention vector. Three theoretical guarantees support this design: exponential decay of initial-state influence when $\bar{\gamma} < 1$, per-channel memory horizons $i$0 yielding heterogeneous implicit partitioning into fast and slow subspaces, and a uniform Frobenius-norm bound $i$1 that eliminates the need for periodic resets. The paper also establishes that the update is equivalent to discounted test-time training, extending the TTT–linear-attention equivalence to the gated setting.

Ablations on vKITTI2 confirm each element matters, with degradation growing with sequence length: removing Geometric Linear Attention raises ATE from 1.20 m to 5.38 m at 1000 frames; replacing channel-wise gating with scalar gating yields 3.21 m; substituting TTT-like fast weights gives 3.96 m. Learned lifetimes form a continuous spectrum across channels and layers, and replacing this spectrum with any fixed short/medium/long band degrades accuracy — supporting the claim that multi-timescale retention, not any single horizon, is what stabilizes long streams.

Geometric Local Attention and metric readout

Within each window, Geometric Local Attention performs content-based 3D matching with two safeguards against sink formation and noisy correspondences: head-wise sigmoid output gates conditioned on mean-pooled window features, and Spatiotemporal RoPE extended to time, height, and width axes with periodic temporal-index reset. Ablations show these are complementary — removing either increases drift substantially at 1000 frames (4.06 m without head gating; 2.58 m with 2D-only RoPE), while both together keep error flat.

Metric consistency is handled by Metric Readout Tokens (MRT), one per frame, which participate in Geometric Linear Attention and read scale from high-retention channels via $i$2. Pose is estimated by relative pose fusion over pose tokens within the local window rather than sequential keyframe chaining, avoiding compounding composition errors. Notably, the model predicts poses from only a 10-frame window, which the authors argue suffices because long-horizon structure is carried by the recurrent state.

Experimental results

Training uses only 48-frame clips processed in 21-frame chunks, yet inference generalizes to sequences exceeding 10,000 frames with constant memory and linear time. On KITTI, HorizonStream achieves 19.75 m average ATE versus 25.29 m for Lingbot-map and 51.90 m for LongStream among online methods, approaching optimization-based LoGeR* (16.35 m); with loop closure it reaches 16.44 m. On VBR sequences of 8,815–18,846 frames (up to 5.2 km), it attains 25.30 m average ATE without loop closure and 18.84 m with, outperforming all online baselines including Lingbot-map (27.53 m) and offline LoGeR* (25.24 m). On Oxford Spires it achieves 9.38 m average ATE, well below Lingbot-map's 15.46 m. Across six datasets it posts the best or near-best online results (e.g., Waymo 0.46 m, TUM 0.04 m), and on KITTI video depth it records Abs Rel 0.057, competitive with offline VGGT (0.061).

Two controlled comparisons strengthen the causal argument. Refresh/no-refresh variants of CUT3R, TTT3R, and LoGeR show sharp degradation without periodic reset, indicating state contamination rather than limited capacity. Linear probes on frozen states show segment-scale log error is reliably decodable, with probe weight distributed across short-, medium-, and long-retention bands — consistent with metric evidence being spread across the learned spectrum rather than confined to slow channels.

Limitations and open questions

The paper concedes several constraints. The fixed-size recurrent state loses fine-grained detail on ultra-long sequences with dense revisits, which is the dominant failure mode; loop closure mitigates but does not eliminate it, and the module is parameterized separately with untuned optimization settings. Dynamic foreground objects can corrupt local geometric evidence. The 10-frame pose window may limit internal loop-closure ability, though enlarging it trades off memory and speed. Several evaluation caveats also apply: vKITTI2, 7Scenes, and Waymo appear in training data (with held-out Waymo segments), and ScanNet++ ground truth was unavailable for COLMAP comparison. An open question is whether the learned retention spectrum can be made adaptive at inference to revisit density, rather than fixed by training statistics.

Conclusion

HorizonStream reframes long-sequence streaming reconstruction failures as consequences of pathological evidence-influence kernels and resolves them with a factorized architecture: channel-wise gated linear attention for bounded multi-timescale propagation, gated local attention with spatiotemporal RoPE for reliable matching, and metric readout tokens for stable scale and pose. Trained on 48-frame clips, it generalizes past 10,000 frames with $i$3 memory and linear time, achieving state-of-the-art streaming results across indoor, outdoor, and kilometer-scale benchmarks.