Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstruction

Abstract: Streaming 3D reconstruction under a strict constant-memory budget hinges on how the recurrent state is updated as the stream evolves. We profile TTT3R-style per-token gates across five benchmarks and discover a structural bottleneck: the gate is intrinsically bounded in magnitude (median $0.31$; never exceeding $0.6$) and nearly frame-invariant, yielding an effective memory horizon of only $\sim$3 frames per state token, which serves as the structural origin of long-sequence drift. We trace this to a missing axis: existing inference-time methods modulate updates only at the per-token, intra-frame level, while the orthogonal frame-level question of \emph{how strongly each frame should contribute to the state} has been treated as content-independent. We close this gap with a scalar frame-level gate $α_t \in (0, 1]$ derived in closed form from frame-to-frame changes of internal features -- a continuous relaxation of classical Simultaneous Localization and Mapping (SLAM) keyframe selection that requires no parameters, no training, and no extra forward pass. Across six benchmarks spanning camera pose, video depth, and 3D reconstruction at sequence lengths up to $4,541$ frames, our gate cuts ATE by $51\%$ on long TUM-RGBD pose sequences, reduces AbsRel by $12.8\%$ on Bonn video depth, and on KITTI long-sequence pose estimation surpasses both LongStream and Keyframe-VO, while retaining strictly constant memory at zero training cost.

• The paper introduces Adaptive Frame Gating, which mitigates state drift in recurrent streaming 3D reconstruction by dynamically modulating frame updates.
• It replaces constant per-token gating with content-adaptive frame-level gates computed from image and pose feature deltas without additional parameters.
• Experimental results demonstrate significant reductions in pose error and improved 3D reconstruction quality across diverse benchmarks.

Adaptive Frame Gating for Robust Constant-Memory Streaming 3D Reconstruction

Introduction and Motivation

Streaming 3D reconstruction in the constant-memory regime requires continual updating of a fixed-size state based on a sequence of incoming RGB frames. Unlike offline approaches that aggregate all frames at quadratic cost, e.g., DUSt3R, MASt3R, Fast3R, or VGGT, constant-memory methods such as CUT3R employ a recurrent state that absorbs image features via cross-attention and seeks to maintain a sufficient scene representation under a strict memory budget. The central challenge is drift: over long sequences, accumulation of small errors during incremental updates leads to growing divergence in estimated geometry and camera pose. This paper undertakes a structural analysis of this drift, isolating a critical failure mode in the per-token gating mechanism that governs state updates in modern test-time training (TTT) style methods like TTT3R.

Structural Analysis of State Update Gating

Contemporary streaming models modulate recurrent updates at inference time via per-token gates ($\beta_t \in [0,1]^N$), which are derived as soft learning rates from cross-attention statistics. However, empirical profiling shows that these gates are bounded in magnitude ($\mathrm{median}\approx0.31$, $\mathrm{max}<0.6$) and exhibit negligible frame-to-frame variation across diverse benchmarks (TUM-RGBD, ScanNet, Bonn, NRGBD, 7-Scenes). Their distribution is dominated by architecture-intrinsic averaging, leading to nearly frame-invariant gating where redundancy in the input (e.g., repeated frames) is not recognized by the update rule.

This bounded and content-insensitive gating results in a very short effective memory horizon: each frame's effect on the state diminishes exponentially, with a time constant defined by $1/\bar{\beta}$—empirically about three frames. Informative keyframes are rapidly overwritten, and early evidence is lost irretrievably within several steps, providing a principled explanation for long-sequence drift observed in these models.

Figure 1: Distributional properties and invariance of the per-token gate $\beta$ across datasets and frames.

Adaptive Frame Gating: Methodology

To address this core limitation, the paper introduces Adaptive Frame Gating (AFG), replacing the constant, content-invariant per-frame update with a scalar frame-level gate $\alpha_t \in (0,1]$. This gate is computed in closed form as a function of internal feature deltas across frames, requiring no additional parameters, training, or forward computation.

Two instantiations are considered:

• AFG-Img: Computes $\alpha_t$ from the $\ell_2$ norm between consecutive global encoder features, capturing raw visual novelty.
• AFG-Pose: Uses the difference between decoder pose token features across frames, making the gating signal sensitive to predicted camera motion.

The state update becomes $$S_t = S_{t-1} + \alpha_t \cdot \beta_t \odot \Delta S_t$$, ensuring that only frames exhibiting sufficient novelty strongly influence the model's memory. The strict lower bound of $\sigma(-\tau)$ (from the logistic) sidesteps zeroing out, yielding stability even under complete redundancy.

Figure 2: Adaptive Frame Gating—$\mathrm{median}\approx0.31$0 responds sharply to scene-change; existing methods update indiscriminately and accumulate drift.

Experimental Results

Camera Pose Estimation: On TUM-RGBD and ScanNet, both AFG-Img and AFG-Pose outperform TTT3R, MeMix, and TTSA3R not only at all length scales but with a widening advantage as the trajectory grows. On long (>600 frame) TUM-RGBD sequences, AFG-Pose achieves a 51% ATE reduction relative to TTT3R. Similar trends are observed on ScanNet, where AFG-Img excels due to smoother motion regimes.

Figure 3: Camera pose estimation error (ATE) scales poorly with sequence length in previous methods but is robustly suppressed by AFG.

On KITTI Odometry (outdoor, high-speed), AFG-Pose outperforms both specialized retrained models (LongStream, Keyframe-VO), achieving the lowest average ATE (43.96m versus 51.9m for LongStream and 68.5m for TTT3R), despite its parameter-free, non-retrained nature.

Video Depth Estimation: On the dynamic Bonn dataset, AFG variants maintain lower Absolute Relative (AbsRel) and RMSE errors across all lengths compared to prior methods, with improvement scaling on long sequences. No performance degradation is observed on short sequences, indicating no trade-off between short-term and long-horizon accuracy.

Figure 4: Metric-aligned video depth estimation on Bonn—AFG variants are consistently better, especially over long runs.

3D Reconstruction: Evaluations on 7-Scenes and NRGBD indicate that without adaptive gating, recurrent models accumulate pose drift and yield fragmented reconstructions. AFG strongly stabilizes geometry and surface completeness as measured by multiview accuracy, completeness, and normal consistency, with especially pronounced gains at larger sequence lengths.

Figure 5: Qualitative 3D reconstructions—adaptive gating yields coherent surfaces, unlike drift-ridden baselines.

Ablations and Analysis: Fixed-$\mathrm{median}\approx0.31$1 gating improves over TTT3R, but is always outperformed by the content-adaptive AFG, confirming the necessity of samplewise adaptivity. Frame-level gates alone account for most gains, with token-level $\mathrm{median}\approx0.31$2 providing complementary selectivity. Attempts to combine AFG-Img and AFG-Pose result in negligible or negative improvements, indicating substantial correlation and redundancy in their signals. Threshold sweep experiments demonstrate insensitivity to hyperparameter choice.

Mechanism Verification

A controlled redundancy injection experiment validates the mechanism: upon appending pixel-identical frames to a sequence, traditional gates ($\mathrm{median}\approx0.31$3) remain unchanged and allow drift to accumulate, but the frame-level gate ($\mathrm{median}\approx0.31$4) rapidly suppresses further updates, reducing cumulative geometric drift by over an order of magnitude.

Figure 6: When inserting frame redundancy, only adaptive gates respond—posing and position drift are suppressed by up to 18$\mathrm{median}\approx0.31$5.

Theoretical and Practical Implications

The analysis reveals a structural bottleneck in current inference-time gating designs for streaming 3D reconstruction. By restoring frame-level adaptivity, AFG demonstrably extends the usable memory horizon of constant-memory architectures without retraining or parameterization. Practically, this approach enables high-fidelity, low-drift online mapping on long video sequences with strict compute constraints. The method is plug-and-play for recurrent architectures, posing minimal engineering overhead.

Theoretically, the work highlights that token-level attention statistics are insufficient for redundancy rejection in streaming settings, and classic concepts such as keyframe selection must be reintroduced in relaxed, differentiable forms to modern pipelines. The fact that AFG closes most of the gap to retraining-based long-horizon models suggests that architectural capacity is not always the limiting factor; data-dependent admission is equally crucial.

Future Directions

While AFG addresses the content-independence failure mode of existing recurrent gates, it does not solve state saturation under sustained novelty, and the gating signals provided by image/pose features are not fully orthogonal. Future work could leverage a learned router to dynamically select between gating sources under scene-dependent priors, or introduce lightweight architectures that enable longer intrinsic memory horizons without architectural retraining. Extension to additional modalities or joint optimization with downstream object or semantic supervision also presents a promising avenue.

Conclusion

This work identifies and directly addresses a key architectural bottleneck in recurrent streaming 3D reconstruction: the absence of content-aware, frame-level gating in state updates. By deploying a minimal, parameter- and training-free adaptive gate, computed from existing model features, strong numerical and qualitative gains are realized across pose, depth, and geometry benchmarks, all while preserving strict constant-memory operation. The findings reinforce the value of principled modular innovations in neural architecture design and highlight the enduring importance of data-driven admission control in sequential vision tasks.