RayMap3R: Streaming Dynamic 3D Reconstruction

• RayMap3R is a training-free streaming framework for dynamic 3D reconstruction that jointly estimates scene geometry and camera poses while suppressing moving objects.
• It leverages a dual-branch inference strategy by contrasting image-conditioned and RayMap-only predictions to detect dynamic regions without extra fine-tuning.
• The method employs reset metric alignment and state-aware smoothing to maintain metric consistency and stabilize trajectories in long-horizon streaming scenarios.

RayMap3R is a training-free, streaming, feed-forward framework for dynamic 3D reconstruction from RGB image sequences that performs real-time joint estimation of scene geometry and camera poses while explicitly suppressing the effects of moving objects at inference time. Its defining idea is to exploit an intrinsic static-scene bias in RayMap-based predictions: by contrasting a standard image-conditioned prediction with a RayMap-only prediction generated from the same latent state, the method identifies dynamic regions without fine-tuning, auxiliary networks, or test-time backpropagation. It further introduces reset metric alignment and state-aware smoothing to preserve metric consistency and stabilize long-horizon trajectories, and it is reported to achieve state-of-the-art performance among streaming approaches on dynamic scene reconstruction across multiple benchmarks (Wang et al., 21 Mar 2026).

1. Problem setting and architectural scope

RayMap3R addresses streaming feed-forward 3D reconstruction in dynamic scenes. In this setting, a sequence of RGB frames $I_1, I_2, \ldots$ is processed causally, and at each time $t$ the system estimates the camera pose $T_t$, depth $z_t$, and point cloud $P_t$ while maintaining constant compute and memory. The central difficulty is that moving objects violate the static-scene assumption encoded in many feed-forward backbones, so naively writing dynamic content into persistent memory produces transient geometry, pose jitter, and accumulated drift, especially when the memory is periodically reset to mitigate forgetting (Wang et al., 21 Mar 2026).

The framework is built on a pretrained CUT3R backbone with a persistent latent state $s_{t-1}$. At time $t$, the image $I_t$ is patchified into image tokens $f_t$, the corresponding RayMap is patchified into RayMap tokens $r_t$, and both are decoded through the same recurrent model. The contribution of RayMap3R is not a new learned module, but an inference-time control scheme over the frozen backbone: dynamic identification via branch contrast, memory gating, reset-aware Sim(3) alignment, and adaptive trajectory smoothing. This places the method in a distinct regime relative to offline global-attention reconstruction pipelines, since it targets constant-memory streaming rather than all-frame optimization (Wang et al., 21 Mar 2026).

2. RayMap representation and the static-scene bias

For a camera with intrinsics $t$0 and extrinsics $t$1, the per-pixel RayMap stores the world-space camera center and a unit ray direction for each pixel. The paper gives

$t$2

and

$t$3

where $t$4. The resulting RayMap is an $t$5 tensor containing per-pixel $t$6 (Wang et al., 21 Mar 2026).

The key empirical observation is that RayMap-only predictions exhibit a static-scene bias. When the decoder is queried using RayMap tokens alone, without appearance features, it reconstructs geometry by relying on memory and geometric consistency; static structures remain consistent across time and are recalled, whereas dynamic objects are transient and tend to be suppressed. RayMap3R converts this bias into a dynamic cue by comparing the standard image+RayMap prediction against a RayMap-only prediction produced from the same frozen state. The paper reports that the resulting per-pixel depth discrepancy correlates with the ground-truth dynamic ratio, with Spearman $t$7 and $t$8 across large-scale evaluations, which supports using this contrast as an internal dynamic detector rather than introducing an external segmentation or tracking stack (Wang et al., 21 Mar 2026).

A plausible implication is that the method leverages a representational asymmetry already present in the pretrained backbone: appearance introduces dynamic contamination, whereas RayMap-conditioned memory retrieval preferentially recalls stable geometry. RayMap3R therefore operationalizes a bias that would otherwise remain implicit.

3. Dual-branch inference and gated state update

At each time step, RayMap3R runs two decoders from the same frozen state $t$9. The main branch uses both appearance and RayMap:

$T_t$0

producing $T_t$1, $T_t$2, and $T_t$3. The RayMap branch reconstructs a second prediction by first rebuilding a RayMap $T_t$4 from the main branch’s predicted pose and then decoding

$T_t$5

which yields $T_t$6 and $T_t$7 (Wang et al., 21 Mar 2026).

The discrepancy signal is defined per pixel as

$T_t$8

These pixelwise discrepancies are pooled into image-token scores $T_t$9 via confidence-weighted averaging within patches using $z_t$0, and then projected into state-token scores $z_t$1 using the decoder’s cross-attention $z_t$2:

$z_t$3

Staticness weights are then computed with a robust normalization based on the median and interquartile range:

$z_t$4

where $z_t$5 is the sigmoid and $z_t$6 controls dynamic-static sensitivity. State tokens with higher discrepancy receive lower $z_t$7, and an EMA over $z_t$8 is maintained for temporal stability (Wang et al., 21 Mar 2026).

Memory is updated by gating the decoder’s proposed latent update $z_t$9:

$P_t$0

The equivalent mask-style form reported in the paper is

$P_t$1

This prevents state corruption by dynamic content while retaining static structure. The paper also notes that the pixel-level staticness map derived from $P_t$2 can bias pose retrieval toward static regions during memory update, further coupling dynamic suppression with pose stabilization (Wang et al., 21 Mar 2026).

4. Reset metric alignment and state-aware smoothing

Streaming backbones periodically reset memory to control forgetting. RayMap3R uses a 50-frame reset interval, but observes that processing the repeated frame immediately before and after a reset can produce metric inconsistencies, including scale and pose offsets, which then propagate as drift. To correct this, it introduces reset metric alignment: given corresponding points $P_t$3 from the repeated frame before reset and $P_t$4 after reset, it estimates a Sim(3) transform by minimizing

$P_t$5

where $P_t$6 are confidence weights derived from the staticness map of the last pre-reset frame and therefore upweight stable background structure (Wang et al., 21 Mar 2026).

The paper gives a weighted Umeyama/Horn-style closed form. With weighted centroids $P_t$7 and covariance $P_t$8, one computes

$P_t$9

followed by

$s_{t-1}$0

$s_{t-1}$1

and

$s_{t-1}$2

This transform is then applied to all poses and accumulated geometry in the new segment to restore metric consistency across the reset boundary (Wang et al., 21 Mar 2026).

Trajectory stabilization is handled by state-aware smoothing. The proposed state-change magnitude is

$s_{t-1}$3

and the trajectory acceleration is

$s_{t-1}$4

These signals define an adaptive coefficient

$s_{t-1}$5

with $s_{t-1}$6 controlling sensitivity. Translation is filtered as

$s_{t-1}$7

and

$s_{t-1}$8

When both acceleration and state change are large, $s_{t-1}$9 and smoothing strengthens; under sustained constant-velocity motion, acceleration is small and $t$0, so the filter remains minimally intrusive. This mechanism is explicitly designed to stabilize dynamic sequences without imposing uniform temporal damping (Wang et al., 21 Mar 2026).

5. Inference pipeline, evaluation, and computational profile

The full inference-time pipeline begins with a short warmup in which only the main branch is run. Thereafter, each frame is tokenized, decoded through the main branch, remapped into a pose-consistent RayMap for the RayMap-only branch, contrasted through the discrepancy field $t$1, aggregated into state-token staticness weights, written into the persistent state through gated updates, smoothed in translation space, and finally integrated into the global geometry. Every 50 frames, the state is reset and the repeated frame is used for reset metric alignment before processing continues (Wang et al., 21 Mar 2026).

Evaluation spans dynamic datasets—MPI Sintel, TUM-dynamics, KITTI, and Bonn—and static datasets—ScanNet and 7-Scenes. For depth, the reported per-sequence-scale scores are: KITTI AbsRel $t$2, $t$3 $t$4; Bonn AbsRel $t$5, $t$6 $t$7; and Sintel AbsRel $t$8, $t$9 $I_t$0. Under metric-scale evaluation, KITTI reaches AbsRel $I_t$1, $I_t$2 $I_t$3; Bonn reaches AbsRel $I_t$4, $I_t$5 $I_t$6; and Sintel reaches AbsRel $I_t$7, $I_t$8 $I_t$9. For pose, RayMap3R reports on Sintel ATE $f_t$0, RPE$f_t$1 $f_t$2, and RPE$f_t$3 $f_t$4; on TUM-dynamics ATE $f_t$5, RPE$f_t$6 $f_t$7, and RPE$f_t$8 $f_t$9; and on ScanNet ATE $r_t$0, RPE$r_t$1 $r_t$2, and RPE$r_t$3 $r_t$4. For 3D reconstruction on 7-Scenes, the mean metrics are Acc $r_t$5, Comp $r_t$6, NC $r_t$7, and Chamfer $r_t$8 (Wang et al., 21 Mar 2026).

Ablations show that the components are complementary rather than redundant. Relative to the CUT3R base, adding the dual-branch scheme ($r_t$9) improves depth AbsRel from $t$00 to $t$01 and Chamfer from $t$02 to $t$03. Adding metric alignment to the dual-branch model ($t$04) reduces Chamfer further from $t$05 to $t$06. Adding smoothing ($t$07) improves ATE from $t$08 to $t$09. The full model $t$10 reaches ATE $t$11, AbsRel $t$12, and Chamfer $t$13 (Wang et al., 21 Mar 2026).

The computational profile is explicitly streaming. On an RTX A6000 with ScanNet, the method uses $t$14 GB at $t$15 views and $t$16 GB at $t$17 views, while maintaining $t$18 FPS in both cases. The paper attributes the dominant per-frame cost to the two decodes, with discrepancy computation scaling as $t$19, aggregation through cross-attention scaling as $t$20 for $t$21 state tokens and $t$22 image tokens, and reset alignment using weighted SVD on repeated-frame correspondences. This constant-memory behavior is a defining property of the method’s streaming orientation (Wang et al., 21 Mar 2026).

6. Limitations, interpretation, and relation to adjacent ray-centric methods

The reported failure modes are specific to the inference-time design. Extremely large, persistent dynamic occlusions can still corrupt the state despite gating. Textureless or repetitive environments reduce the discriminability of the image-versus-RayMap contrast. Very fast motion and strong rolling-shutter effects can increase pose jitter, although smoothing alleviates rather than eliminates the issue. The strength of the static-scene bias also depends on the backbone’s training distribution, so insufficient exposure to dynamics during pretraining may weaken the internal cue. In addition, the hyperparameters $t$23, $t$24, and the reset interval require modest tuning by application domain (Wang et al., 21 Mar 2026).

A common misconception is to treat RayMap3R as synonymous with any ray-based reconstruction or mapping method. In the 3D vision literature, the closely related CAM3R framework uses a camera-agnostic per-pixel ray mapping formulation in which a Ray Module predicts a unit ray field $t$25, a Cross-View Module predicts radial distances and relative poses, and a Ray-Aware Global Alignment optimizes poses and per-image scales while freezing the ray fields; that formulation targets camera-agnostic reconstruction across pinhole, fisheye, and panorama imagery rather than dynamic-streaming suppression (Guruprasad et al., 23 Mar 2026). In radio mapping, by contrast, RadioFormer3D reconstructs volumetric radio maps from sparse 3D measurements and building-height priors using a dual-stream volumetric architecture and a Joint Spectrum Integrity Loss, while RadioDiff-3D defines a 3D$t$263D radio map dataset and a conditional diffusion benchmark over pathloss, DoA, and ToA; both address learned volumetric spectrum-field estimation rather than RGB-based geometric reconstruction (Fang et al., 28 May 2026, Wang et al., 16 Jul 2025).

This suggests that RayMap3R is best understood as a narrowly defined inference-time method within streaming 3D reconstruction: it does not replace RayMap-centric geometry estimation, but rather augments a pretrained RayMap-based backbone with dynamic-aware state control. Its central contribution is methodological rather than representational. By turning a latent static-scene bias into an explicit gating signal, and by coupling that signal to reset-aware alignment and adaptive smoothing, it narrows the gap between feed-forward streaming reconstruction and the demands of dynamic real-world operation (Wang et al., 21 Mar 2026).