Mem4D: Online Dynamic 3D Reconstruction

• Mem4D is an online method for dense 3D scene reconstruction that decouples persistent structure from transient motion via a dual-memory architecture.
• It combines a ViT encoder with a Temporal Context Aggregator and alternating queries to Transient Dynamics Memory (TDM) and Persistent Structure Memory (PSM) to resolve motion ambiguity.
• Mem4D achieves state-of-the-art depth and camera pose estimation at about 16 FPS, offering robust online inference in dynamic environments.

Searching arXiv for Mem4D and closely related baseline papers to ground citations. Mem4D is an online method for dense dynamic scene reconstruction from monocular video that reconstructs dense 3D pointmaps and camera parameters while explicitly separating transient motion from persistent structure (Cai et al., 11 Aug 2025). It is motivated by what the paper terms the Memory Demand Dilemma: a single memory representation cannot simultaneously preserve the long-term stability required for static structures and the rapid, high-fidelity detail retention required for dynamic motion. In unified-memory systems, making memory sufficiently stable to keep walls, floors, and other static structures drift-free over long sequences tends to average out transient motion, whereas making memory sufficiently responsive to preserve motion destabilizes static geometry. Mem4D addresses this by introducing a dual-memory architecture consisting of a Transient Dynamics Memory (TDM) and a Persistent Structure Memory (PSM), queried alternately during decoding. The reported result is state-of-the-art or competitive performance on video depth estimation, camera pose estimation, and 3D scene reconstruction, while maintaining fully online inference and about 16 FPS on Stereo4D at $512 \times 384$ (Cai et al., 11 Aug 2025).

1. Problem setting and the Memory Demand Dilemma

Mem4D takes a monocular video stream

$\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$

and reconstructs dense 3D pointmaps

$\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$

together with camera parameters

$$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$

where $K_t$ is the intrinsics matrix and $T_t$ is the pose at time $t$ (Cai et al., 11 Aug 2025). The method is fully online: it processes frames sequentially and updates memory incrementally, rather than waiting for the whole video or requiring costly global alignment.

The central diagnosis is that prior memory-based reconstructions force static geometry and dynamic motion to compete for the same memory budget. In the formulation given in the paper, if the memory is tuned for long-term stability, transient motion is blurred; if it is tuned for responsiveness, static geometry drifts. The paper presents this not as an implementation artifact but as an inherent representational conflict in unified-memory systems. A common misconception would be to treat this trade-off as merely a capacity problem. Mem4D instead frames it as a mismatch between two distinct temporal regimes: persistent structure and high-frequency motion. This suggests that the decisive issue is not only memory size, but also the temporal semantics assigned to memory contents.

2. Front-end temporal encoding

The architecture begins with a ViT encoder that converts each frame $I_t$ into a feature map

$F_t \in \mathbb{R}^{H \times W \times C}.$

Because a single frame can be ambiguous in dynamic scenes, Mem4D first applies a Temporal Context Aggregator (TCA) to enrich the current frame with a local history of past features (Cai et al., 11 Aug 2025).

TCA considers a short window of $k_t$ previous feature maps $\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$0, compresses them according to temporal distance, concatenates them with the current tokens, and runs self-attention over the resulting sequence. The distance-aware compression uses strides

$\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$1

and compressed features

$\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$2

These compressed past tokens are concatenated with the current frame tokens and processed by four self-attention layers using 3D Rotary Position Embeddings (3DRoPE) to encode spatio-temporal positions jointly. The result is an enriched representation $\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$3, which serves as the motion-aware input to the dual-memory system.

In implementation, the image encoder is ViT-Large, initialized from CUT3R weights and frozen during training. The TCA window size is $\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$4. The emphasis on short-range aggregation before explicit memory readout indicates that Mem4D does not rely on persistent memory alone for temporal reasoning; it first resolves immediate temporal ambiguity through local context, then uses memory for longer-range stabilization and motion retention.

3. Dual-memory architecture

The defining feature of Mem4D is its explicit decoupling of static geometry and dynamic motion into two specialized memories (Cai et al., 11 Aug 2025).

Component	Primary role	Key implementation detail
TDM	High-frequency, short-term motion cues	$\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$5; built on-the-fly from recent correlations
PSM	Long-term static consistency	$\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$6; FIFO queue with permanent initial anchor
Shared design choice	Memory querying	Alternating readout in the decoder

The Transient Dynamics Memory (TDM) is designed to preserve high-frequency, short-term motion cues. It stores $\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$7 motion-aware feature maps: $\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$8 Rather than serving as a persistent bank that accumulates all history, the TDM is built on-the-fly at each timestep from correlations between the current enriched feature map $\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}$9 and recent past features $\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$0, for $\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$1. For each past frame, Mem4D computes a 4D correlation volume

$\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$2

The paper notes that this is analogous in spirit to RAFT-style matching, since the 4D volume captures pixel-to-pixel correspondences across the current and past frame. A multi-scale correlation pyramid $\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$3 is created by pooling the last two dimensions at different scales, then concatenated and projected by an MLP: $\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$4 where $\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$5 means concatenation. This motion feature is refined by a motion encoder $\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$6 consisting of four self-attention layers with 3DRoPE: $\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$7

The Persistent Structure Memory (PSM) is designed for long-term static consistency. It stores a feature bank

$\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$8

At time $\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}$9, Mem4D updates this memory by appending a new feature map $$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$0, produced by a lightweight encoder $$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$1 that takes the predicted global pointmap $$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$2 and compresses it into a structural representation: $$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$3 The PSM is maintained as a FIFO queue of size $$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$4, with a crucial exception: the initial memory $$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$5 is never removed and acts as a permanent global anchor to combat drift. Before readout, the PSM is temporally compressed using 3D convolutions whose kernel sizes depend on temporal distance $$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$6: $$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$7 Older memories are compressed more aggressively, while recent memories retain more detail; the initial frame is never compressed.

The feature dimension for both TDM and PSM is 768 channels. Architecturally, the separation is substantive rather than nominal: the TDM stores correlation-derived motion information, whereas the PSM stores compressed geometric structure derived from predicted global pointmaps. This suggests a division not only by time scale but also by representational type.

4. Alternating readout, prediction heads, and online update

The decoupling of static geometry and dynamic motion persists through the decoding stage (Cai et al., 11 Aug 2025). Mem4D does not merge the TDM and PSM into a single latent state; instead, it alternates between them during an iterative fusion decoder over $$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$8 stages.

The current frame tokens $$\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},$$9 and a learnable camera token $K_t$0 are refined by alternating memory queries: $K_t$1 The algorithm described in the paper is:

• start from $K_t$2, $K_t$3, TDM $K_t$4, and compressed PSM $K_t$5;
• for each layer $K_t$6:
  • use MotionReadBlock to attend to TDM and refine the current tokens, giving $K_t$7, $K_t$8;
  • use SpatialReadBlock to attend to PSM and further refine them, giving $K_t$9, $T_t$0;
• output final tokens $T_t$1, $T_t$2.

In the paper’s description, Mem4D first injects high-frequency motion detail from the TDM and then grounds that result in long-term structural context from the PSM. The ordering matters: motion is resolved first, then stabilized spatially. Both read blocks are implemented with self-attention and 3DRoPE.

After $T_t$3 iterations, the final frame tokens $T_t$4 are passed to two DPT heads: $T_t$5

$T_t$6

and the camera token $T_t$7 is passed to a camera head: $T_t$8 The predicted global pointmap $T_t$9 is then encoded into the PSM via $t$0 and appended to the FIFO queue, closing the online update loop. The memory therefore evolves as the video streams in, rather than being recomputed from scratch.

5. Training objective, implementation, and empirical performance

Mem4D is trained end-to-end on sequences of $t$1 images with a weighted sum of a pointmap reconstruction loss and camera losses (Cai et al., 11 Aug 2025). The paper states that the 3D regression loss follows MASt3R’s confidence-aware formulation; the printed equation in the excerpt appears garbled in the PDF text, but the intended form is the standard confidence-weighted robust regression used in MASt3R. The variables identified in the text are the predicted 3D point $t$2, the ground-truth 3D point $t$3, the predicted confidence $t$4, a scale normalization factor $t$5 from ground truth, and a weighting constant $t$6.

The absolute pose and intrinsics loss is

$t$7

where $t$8 are predicted and ground-truth quaternions, $t$9 are predicted and ground-truth translations, and $I_t$0 are ground-truth and predicted intrinsics. The relative pose loss is

$I_t$1

with $I_t$2. The total objective is

$I_t$3

The implementation is intentionally lightweight for online use. Optimization uses AdamW with learning rate $I_t$4 and linear warmup + cosine decay. Training uses 8 NVIDIA A100 GPUs, batch size 4 per GPU, and a two-stage curriculum: first fixed-length 11-frame clips, then longer clips by sampling 11 to 48 frames. The paper states that this curriculum is important because the second stage teaches the model to handle long-range drift and extended video sequences.

Evaluation spans three tasks: video depth estimation, camera pose estimation, and 3D scene reconstruction. The video depth benchmarks are Sintel, BONN, and KITTI, using Abs Rel and $I_t$5, in both Per-Scene alignment and Metric settings. The camera pose benchmarks are Stereo4D, Sintel, and TUM-dynamics, using ATE, RPE trans, and RPE rot. The 3D reconstruction benchmarks are 7-Scenes and NRGBD, using Acc and Comp.

The main reported results are as follows. In video depth estimation, Mem4D is strongest in the metric, no-alignment setting: Sintel metric Abs Rel is 0.846 for Mem4D versus 1.029 for CUT3R, and BONN metric Abs Rel is 0.086 versus 0.103; the paper states that these correspond to improvements of 21.6% on Sintel and 19.7% on Bonn relative to CUT3R. In the per-scene aligned setting, Mem4D reaches 0.072 Abs Rel on BONN, slightly better than CUT3R’s 0.078, and remains competitive on KITTI. In camera pose estimation, representative ATE values are 0.495 on Stereo4D, 0.263 on Sintel, and 0.061 on TUM-dynamics. The paper emphasizes that Mem4D is fully online and does not use post-processing, unlike some higher-accuracy alignment-based baselines. In 3D reconstruction, reported means are 0.185 Acc and 0.178 Comp on 7-Scenes, and 0.271 Acc and 0.212 Comp on NRGBD. Efficiency-wise, the paper reports about 16 FPS on Stereo4D at $I_t$6, and elsewhere reports 16.12 FPS on Stereo4D.

6. Ablations, qualitative behavior, limitations, and significance

The ablation study on Sintel isolates the role of each major component (Cai et al., 11 Aug 2025). Removing second-stage long-sequence training hurts performance, indicating the importance of long-clip training for controlling drift. Removing relative pose loss degrades pose estimation, showing that temporal consistency supervision matters. Removing the TDM causes the most severe depth drop, confirming that motion-aware detail is essential for dynamic scenes. Removing the PSM significantly hurts nearly all metrics, validating it as the long-term geometric anchor. Removing the TCA sharply degrades pose accuracy, showing that local temporal context is important for disambiguating motion. The full model performs best across the reported metrics.

The qualitative findings align with the paper’s central claim. In the introductory illustration, unified memory causes wall drift and motion blur on a dancer, whereas Mem4D’s decoupled design produces stable walls and sharp moving humans. On Bonn and DAVIS, Mem4D is reported to outperform MonST3R and CUT3R visually, with better geometry on background structures and fewer artifacts on moving objects. The paper emphasizes that the PSM suppresses cumulative drift over time, while the TDM prevents moving regions from being over-smoothed.

Two limitations are explicitly noted. First, as a feed-forward online method, Mem4D can still accumulate drift over extremely long sequences. Second, the method is supervised and depends on dense 4D ground truth, which is scarce and hard to obtain; this limits scalability and generalization. The paper suggests that integrating Bundle Adjustment could further improve long-term accuracy, and that self-supervised training would be an important future direction.

Within the landscape of dynamic scene reconstruction, Mem4D’s significance lies in its conceptual reframing of memory usage. The paper’s contribution is not only a stronger memory module, but the claim that static and dynamic content have different temporal behaviors and should not compete for the same memory budget. By splitting the problem into a transient motion memory and a persistent structural memory, and by alternating queries between them during decoding, Mem4D resolves the stability-versus-fidelity trade-off that hinders prior unified-memory systems.