---
title: 'Mem4D: Online Dynamic 3D Reconstruction'
url: https://www.emergentmind.com/topics/mem4d
type: topic
---

# Mem4D: Online Dynamic 3D Reconstruction

Searching arXiv for Mem4D and closely related baseline papers to ground citations.
Mem4D is an online method for dense dynamic scene reconstruction from monocular video that reconstructs dense 3D pointmaps and camera parameters while explicitly separating transient motion from persistent structure [2508.07908]. It is motivated by what the paper terms the **Memory Demand Dilemma**: a single memory representation cannot simultaneously preserve the long-term stability required for static structures and the rapid, high-fidelity detail retention required for dynamic motion. In unified-memory systems, making memory sufficiently stable to keep walls, floors, and other static structures drift-free over long sequences tends to average out transient motion, whereas making memory sufficiently responsive to preserve motion destabilizes static geometry. Mem4D addresses this by introducing a dual-memory architecture consisting of a **Transient Dynamics Memory (TDM)** and a **Persistent Structure Memory (PSM)**, queried alternately during decoding. The reported result is state-of-the-art or competitive performance on video depth estimation, camera pose estimation, and 3D scene reconstruction, while maintaining fully online inference and about 16 FPS on Stereo4D at \(512 \times 384\) [2508.07908].

## 1. Problem setting and the Memory Demand Dilemma

Mem4D takes a monocular video stream
\[
\mathcal{I}=\left\{I_1, I_2, \ldots, I_T\right\}
\]
and reconstructs dense 3D pointmaps
\[
\mathcal{X}=\left\{X_1, X_2, \ldots, X_T\right\}
\]
together with camera parameters
\[
\mathcal{P}=\left\{\left(K_1, T_1\right),\left(K_2, T_2\right), \ldots,\left(K_T, T_T\right)\right\},
\]
where \(K_t\) is the intrinsics matrix and \(T_t\) is the pose at time \(t\) [2508.07908]. The method is fully online: it processes frames sequentially and updates memory incrementally, rather than waiting for the whole video or requiring costly global alignment.

The central diagnosis is that prior memory-based reconstructions force static geometry and dynamic motion to compete for the same memory budget. In the formulation given in the paper, if the memory is tuned for long-term stability, transient motion is blurred; if it is tuned for responsiveness, static geometry drifts. The paper presents this not as an implementation artifact but as an inherent representational conflict in unified-memory systems. A common misconception would be to treat this trade-off as merely a capacity problem. Mem4D instead frames it as a mismatch between two distinct temporal regimes: persistent structure and high-frequency motion. This suggests that the decisive issue is not only memory size, but also the temporal semantics assigned to memory contents.

## 2. Front-end temporal encoding

The architecture begins with a **ViT encoder** that converts each frame \(I_t\) into a feature map
\[
F_t \in \mathbb{R}^{H \times W \times C}.
\]
Because a single frame can be ambiguous in dynamic scenes, Mem4D first applies a **Temporal Context Aggregator (TCA)** to enrich the current frame with a local history of past features [2508.07908].

TCA considers a short window of \(k_t\) previous feature maps \(\{F_{t-j}\}_{j=1}^{k_t}\), compresses them according to temporal distance, concatenates them with the current tokens, and runs self-attention over the resulting sequence. The distance-aware compression uses strides
\[
s_j = \phi(j) =
\begin{cases}
1, & \text{if } 0 \le j < 2 \\
2, & \text{if } 2 \le j < 4 \\
4, & \text{if } j \ge 4
\end{cases}
\]
and compressed features
\[
\tilde{F}_{t-j} = \text{Conv}_{\phi(j)}(F_{t-j}) \quad \forall j \in \{1, \dots, k_t\}.
\]
These compressed past tokens are concatenated with the current frame tokens and processed by four self-attention layers using **3D Rotary Position Embeddings (3DRoPE)** to encode spatio-temporal positions jointly. The result is an enriched representation \(T_t'\), which serves as the motion-aware input to the dual-memory system.

In implementation, the image encoder is **ViT-Large**, initialized from CUT3R weights and **frozen** during training. The TCA window size is \(k_t=5\). The emphasis on short-range aggregation before explicit memory readout indicates that Mem4D does not rely on persistent memory alone for temporal reasoning; it first resolves immediate temporal ambiguity through local context, then uses memory for longer-range stabilization and motion retention.

## 3. Dual-memory architecture

The defining feature of Mem4D is its explicit decoupling of static geometry and dynamic motion into two specialized memories [2508.07908].

| Component | Primary role | Key implementation detail |
|---|---|---|
| TDM | High-frequency, short-term motion cues | \(k_d=2\); built on-the-fly from recent correlations |
| PSM | Long-term static consistency | \(k_s=100\); FIFO queue with permanent initial anchor |
| Shared design choice | Memory querying | Alternating readout in the decoder |

The **Transient Dynamics Memory (TDM)** is designed to preserve high-frequency, short-term motion cues. It stores \(k_d\) motion-aware feature maps:
\[
\mathcal{M}^D \in \mathbb{R}^{k_d \times H \times W \times C_m}.
\]
Rather than serving as a persistent bank that accumulates all history, the TDM is built on-the-fly at each timestep from correlations between the current enriched feature map \(T_t'\) and recent past features \(T_{t-j}'\), for \(j=1,\dots,k_d\). For each past frame, Mem4D computes a 4D correlation volume
\[
C_{t,t-j} = T_t^{\prime} \times (T_{t-j}^{\prime})^T \in \mathbb{R}^{H \times W \times H \times W}.
\]
The paper notes that this is analogous in spirit to RAFT-style matching, since the 4D volume captures pixel-to-pixel correspondences across the current and past frame. A multi-scale correlation pyramid \(\{C^l_{t,t-j}\}_{l=1}^{L}\) is created by pooling the last two dimensions at different scales, then concatenated and projected by an MLP:
\[
M_{t,t-j} = \text{MLP}\left(C^{1}_{t,t-j}\oplus C^{2}_{t,t-j}, \ldots, C^{L}_{t,t-j}\right),
\]
where \(\oplus\) means concatenation. This motion feature is refined by a motion encoder \(E_m\) consisting of four self-attention layers with 3DRoPE:
\[
\mathcal{M}^D_{t,t-j} = E_m\left(M_{t,t-j}\right) \in \mathbb{R}^{H \times W \times C_m}.
\]

The **Persistent Structure Memory (PSM)** is designed for long-term static consistency. It stores a feature bank
\[
\mathcal{M}^S \in \mathbb{R}^{k_s \times H \times W \times C_s}.
\]
At time \(t\), Mem4D updates this memory by appending a new feature map \(\mathcal{M}^S_t\), produced by a lightweight encoder \(E_s\) that takes the predicted global pointmap \(\hat{X}^{global}_t\) and compresses it into a structural representation:
\[
\mathcal{M}^S_t = E_s\left(\hat{X}^{global}_t\right) \in \mathbb{R}^{H \times W \times C_s}.
\]
The PSM is maintained as a **FIFO queue** of size \(k_s\), with a crucial exception: the initial memory \(\mathcal{M}^S_0\) is never removed and acts as a permanent global anchor to combat drift. Before readout, the PSM is temporally compressed using 3D convolutions whose kernel sizes depend on temporal distance \(d=t-j\):
\[
s_d =
\begin{cases}
(4, 8, 8), & \text{if } d \ge 6 \\
(2, 4, 4), & \text{if } 4 \le d < 6 \\
(1, 2, 2), & \text{if } 2 \le d < 4 \\
(1, 1, 1), & \text{otherwise.}
\end{cases}
\]
Older memories are compressed more aggressively, while recent memories retain more detail; the initial frame is never compressed.

The feature dimension for both TDM and PSM is **768 channels**. Architecturally, the separation is substantive rather than nominal: the TDM stores correlation-derived motion information, whereas the PSM stores compressed geometric structure derived from predicted global pointmaps. This suggests a division not only by time scale but also by representational type.

## 4. Alternating readout, prediction heads, and online update

The decoupling of static geometry and dynamic motion persists through the decoding stage [2508.07908]. Mem4D does not merge the TDM and PSM into a single latent state; instead, it alternates between them during an **iterative fusion decoder** over \(L\) stages.

The current frame tokens \(T_t^{\prime(0)}\) and a learnable camera token \(T_t^{c(0)}\) are refined by alternating memory queries:
\[
\text{MotionReadBlock} \rightarrow \text{SpatialReadBlock} \rightarrow \text{MotionReadBlock} \rightarrow \cdots
\]
The algorithm described in the paper is:

- start from \(T_t^{\prime(0)}\), \(T_t^{c(0)}\), TDM \(\mathcal{M}^{D(0)}\), and compressed PSM \(\mathcal{M}^{S(0)}\);
- for each layer \(l=1,\dots,L\):
  - use **MotionReadBlock** to attend to TDM and refine the current tokens, giving \(\tilde{T}_t^{c(l)}\), \(\tilde{T}_t^{\prime(l)}\);
  - use **SpatialReadBlock** to attend to PSM and further refine them, giving \(T_t^{c(l)}\), \(T_t^{\prime(l)}\);
- output final tokens \(T_t^{c(L)}\), \(T_t^{\prime(L)}\).

In the paper’s description, Mem4D first injects high-frequency motion detail from the TDM and then grounds that result in long-term structural context from the PSM. The ordering matters: motion is resolved first, then stabilized spatially. Both read blocks are implemented with self-attention and 3DRoPE.

After \(L\) iterations, the final frame tokens \(\hat{T}_t' = T_t^{\prime(L)}\) are passed to two DPT heads:
\[
\hat{X}^{global}_t, C^{global}_t = \text{Head}_{global}\left(\hat{T}_t^{\prime}\right),
\]
\[
\hat{X}^{self}_t, C^{self}_t = \text{Head}_{self}\left(\hat{T}_t^{\prime}\right),
\]
and the camera token \(\hat{T}_t^c = T_t^{c(L)}\) is passed to a camera head:
\[
\hat{K}_t, \hat{T}_t = \text{Head}_{camera}\left(\hat{T}_t^c\right).
\]
The predicted global pointmap \(\hat{X}^{global}_t\) is then encoded into the PSM via \(E_s\) and appended to the FIFO queue, closing the online update loop. The memory therefore evolves as the video streams in, rather than being recomputed from scratch.

## 5. Training objective, implementation, and empirical performance

Mem4D is trained end-to-end on sequences of \(N\) images with a weighted sum of a pointmap reconstruction loss and camera losses [2508.07908]. The paper states that the 3D regression loss follows MASt3R’s confidence-aware formulation; the printed equation in the excerpt appears garbled in the PDF text, but the intended form is the standard confidence-weighted robust regression used in MASt3R. The variables identified in the text are the predicted 3D point \(\hat{\boldsymbol{x}}\), the ground-truth 3D point \(\boldsymbol{x}\), the predicted confidence \(c\), a scale normalization factor \(s\) from ground truth, and a weighting constant \(\alpha\).

The absolute pose and intrinsics loss is
\[
\mathcal{L}_{\text {abspose} }= \sum_{t=1}^N\left(\left\|\hat{\boldsymbol{q}}_t-\boldsymbol{q}_t\right\|_2+\left\|\frac{\hat{\boldsymbol{\tau}}_t}{s}-\frac{\boldsymbol{\tau}_t}{s}\right\|_2 + \beta\left\| K_t -\hat{K}_t \right\|\right),
\]
where \(\hat{\boldsymbol{q}}_t, \boldsymbol{q}_t\) are predicted and ground-truth quaternions, \(\hat{\boldsymbol{\tau}}_t, \boldsymbol{\tau}_t\) are predicted and ground-truth translations, and \(K_t, \hat{K}_t\) are ground-truth and predicted intrinsics. The relative pose loss is
\[
\mathcal{L}_{\text{relpose} }= \sum_{t=2}^{N} \left( \| \hat{\boldsymbol{q}}_{t,t-1} - \boldsymbol{q}_{t,t-1} \|_2 +\left\|\frac{\hat{\boldsymbol{\tau}}_{t,t-1}}{s}-\frac{\boldsymbol{\tau}_{t,t-1}}{s}\right\|_2 \right),
\]
with \(\hat{T}_{t,t-1} = \hat{T}_t \times \hat{T}_{t-1}^{-1}\). The total objective is
\[
\mathcal{L} = \lambda_1 \mathcal{L}_{\text{conf}} + \lambda_2 \mathcal{L}_{\text{abspose}} + \lambda_3 \mathcal{L}_{\text{relpose}}.
\]

The implementation is intentionally lightweight for online use. Optimization uses **AdamW** with learning rate \(1\times 10^{-5}\) and **linear warmup + cosine decay**. Training uses **8 NVIDIA A100 GPUs**, batch size **4 per GPU**, and a two-stage curriculum: first fixed-length **11-frame clips**, then longer clips by sampling **11 to 48 frames**. The paper states that this curriculum is important because the second stage teaches the model to handle long-range drift and extended video sequences.

Evaluation spans three tasks: **video depth estimation**, **camera pose estimation**, and **3D scene reconstruction**. The video depth benchmarks are **Sintel**, **BONN**, and **KITTI**, using **Abs Rel** and \(\delta < 1.25\), in both **Per-Scene alignment** and **Metric** settings. The camera pose benchmarks are **Stereo4D**, **Sintel**, and **TUM-dynamics**, using **ATE**, **RPE trans**, and **RPE rot**. The 3D reconstruction benchmarks are **7-Scenes** and **NRGBD**, using **Acc** and **Comp**.

The main reported results are as follows. In video depth estimation, Mem4D is strongest in the metric, no-alignment setting: **Sintel metric Abs Rel** is **0.846** for Mem4D versus **1.029** for CUT3R, and **BONN metric Abs Rel** is **0.086** versus **0.103**; the paper states that these correspond to improvements of **21.6%** on Sintel and **19.7%** on Bonn relative to CUT3R. In the per-scene aligned setting, Mem4D reaches **0.072 Abs Rel** on BONN, slightly better than CUT3R’s **0.078**, and remains competitive on KITTI. In camera pose estimation, representative ATE values are **0.495** on Stereo4D, **0.263** on Sintel, and **0.061** on TUM-dynamics. The paper emphasizes that Mem4D is fully online and does not use post-processing, unlike some higher-accuracy alignment-based baselines. In 3D reconstruction, reported means are **0.185 Acc** and **0.178 Comp** on 7-Scenes, and **0.271 Acc** and **0.212 Comp** on NRGBD. Efficiency-wise, the paper reports about **16 FPS** on Stereo4D at \(512 \times 384\), and elsewhere reports **16.12 FPS** on Stereo4D.

## 6. Ablations, qualitative behavior, limitations, and significance

The ablation study on Sintel isolates the role of each major component [2508.07908]. Removing second-stage long-sequence training hurts performance, indicating the importance of long-clip training for controlling drift. Removing relative pose loss degrades pose estimation, showing that temporal consistency supervision matters. Removing the TDM causes the most severe depth drop, confirming that motion-aware detail is essential for dynamic scenes. Removing the PSM significantly hurts nearly all metrics, validating it as the long-term geometric anchor. Removing the TCA sharply degrades pose accuracy, showing that local temporal context is important for disambiguating motion. The full model performs best across the reported metrics.

The qualitative findings align with the paper’s central claim. In the introductory illustration, unified memory causes **wall drift** and **motion blur** on a dancer, whereas Mem4D’s decoupled design produces **stable walls** and **sharp moving humans**. On Bonn and DAVIS, Mem4D is reported to outperform MonST3R and CUT3R visually, with better geometry on background structures and fewer artifacts on moving objects. The paper emphasizes that the PSM suppresses cumulative drift over time, while the TDM prevents moving regions from being over-smoothed.

Two limitations are explicitly noted. First, as a feed-forward online method, Mem4D can still accumulate **drift over extremely long sequences**. Second, the method is **supervised** and depends on dense 4D ground truth, which is scarce and hard to obtain; this limits scalability and generalization. The paper suggests that integrating **Bundle Adjustment** could further improve long-term accuracy, and that **self-supervised training** would be an important future direction.

Within the landscape of dynamic scene reconstruction, Mem4D’s significance lies in its conceptual reframing of memory usage. The paper’s contribution is not only a stronger memory module, but the claim that static and dynamic content have different temporal behaviors and should not compete for the same memory budget. By splitting the problem into a transient motion memory and a persistent structural memory, and by alternating queries between them during decoding, Mem4D resolves the stability-versus-fidelity trade-off that hinders prior unified-memory systems.

Source: https://www.emergentmind.com/topics/mem4d