Multi-Scale Temporal Hashing (MSTH)

• Multi-Scale Temporal Hashing (MSTH) is a method that encodes temporal structures in sequential data using hierarchical and mask-guided hashing.
• It distinguishes between proximal codes for fine-grained, short-term details and distal codes for long-horizon, global anchoring, enhancing planning and reactivity.
• MSTH improves computational efficiency and training speed in applications like goal-conditioned robotic control and dynamic scene reconstruction by reducing memory usage and hash collisions.

Multi-Scale Temporal Hashing (MSTH) refers to a suite of architectural and algorithmic mechanisms that encode temporal structure in high-dimensional sequential data through hierarchical or mask-guided hashing across multiple temporal or spatio-temporal scales. Distinct instantiations of MSTH have emerged independently in visual goal-conditioned robotic control (Zhou et al., 29 Dec 2025) and in dynamic scene reconstruction (Wang et al., 2023), each leveraging multi-scale hashing for tractable representation and efficient learning over time-varying signals. In both domains, MSTH exploits structured temporal or spatio-temporal sparsity to focus compute and memory on salient, task-relevant features, balancing fine-grained local detail with coarse global anchoring.

1. Key Objectives and Motivations

MSTH arises from the limitations of existing temporally uniform architectures in high-dimensional sequential prediction and learning. In goal-conditioned policy learning (Zhou et al., 29 Dec 2025), flat rollouts dilute the agent’s ability to simultaneously achieve long-horizon planning (anchoring towards distant goals) and rapid, closed-loop reactivity (adjusting for disturbances), as all future states are represented at uniform time intervals. In dynamic scene reconstruction (Wang et al., 2023), uniform spatio-temporal encodings produce redundancy and excessive collisions when representing primarily static regions, inhibiting memory and training efficiency.

MSTH addresses these deficiencies by decomposing trajectories or fields into multi-scale temporal (or spatio-temporal) summaries. In goal-conditioned control, MSTH selects a dense array of short-horizon (proximal) states to drive feedback, and sparse, long-horizon (distal) anchors for global consistency. For dynamic NeRF-style reconstructions, MSTH allocates spatial and temporal encoding capacity only where dynamic content merits, using learned masks to separate static and dynamic support.

2. Mathematical Formulation and Hashing Procedures

2.1 Proximal/Distal Temporal Hashing in Goal-Conditioned Agents

Let $t$ denote the current time, $K$ the total planned horizon, $P\leq K$ the proximal horizon, $r$ the stride for proximal sampling, and $M$ the number of distal bins. Proximal indices are defined as: $$S_\mathrm{prox} = \{\, s_{t + k r}\,\mid\, k = 1,\ldots,\lfloor P/r\rfloor\,\}$$ Distal indices utilize logarithmic spacing: $$d_m = P + \left\lfloor \frac{K-P}{\log(M+1)} \cdot \log(m+1) \right\rfloor, \quad m=1,\ldots,M$$

$$S_\mathrm{dist} = \{\, s_{t + d_m}\,\mid\, m = 1,\ldots,M\,\}$$

Each selected frame $s$ is mapped by a vision encoder $\phi_v(\cdot)$ into a latent feature, then hashed by MLPs $H^\mathrm{prox},H^\mathrm{dist}$: $$h_k^\mathrm{prox} = H^\mathrm{prox}\big(\phi_v(s_{t + k r})\big)$$

$$h_m^\mathrm{dist} = H^\mathrm{dist}\big(\phi_v(s_{t + d_m})\big)$$

The set of multi-scale codes is $$C_w = \{h_1^\mathrm{prox},...,h_{\lfloor P/r\rfloor}^\mathrm{prox}\}\cup\{h_1^\mathrm{dist},...,h_M^\mathrm{dist}\}$$.

2.2 Masked Spatio-Temporal Hashing in Dynamic Scene Reconstruction

Given $\mathbf{x}\in\mathbb{R}^3$, $t\in[1,T]$, the encoded feature is formed by a mixture of 3D and 4D hashed encodings gated by a sigmoid mask: $$\mathrm{enc}(\mathbf{x},t) = m(\mathbf{x})\,H_{3\mathrm{D}}(\mathbf{x}) + (1-m(\mathbf{x}))\,H_{4\mathrm{D}}(\mathbf{x},t)$$ where $H_{3\mathrm{D}}$ and $H_{4\mathrm{D}}$ are feature vectors from multi-resolution spatial and space-time hash tables, and $m(\mathbf{x})=\sigma(\tilde m(\mathbf{x}))$ is a learned mask voxel grid.

Multi-resolution hashing is performed using per-level hash functions: $$\texttt{hash}_{\mathrm{3D}}(\mathbf{i}) = (i_x p_0 \oplus i_y p_1 \oplus i_z p_2)\bmod T$$

$$\texttt{hash}_{\mathrm{4D}}(\mathbf{i},t) = (i_x p_0 \oplus i_y p_1 \oplus i_z p_2 \oplus t p_3)\bmod T$$

Each code is interpolated over neighboring grid entries to yield per-level features, stacked to form $H_{3\mathrm{D}}$ or $H_{4\mathrm{D}}$.

Temporal pyramid resolutions $T_\ell$ for 4D hashing grow geometrically: $$T_\ell = \mathrm{round}(T_{\min} \cdot \alpha^{\ell-1})$$.

3. Integration in Architectural Pipelines

In robotic control (Zhou et al., 29 Dec 2025), MSTH is situated within a three-module policy:

• A Goal-Conditioned World Model (GCWM) predicts future latent frames $z_{t+1},\ldots,z_{t+K}$.
• MSTH selects and hashes subsets into $C_w$.
• An Action Expert policy applies cross-attention, with $C_w$ as keys/values and proprioception as queries. Proximal frames steer short-horizon execution; distal frames guide global structure. The system achieves joint training through flow-matching losses on both vision and action sequences.

For dynamic NeRFs (Wang et al., 2023), MSTH organizes scene encoding:

• Spatially static regions ($m(\mathbf{x})\approx1$) use only 3D hashes, saving memory and avoiding collision in the expensive 4D tables.
• Dynamically changing points ($m(\mathbf{x})\approx0$) use 4D spatio-temporal hashes.
• The mask $m(\mathbf{x})$ is optimized via uncertainty-guided objectives and regularization, with auxiliary sub-branches predicting per-voxel color variance.

4. Learning, Losses, and Optimization

In goal-conditioned control (Zhou et al., 29 Dec 2025), MSTH parameters participate in end-to-end training, backpropagating gradients from action losses via cross-attention layers: $$L_v = \mathbb{E}_{t, z_0, z_1, z_t, z_g}\left[\left\| v_\theta - (z_1 - z_0) \right\|^2\right]$$

$$L_a = \mathbb{E}_{t, a_0, a_1, c_w, c_p}\left[\left\| u_\phi - (a_1 - a_0) \right\|^2\right]$$

$L_\mathrm{stage1} = L_v + \lambda L_a$

Gradients from $L_a$ flow into the hashing projections, aligning temporal code selection with optimal control.

In scene reconstruction (Wang et al., 2023), the total loss comprises photometric, aleatoric uncertainty, and mutual information terms: $$L = L_r + \lambda \mathbb{E}[L_u] - \gamma \hat{I}_\Theta(m, u)$$ The uncertainty-driven auxiliary subnetwork encourages $m(\mathbf{x})$ to focus dynamic content where color variance is high, while mask sparsity and mutual information regularization ensure crisp separation of static/dynamic content and low hash collision rates.

5. Computational Efficiency and Complexity

MSTH reduces both computational and memory overhead relative to flat-uniform encoding. In goal-conditioned policies, the cross-attention complexity drops from $O(K^2)$ (for $K$ full steps) to $O((P/r + M)^2)$, where typically $P/r + M \ll K$ (e.g., 11 hash codes instead of 54, yielding $\approx200$ ms inference latency for 50 actions on a single GPU (Zhou et al., 29 Dec 2025)). In masked hash encoding, exclusion of static points from 4D tables reduces memory to $\sim130$ MB and drastically limits hash collisions (Wang et al., 2023). Gradient-based occupancy masks additionally concentrate training on complex, dynamic scene content, speeding convergence and improving stability.

6. Empirical Results and Comparative Evaluation

6.1 Goal-Conditioned Long-Horizon Control

On challenging robotic writing tasks, MSTH delivers large performance improvements, especially outside the training distribution. For medium and long words, out-of-distribution (OOD) success rates increase from $0.20$ $\to$ $0.90$ (medium) and $0.00$ $\to$ $0.88$ (long) with MSTH, while uniform-sequence policies collapse (see Table 6 (Zhou et al., 29 Dec 2025)). Qualitative rollouts reveal that proximal frames encode fine-grained trajectories (e.g., pen strokes), whereas distal frames fix global structure (overall letter shapes).

Word Length	w/o MSTH (OOD)	w/ MSTH (OOD)
Short (≤3)	0.60	0.93
Medium (4–6)	0.20	0.90
Long (≥7)	0.00	0.88

6.2 Dynamic Scene Reconstruction

On benchmarks including Plenoptic Video and Google Immersive Video, masked MSTH outperforms prior fast dynamic NeRFs:

• PSNR gain of $+1.7$ dB over HexPlane; $+0.7$–$1.4$ dB over other baselines.
• LPIPS reduced by $30\%$.
• $20$ minutes training (vs. $2$–$12$ hours for baselines); model size $135$ MB (vs. $200$–$500$ MB) (Wang et al., 2023).

Ablations demonstrate that omitting either the mask or the multi-scale temporal hierarchy degrades both quality and compression. Removing mutual information penalties yields suboptimal mask values and increased table collisions.

7. Limitations and Extensions

MSTH, while providing substantial practical gains, exhibits certain domain-specific limitations. In monocular dynamic NeRF inference, underobserved regions may result in “ghosting” or “flickering” artifacts; extreme nonrigid (e.g., fluids) deformations may undercut a static/dynamic spatial split (Wang et al., 2023). Generalization to topology-altering dynamics may require explicit deformation fields. In robotic control, the two-level (proximal/distal) hierarchy may need further refinement for tasks with multiplanar or multi-goal structure (Zhou et al., 29 Dec 2025). Noted extensions include incorporation of learned deformation priors, real-time rendering primitives (e.g., grid splatting), or cross-scene meta-learning of mask distributions.

---

MSTH represents a modular class of strategies for temporally-aware, memory-efficient representation in high-dimensional, nonstationary domains. By hierarchically decomposing temporal or spatio-temporal support, it demonstrably advances both long-horizon policy learning and dynamic scene reconstruction (Zhou et al., 29 Dec 2025, Wang et al., 2023).