---
title: Dual Implicit Neural Memory
url: https://www.emergentmind.com/topics/dual-implicit-neural-memory
type: topic
---

# Dual Implicit Neural Memory

Dual implicit neural memory denotes a class of architectures in which two complementary memory pathways are maintained as neural representations rather than as explicit raw histories or purely symbolic stores. In the cited literature, the paired memories separate visual-semantic from spatial-geometric information in Vision-and-Language Navigation, object semantics from occupancy and exploration in embodied navigation, positional embeddings from local k-space feature representations in cardiac cine MRI, and physically meaningful signal from effective background contribution in source separation [2509.22548] [2210.05129] [2508.12147] [2507.05249]. The recurring design principle is to decouple heterogeneous structure into two learnable stores, update them incrementally, and fuse them for downstream inference.

## 1. Genealogy of the concept

A plausible lineage of dual implicit neural memory combines two earlier strands of memory-augmented modeling. One strand emphasizes *dual memory organization*. The "Dual Memory Neural Computer for Asynchronous Two-view Sequential Learning" uses two encoders and two external memories to model two asynchronous sequential views, with late-fusion and early-fusion modes for inter-view interaction [1802.00662]. In late-fusion, the memories are separated and contain only view-specific contents; in early-fusion, the memories share the same addressing space, allowing cross-memory accessing. This establishes the idea that heterogeneous streams may require distinct but coordinated memory substrates.

A second strand emphasizes *implicit internal memory*. "Multigrid Neural Memory" introduces memory that is internal, distributed, co-located alongside computation, and implicitly addressed, rather than handled by external memory banks with custom controllers [1906.05948]. Its multigrid topology and distributed memory cells show that long-term, large-scale memory can emerge from network organization itself. This suggests that later dual-memory designs need not rely on explicit external matrices; the two memories may instead be encoded in key-value caches, recurrent states, or the parameters of neural implicit representations.

The resulting concept is therefore not restricted to a single mechanism. In some works, the duality is between two external memories; in others, it is between two internal neural representations that serve as memory. The commonality is structural separation plus downstream integration.

## 2. Decoupling principles and representational roles

The clearest modern formulation appears in "JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation" [2509.22548]. JanusVLN proposes a framework featuring a dual implicit neural memory that models spatial-geometric and visual-semantic memory as separate, compact, and fixed-size neural representations. The visual-semantic encoder is MLLM-based, using Qwen2.5-VL, while the spatial-geometric encoder uses VGGT pretrained on pixel-to-3D point cloud pairs. The decoupling is motivated by the observation that explicit semantic memory, such as textual cognitive maps or stored historical frames, suffers from spatial information loss, computational redundancy, and memory bloat.

An analogous decomposition appears in "Multi-Object Navigation with dynamically learned neural implicit representations" [2210.05129]. There, dual implicit memory is implemented through two dynamically learned neural implicit representations: a Semantic Finder, which predicts the position of a previously seen queried object, and an Occupancy and Exploration Implicit Representation, which encapsulates information about explored area and obstacles. The first acts as semantic memory; the second acts as spatial or episodic memory. Both are learned online during each episode, and their weights encode the memory.

Outside navigation, dual decomposition is applied to other forms of heterogeneity. "KP-INR: A Dual-Branch Implicit Neural Representation Model for Cardiac Cine MRI Reconstruction" uses one branch for positional embedding of k-space coordinates and another for local multi-scale k-space feature representations at those coordinates [2508.12147]. "Physics-Guided Dual Implicit Neural Representations for Source Separation" jointly trains one neural network for approximating distortions of the physical signal of interest and another for learning the effective background contribution [2507.05249]. In each case, the two memories correspond to different explanatory factors rather than duplicated capacity.

## 3. Memory substrates and update dynamics

The term *implicit* does not refer to a single storage substrate. In JanusVLN, implicit memory refers to the key/value caches generated internally by the transformer and attention blocks of each encoder [2509.22548]. These caches are organized with a hybrid strategy:
$$
\text{Memory} = \{M_\text{initial}, M_\text{sliding}\}
$$
where the initial window is always retained and the sliding window stores the most recent frames in FIFO fashion. The spatial-geometric encoder then conditions current processing on the retained cache:
$$
G_t = \mathrm{Decoder}(\mathrm{CrossAttn}(\mathrm{Encoder}(x_t), \{M_\text{initial}, M_\text{sliding}\}))
$$
An analogous approach applies to the semantic memory. By retaining only the KVs of tokens in the initial and sliding window, redundant computation is avoided and incremental updates become efficient.

In the MultiON framework, memory is stored in the parameters of online-learned neural implicit representations rather than in a separate cache [2210.05129]. The Semantic Finder is a 3-layer MLP with hidden layers of 512 units, ReLU activation, and sigmoid output; the Occupancy and Exploration representation is a 3-layer MLP with hidden layers of 512 units, ReLU activation, and softmax output over three classes. At each time step, object positions and occupancy labels are added to replay buffers, and both implicit memories are updated by SGD within the episode. The memory is therefore the evolving parameter state $\theta_s$ and $\theta_o$.

In Multigrid Neural Memory, the substrate is again different: memory cells are distributed throughout a multigrid-topology network and co-located with computation [1906.05948]. Read and write are not handled by explicit heads; instead, local computations update local pieces of memory based on neighboring spatial and scale positions. The paper describes this as internal attention performed implicitly, with coherent memory subsystems emerging across layers and resolutions.

## 4. Readout, fusion, and interaction mechanisms

Dual implicit memory systems differ not only in what they store, but also in when and how the two memories interact. The earlier DMNC formalized this design choice as late-fusion versus early-fusion [1802.00662]. In late-fusion, inter-view interaction occurs only at decoding; in early-fusion, cross-memory accessing allows inter-view interaction during encoding. This distinction remains useful for later dual implicit systems.

JanusVLN uses a direct feature-fusion rule after aligning the outputs of its semantic and spatial encoders:
$$
F_t = S'_t + \lambda \cdot \mathrm{MLP}(G'_t)
$$
The fused feature $F_t$, together with the instruction embedding, is passed to the MLLM backbone for action prediction [2509.22548]. This design keeps the two memory channels distinct during update while allowing their contributions to be combined at the decision stage.

The MultiON architecture uses two different read mechanisms. The semantic memory is queried directly with the current goal, while the occupancy memory is summarized by a global read operation:
$$
\mathbf{e}_t = r(\theta_{o,t}; \theta_r)
$$
where $r$ is a Transformer-based model that maps the weights of the occupancy representation into a usable embedding space [2210.05129]. The policy then aggregates semantic readout, uncertainty, the occupancy embedding, visual features, and action or goal embeddings through a GRU.

KP-INR implements interaction as bidirectional information exchange between its two branches [2508.12147]. The reported mechanism has three parts: linear cross-input, intermediate fusion through an MLP on concatenated features, and concatenation of fused features back into each branch before final prediction. At inference, the full Cartesian grid is queried and the outputs are averaged:
$$
\bm{y}_{rec} = \frac{1}{2}(\bm{y}_{kv} + \bm{y}_{pv})
$$
followed by hard data consistency and image reconstruction.

In the physics-guided source-separation framework, the fusion operation is additive at the level of the reconstructed signal [2507.05249]. One branch models the signal of interest, possibly via a physical forward model convolved with a learned kernel, and the other models the background. The final prediction is the sum of both pathways, trained jointly by a reconstruction-based loss with regularization on the background component.

## 5. Empirical realizations across domains

The empirical literature spans embodied navigation, sequential learning, reconstruction, source separation, and generic memory benchmarks. The following table summarizes representative reported results.

| System | Domain | Reported result |
|---|---|---|
| JanusVLN | VLN-CE, R2R-CE unseen | SR 60.5 |
| MultiON dual implicit agent | Test split | NoMap 42.3% success, 28.1% SPL; +Semantic 47.0%, 34.5%; +Occupancy 46.7%, 35.1% |
| KP-INR | Uniform 4x sampling | PSNR 41.99, SSIM 0.9748, DISTS 0.9430 |
| Multigrid Neural Memory | bAbI tasks | Mean Error 9.2 ± 1.0; #Failed Tasks 5.4 ± 1.4; Memory Size 2.86K |

In VLN, JanusVLN reports state-of-the-art performance without depth, panorama, or odometry sensors, using only RGB input [2509.22548]. The success rate improves by 10.5-35.5 compared to methods using multiple data types as input and by 3.6-10.8 compared to methods using more RGB training data. The ablations are also diagnostic: removing spatial memory causes SPL to drop from 49.2 to 40.9; removing semantic memory causes SR to drop by 13.8%; removing both yields SR 24.8. These results support the claim that the two memories are complementary rather than redundant.

In embodied multi-object navigation, the dual implicit approach improves over vector-memory and single-memory variants [2210.05129]. Table 1 reports NoMap at 42.3% Success and 28.1% SPL on the test split, with improvements to 47.0% Success and 34.5% SPL when adding the Semantic Finder, and 46.7% Success and 35.1% SPL when adding the Occupancy representation. The paper also reports that including uncertainty estimates in the semantic memory boosts success and SPL by up to 8 percentage points.

In cardiac cine MRI, KP-INR reports improved performance over baseline models on the CMRxRecon2024 dataset [2508.12147]. Under uniform 4x sampling, KP-INR achieves PSNR 41.99, SSIM 0.9748, and DISTS 0.9430; under uniform 8x, PSNR 37.85, SSIM 0.9502, and DISTS 0.9133. The paper states that KP-INR statistically significantly outperforms all baselines, especially at higher acceleration and under more challenging sampling schemes.

Multigrid Neural Memory demonstrates that implicitly addressed internal memory can serve as a generic substrate rather than a domain-specific map [1906.05948]. On exploration and mapping tasks, it retains long-term memory for trajectories of thousands of time steps. On bAbI, it reports mean error 9.2 ± 1.0 and 5.4 ± 1.4 failed tasks, compared with 16.7 ± 7.6 and 11.2 ± 5.4 for DNC, despite a smaller memory size of 2.86K versus 16.38K.

In source separation, the physics-guided dual INR framework is reported to successfully separate physically meaningful signals from a complex or structured background even when the signal characteristics vary across all four dimensions of the parameter space [2507.05249]. The method is self-supervised, requires no labeled data or predefined dictionaries, and reports a 792:1 reduction in one example. This indicates that dual implicit representations are applicable even when the objective is decomposition rather than navigation or sequence modeling.

## 6. Conceptual issues, misconceptions, and related directions

The heterogeneity of current usage suggests that dual implicit neural memory is better understood as a design pattern than as a fixed architecture. One common misconception is to equate *implicit* with one specific storage mechanism. In fact, the surveyed works use internal key/value caches, online-updated neural fields, distributed recurrent memory cells, and activation-space steering vectors as their memory substrate [2509.22548] [2210.05129] [1906.05948] [2606.29824]. What unifies them is that memory is represented by neural state, parameters, or activations, rather than by a hand-maintained symbolic history alone.

A second misconception is to equate *dual memory* with a simple duplication of capacity. The duality is typically functional. JanusVLN separates semantics from spatiality; the MultiON agent separates queried object memory from occupancy and exploration; physics-guided dual INRs separate signal from background; KP-INR separates positional from local feature representations. This suggests that the key benefit is factorization of heterogeneous structure, not mere parameter scaling.

Related work in lifelong learning and LLM agents broadens the surrounding conceptual space. DUCA introduces implicit and explicit knowledge representation dichotomy, an inductive bias learner, and a semantic memory submodule within a multi-memory system for lifelong learning [2310.11341]. Neural Procedural Memory represents agent memory through implicit activation steering rather than explicit instructions and reports that combining implicit steering with explicit workflows provides complementary advantages [2606.29824]. DCPM reorganizes agent memory along a cognitive capability hierarchy with a synchronous daytime writer and an asynchronous nighttime engine; enabling System2 contributes most where the benchmark rewards implicit cross-session inference, with gains of up to +5.20 on PersonaMem-v2, and least on span recall [2606.09483]. These systems are not identical to dual implicit neural memory as defined in embodied perception or reconstruction, but they indicate that the same underlying question recurs across fields: how to maintain multiple interacting memory traces inside neural computation without reducing memory to flat retrieval.

The broader significance of dual implicit neural memory therefore lies in its treatment of memory as structured internal state. Across navigation, multi-view sequence learning, reconstruction, source separation, and agent systems, the literature converges on the view that separate but interacting neural memories can improve retention of heterogeneous information, reduce redundancy, and support richer downstream reasoning [1802.00662] [2509.22548].

Source: https://www.emergentmind.com/topics/dual-implicit-neural-memory