Modular Neural Computer (MNC)

• Modular Neural Computer (MNC) is a neural architecture that separates computation into distinct modules coordinated by explicit controllers and external memory.
• It employs conditional routing, one-hot module gating, and both learned and analytically specified operations to execute precise algorithms.
• MNCs have been applied in tasks like language modeling, sorting, and planning, demonstrating efficient specialization and reconfigurability.

Searching arXiv for the cited MNC-related papers to ground the article in published sources. Modular Neural Computer (MNC) denotes a class of modular neural architectures in which computation is decomposed into reusable functional components coordinated by an explicit control mechanism, typically with input-dependent routing and, in several formulations, an external memory. Across the literature, the term covers distinct but related designs: modular layers with learned conditional computation (Kirsch et al., 2018), externally stored and composable neural programs (Le et al., 2020), hierarchical assemblies of independently trained planning modules mediated by hyperdimensional computing (McDonald, 2024), decompositions into control and primitive-operation modules for algorithmic tasks (Castillo-Bolado et al., 2019), and a 2026 memory-augmented architecture that compiles exact algorithms into analytically specified neural components with one-hot module gates and scalar external memory (Leon, 4 Mar 2026). The unifying idea is that neural computation is not monolithic: control flow, memory access, and functional transformations are separated into modules with fixed interfaces, so that only a subset of capacity is active at a given step or for a given input.

1. Conceptual scope and defining characteristics

The 2018 modular networks framework organizes a network into $L$ modular layers; at each layer $l$, a controller $g^{(l)}(\cdot;\phi^{(l)})$ selects $K$ out of $M$ parameterized modules $\{f_1,\dots,f_M\}$ based on the current layer input $x^{(l)}$, and the selected outputs are combined by summation or concatenation (Kirsch et al., 2018). In that formulation, the controller produces a discrete selection $a^{(l)} \in \{1,\dots,M\}^K$ sampled from $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$, while each selected module processes the same layer input.

A related but more explicitly programmatic interpretation appears in Neurocoder, where a Program Controller queries an external Program Memory composed of three banks $M^U \in \mathbb{R}^{P \times d}$, $l$0, and $l$1, storing singular vectors and singular values that are recomposed into an active program $l$2 and then integrated into a Main Network through $l$3 (Le et al., 2020). Here, the “program” itself is a datum in memory rather than a fixed parameter tensor.

The 2026 architecture titled “Modular Neural Computer” defines MNC more narrowly as a feed-forward, memory-augmented neural machine with four principal components: an external associative memory of scalar cells, explicit read and write heads with continuous addressing, a controller MLP that emits a one-hot module gate together with read and write addresses, and a homogeneous set of functional MLP modules, exactly one of which is active per step (Leon, 4 Mar 2026). In that design, computation unfolds as a sequence of memory transformations generated by a fixed graph.

The modular list-sorting architecture of 2019 adopts a decomposition into a control module and primitive functional modules $l$4, with a one-hot operator selection vector $l$5 choosing exactly one operator at each step and updating the environment representation by

$l$6

Each operator is decomposed into a selective-input mask, a functional kernel, and a selective-update mask (Castillo-Bolado et al., 2019).

The hierarchical CML-HDC system of 2024 instantiates the same broad MNC idea as a three-tier hierarchy consisting of a hyperdimensional “policy” vector, an object-level Cognitive Map Learner, and a grid-level CML, with all communication mediated through a shared hypervector space (McDonald, 2024). This suggests that, in the literature, MNC is better understood as an architectural family than as a single canonical design.

2. Architectural patterns: modules, controllers, and memory

A recurring structural feature is separation between control and functional execution. In modular networks, the controller receives $l$7 and produces either logits for each module or for each $l$8-tuple; for $l$9, a simple controller computes

$g^{(l)}(\cdot;\phi^{(l)})$0

The chosen modules then process the input, and the layer output is

$g^{(l)}(\cdot;\phi^{(l)})$1

or a concatenation of those outputs (Kirsch et al., 2018).

In the 2019 modular list-sorting system, the control path itself is split into a Digestor and a Controller. The Digestor is an LSTM of 100 units that reads $g^{(l)}(\cdot;\phi^{(l)})$2 row by row and produces a fixed-length embedding $g^{(l)}(\cdot;\phi^{(l)})$3; the Controller is an LSTM of 100 units that consumes $g^{(l)}(\cdot;\phi^{(l)})$4, maintains recurrent state across time, and outputs the selection distribution

$g^{(l)}(\cdot;\phi^{(l)})$5

The functional modules then execute primitive operations such as mova, movb, retb, and swap (Castillo-Bolado et al., 2019).

Memory becomes explicit in Neurocoder and in the 2026 MNC. Neurocoder stores modular programs externally as singular components and reconstructs the active weight matrix through low-rank composition: $g^{(l)}(\cdot;\phi^{(l)})$6 Addressing is content-and-usage-based, with controller-emitted queries and interpolation gates mixing content lookup with “least-used attention” (Le et al., 2020).

The 2026 MNC uses scalar-cell external memory over address space $g^{(l)}(\cdot;\phi^{(l)})$7, with memory at step $g^{(l)}(\cdot;\phi^{(l)})$8 represented as $g^{(l)}(\cdot;\phi^{(l)})$9. The controller reads a small control vector $K$0 from memory and emits a one-hot gate $K$1, read addresses $K$2, and write addresses $K$3. Read heads use temperature-scaled softmax attention, and write heads overwrite attended cells. Module outputs are merged as

$K$4

with the invariant $K$5 ensuring that inactive modules are inhibited (Leon, 4 Mar 2026).

The 2024 CML-HDC system introduces a different memory mechanism: a learned “map” hypervector binds each object-state to its current grid position and serves as short-term memory linking abstract subgoals to concrete positions. Because all node-state vectors lie in the same $K$6-dimensional hypervector space, outputs can be bound, bundled, permuted, and unpermuted by HDC operators (McDonald, 2024).

3. Routing, addressing, and composition mechanisms

Conditional routing is central to the modular-network lineage. The 2018 framework selects a discrete module composition per sample and per layer, with only the selected modules receiving gradients in a mini-batch. For $K$7, the joint over selections is written as

$K$8

and per-sample compute depends on $K$9, not $M$0 (Kirsch et al., 2018).

Neurocoder replaces discrete module IDs with differentiable retrieval of singular components. For each head $M$1 at recurrence step $M$2, the controller emits query vectors $M$3 and interpolation gates $M$4. Content weights are computed via cosine similarity to keys produced by $M$5, usage-based weights are derived from slot-usage statistics, and the final attention is

$M$6

Recovered singular vectors and values are then assembled into the active low-rank program (Le et al., 2020).

The 2026 MNC formalizes memory access through continuous addresses. A scalar address $M$7 is mapped into key space by linear interpolation: $M$8 Read attention is then

$M$9

with $\{f_1,\dots,f_M\}$0 in practice, so the access behaves almost exactly like hard discrete addressing (Leon, 4 Mar 2026).

The CML-HDC system composes plans through symbolic operations on hypervectors. Policy sequences are encoded by permutation and bundling, for example

$\{f_1,\dots,f_M\}$1

while object-position associations are stored as

$\{f_1,\dots,f_M\}$2

The next target position is then recovered by unbinding and cleanup: $\{f_1,\dots,f_M\}$3 Routing here is therefore algebraic rather than softmax-based, but still modular and input-dependent (McDonald, 2024).

A common misconception is that modularity necessarily implies sparse gating over interchangeable subnetworks. The literature is broader: it includes discrete controller-selected operators (Castillo-Bolado et al., 2019), soft retrieval of SVD components (Le et al., 2020), hypervector-mediated composition across independently trained planners (McDonald, 2024), and one-hot gating over analytically specified functional MLPs with explicit memory heads (Leon, 4 Mar 2026).

4. Training regimes and the question of exactness

The 2018 modular networks paper trains both decomposition and modules end-to-end under a maximum-likelihood objective without additional regularizers: $\{f_1,\dots,f_M\}$4 Optimization uses a variational Viterbi-EM lower bound with $\{f_1,\dots,f_M\}$5, a partial E-step that samples candidate module compositions, and a partial M-step that performs gradient-ascent updates on $\{f_1,\dots,f_M\}$6 and $\{f_1,\dots,f_M\}$7 using only the chosen modules (Kirsch et al., 2018).

Neurocoder is trained end-to-end by minimizing

$\{f_1,\dots,f_M\}$8

where $\{f_1,\dots,f_M\}$9 is a supervised loss or an RL loss such as A3C, and

$x^{(l)}$0

encourages orthonormal singular-vector banks (Le et al., 2020).

The 2019 list-sorting architecture uses supervised cross-entropy from expert Selection Sort traces. Pointer modules use binary cross-entropy, the swap module uses categorical cross-entropy over 11 classes, and the controller uses cross-entropy on the selection vector against the expert operator index. Modular training proceeds module-wise until quantitative and qualitative saturation criteria fall below 1%, whereas the monolithic baseline is trained end-to-end with teacher forcing on the controller selection (Castillo-Bolado et al., 2019).

By contrast, the 2026 MNC is explicitly not trained for a task. It “realizes a given algorithm through analytically specified neural components with fixed interfaces and exact behavior,” and exactness and determinism are guaranteed by design rather than by learning (Leon, 4 Mar 2026). Known symbolic algorithms are compiled into a fixed memory layout, a controller MLP that analytically implements phase-selection logic, and a small set of MLP modules with exact weights on the finite set of used states.

This creates an important conceptual split within the MNC literature. Some systems are learned conditional-computation architectures whose modular decomposition emerges or is optimized from data (Kirsch et al., 2018, Le et al., 2020, Castillo-Bolado et al., 2019). Others are reconfigurable assemblies of independently trained components manipulated algebraically (McDonald, 2024). The 2026 MNC occupies a different point in the design space: it preserves neural substrates such as MLPs and soft attention but substitutes analytic correctness for statistical learning (Leon, 4 Mar 2026).

5. Empirical behavior and representative use cases

The 2018 modular networks paper reports language modeling on Penn Treebank with a modular GRU in which, at each time step, $x^{(l)}$1 modules update the candidate state. With $x^{(l)}$2 and $x^{(l)}$3, the EM Modular model achieved test perplexity $x^{(l)}$4 after $x^{(l)}$5 steps, compared with $x^{(l)}$6 for a baseline non-modular model, $x^{(l)}$7 for REINFORCE, and $x^{(l)}$8 for Noisy-top-$x^{(l)}$9 (Kirsch et al., 2018). On CIFAR-10, a network with five modular layers trained faster and reached higher training accuracy than a non-modular baseline with the same number of parameters; with a convolutional controller it also generalized better.

Neurocoder is evaluated on instance-based object recognition, sequential learning, Atari reinforcement learning, and continual learning. The reported quantitative gains include approximately $a^{(l)} \in \{1,\dots,M\}^K$0–$a^{(l)} \in \{1,\dots,M\}^K$1 lower error than baseline on MNIST with a linear classifier; up to $a^{(l)} \in \{1,\dots,M\}^K$2 higher accuracy than both “Mixture of Experts” and “Neural Stored-program Memory” baselines on CIFAR-10/100 with LeNet/DenseNet; convergence in approximately $a^{(l)} \in \{1,\dots,M\}^K$3 iterations versus $a^{(l)} \in \{1,\dots,M\}^K$4–$a^{(l)} \in \{1,\dots,M\}^K$5 for HyperNet or plain GRU in polynomial auto-regression; scores of $a^{(l)} \in \{1,\dots,M\}^K$6–$a^{(l)} \in \{1,\dots,M\}^K$7 points on hard Atari games where vanilla and HyperNet-A3C score approximately $a^{(l)} \in \{1,\dots,M\}^K$8; and improved retention in split MNIST, split CIFAR10, and split CIFAR100 continual-learning benchmarks (Le et al., 2020).

The 2019 sorting study compares modular and monolithic networks on list sorting for sequence lengths up to $a^{(l)} \in \{1,\dots,M\}^K$9. Modular module-wise training required approximately $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$0 hour wall-clock for $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$1, while the monolithic system required approximately $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$2 hours; the modular approach showed steadier learning curves, smaller and more stable gradient magnitudes, and faster convergence of individual modules. However, the monolithic model generalized better to longer lists, while the modular model showed a sharp drop after $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$3 for learning rate $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$4, with some improvement at $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$5 (Castillo-Bolado et al., 2019).

The 2024 CML-HDC system targets sequential goal completion in mazes. With hypervector dimension $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$6, full CML-HDC on $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$7 different mazes and policy “home→key→treasure→home” achieved success rate $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$8 and mean path-length overhead within $p(a^{(l)} \mid x^{(l)}, \phi^{(l)})$9 of the true shortest path; a door-removal reconfiguration test also succeeded in $M^U \in \mathbb{R}^{P \times d}$0 trials (McDonald, 2024). At the same time, only approximately $M^U \in \mathbb{R}^{P \times d}$1 of random object layouts yielded unambiguous cleanup, and grid-only CML success from key→treasure over $M^U \in \mathbb{R}^{P \times d}$2 mazes was $M^U \in \mathbb{R}^{P \times d}$3, with dithering along a wall identified as a common failure.

The 2026 MNC uses three case studies to demonstrate exact algorithmic execution on variable-length inputs: computing the minimum of an array by linear scan, sorting an array in place via adjacent-swap bubble passes, and executing A* search on a fixed problem instance with a reserved memory region for the graph, costs, heuristics, and search-node table (Leon, 4 Mar 2026). The paper presents these as constructive examples rather than benchmark competitions.

6. Specialization, interpretability, and modular reconfiguration

A major claim in the modular-network literature is that useful specialization can emerge without auxiliary diversity regularization. The 2018 work measures specialization at layer $M^U \in \mathbb{R}^{P \times d}$4 using sample-wise entropy

$M^U \in \mathbb{R}^{P \times d}$5

and batch entropy

$M^U \in \mathbb{R}^{P \times d}$6

In practice, $M^U \in \mathbb{R}^{P \times d}$7 and $M^U \in \mathbb{R}^{P \times d}$8, indicating deterministic per-sample selection together with broad usage across the batch (Kirsch et al., 2018). Qualitative language-modeling examples show modules specializing in numeral contexts, sentence beginnings, and contexts following the definite article.

Interpretability is explicit in the 2026 MNC because all memory cells are scalar and the complete algorithmic state is externally represented and inspectable (Leon, 4 Mar 2026). Control flow is encoded only by one-hot module gates; there is no symbolic branching or dynamic rewiring. In the A* case study, the search frontier, node records, parent pointers, and phase selector all reside in named memory regions.

The 2024 CML-HDC architecture emphasizes human-editable structure. Changing the subgoal sequence requires only rebuilding the HDC policy vector; changing one object location requires recomputing only the corresponding $M^U \in \mathbb{R}^{P \times d}$9 term in the map; permanently closing a door requires zeroing the corresponding entries in the object-CML gating matrix $l$00 (McDonald, 2024). The authors characterize this as local “surgery” rather than global retraining.

The 2019 list-sorting framework likewise stresses maintainability and extensibility: each primitive’s interface is fixed, and a symbolic or hand-coded implementation can replace a learned module without modifying others (Castillo-Bolado et al., 2019). This suggests a persistent theme across MNC variants: interpretability is not only about inspecting activations, but about exposing control flow, interfaces, and intermediate state in a modular form.

7. Limitations, controversies, and research directions

The literature identifies several nontrivial trade-offs. In modular networks, capacity scales with $l$01 while per-sample compute scales with $l$02, so one can increase total parameter count with small marginal computational cost by keeping $l$03 (Kirsch et al., 2018). A plausible implication is that modularity is partly a systems strategy for decoupling representational capacity from activation cost. However, this does not by itself guarantee better generalization.

The 2019 sorting results illustrate one limitation directly: although modular training is faster and more stable, decomposing the problem can remove the global gradient signal needed for long-range pattern discovery, and the modular system generalized worse to longer unseen list lengths than the monolithic baseline (Castillo-Bolado et al., 2019). The paper also notes design overhead in defining selective-input and selective-update interfaces for each primitive.

The 2024 CML-HDC system exposes a different constraint: reconfigurability depends on robust hypervector cleanup and viable map encodings, yet only approximately one fifth of random object layouts yielded unambiguous cleanup, and additional path-tracking was needed to avoid dithering in the grid-level planner (McDonald, 2024). This indicates that symbolic compositionality in hypervector space can itself become a bottleneck.

The 2026 MNC introduces another axis of debate. Because it compiles known algorithms into analytically specified modules rather than learning them from data, it is exact and deterministic by construction but “less general than end-to-end differentiable learners” (Leon, 4 Mar 2026). This is not a defect under the paper’s stated objective; rather, it places the model in a distinct regime where neural components serve as a substrate for explicit algorithm realization. A plausible implication is that the phrase “Modular Neural Computer” now spans both learned modular computation and compiled neural execution, and care is required when comparing systems under that label.

Future directions explicitly mentioned in the cited works include automated module discovery or dynamic module selection policies via reinforcement learning (Castillo-Bolado et al., 2019), preserving modularity and exactness while searching or evolving the analytically specified modules of the 2026 MNC (Leon, 4 Mar 2026), replacing bipolar HDC with spiking-phasor networks or stacking additional CML layers for semantic or causal abstraction (McDonald, 2024), and continued exploration of modular, recursive, and procedural neural programming with programs stored as data in memory (Le et al., 2020). Taken together, these works position MNC as an umbrella for architectures that separate control, memory, and transformation so that neural computation can be conditionally activated, reconfigured, or compiled with explicit intermediate state.