Neural Cellular Automata Learn General Features in their Hidden Channels

Abstract: Modern deep learning models achieve impressive generalization through over-parameterization, but this paradigm often struggles with overfitting and memorization in few-shot regimes. Neural Cellular Automata (NCAs) offer a highly parameter-efficient alternative, yet research has focused primarily on their output, leaving the role of their internal hidden channels largely unexplored. In this paper, we investigate the internal dynamics of NCA hidden channels and introduce a novel transfer-learning mechanism that injects a pretrained teacher's hidden states into a student model to guide early optimization. Evaluated on few-shot and scale-variant MNIST benchmarks, NCAs outperform comparable recurrent and feed-forward architectures, demonstrating superior generalization with a minimal parameter budget (~9,800 parameters). Mechanistic analysis reveals that the hidden channels decouple feature extraction from uniform classification consensus by absorbing morphological complexity and converging to mutually orthogonal states. Furthermore, we demonstrate that these hidden channels capture general, scale-invariant topological primitives rather than class-specific templates. This allows a student model to achieve strong few-shot performance on unseen classes using features transferred from a teacher trained only on a subset of digits (0-5). Our results highlight the potential of utilizing hidden-state dynamics as a robust, decentralized computational substrate for parameter-efficient transfer learning

• The paper demonstrates that the hidden channels of Neural Cellular Automata (NCAs) can store and transfer spatially structured geometric features, enhancing few-shot learning.
• Tempory hidden state injection allows students to acquire representation and reduces transfences from NCA to similar dataset.
• The NCA’s local, translation-equivariant update rule contributes to its robust performance in moderate scale variations.

Research problem and contribution

The paper investigates whether the hidden channels of Neural Cellular Automata (NCAs) encode transferable, task-relevant representations rather than merely serving as transient message-passing buffers. Its central claim is that NCA hidden states form spatially distributed feature maps with general geometric content, while separate readout channels converge toward a spatially uniform classification consensus. On this basis, the paper introduces a transfer mechanism that injects the detached hidden state of a pretrained teacher NCA into a student NCA.

This approach differs from conventional transfer learning, which generally transfers parameters or intermediate activations within a fixed feed-forward computation. Here, the transferred object is a developmental state: a spatial tensor representing the teacher’s mature internal configuration. The student retains its own parameters and subsequently evolves the injected state according to its own local update rule. The paper therefore treats the NCA hidden state as both a representational substrate and an initialization mechanism for recurrent spatial computation (2609.21870).

The empirical setting is deliberately constrained. All principal models contain approximately 9,800 parameters and are evaluated on few-shot MNIST classification with $k \in \{1,5,10,25,50,100\}$ examples per class. The comparison includes an NCA, a globally perceptive recurrent model, and two feed-forward variants. Models are trained for 3,000 AdamW iterations and evaluated on the full MNIST test set, including linearly downscaled versions of the images. The study also uses 100 random seeds and defines the 80% test accuracy level as a practical threshold for comparing sample efficiency.

NCA architecture and transfer protocol

The NCA operates on a spatial grid through repeated local, shared updates. Its state is divided into visible and hidden channels. Channel 0 carries the input digit, channels 1–10 encode the one-hot classification output, and channels 11–15 constitute the non-solution hidden channels used for internal computation. The paper’s mechanistic analysis focuses primarily on these latter channels.

The transfer protocol extracts the teacher’s non-solution channels after the teacher has processed an input, detaches the resulting tensor from the computational graph, and injects it into the student at developmental step $T=0$. Detachment prevents gradients from propagating into the teacher and ensures that the student cannot improve by directly modifying the teacher’s representation. Two variants are studied. In the inductive-bias condition, injection is used only during the first 1,500 training steps, after which the student must reconstruct the hidden state autonomously. In the transfer-learning condition, the hidden state remains available during both training and evaluation.

Figure 1: Teacher hidden-state extraction and detached injection into the student NCA.

The distinction between these variants is important. Temporary injection tests whether exposure to a mature hidden representation changes the student’s learned dynamics, whereas permanent injection tests whether the representation itself can function as a transferable conditioning signal. The former therefore probes developmental bootstrapping; the latter is closer to a conditioned readout architecture.

Few-shot generalization and scale variation

Without hidden-state transfer, the NCA generally outperforms the matched recurrent and feed-forward baselines in few-shot generalization, with one explicit exception: the classic mapping model performs better in the 1-shot setting. The NCA reaches the 80% threshold at approximately four examples per class, compared with approximately ten examples per class for the classic mapping and global recurrent models and approximately eleven for the other mapping model. This is a substantial sample-efficiency difference given that all models have comparable parameter counts.

Figure 2: Few-shot and scale-variant MNIST performance for the NCA and parameter-matched baselines.

The scale experiments provide a stronger test than interpolation over additional training examples. Digits are downscaled using nearest-neighbor interpolation and padded back to the original $28 \times 28$ canvas. At 75% and 50% scale, the NCA outperforms the comparison models across the evaluated few-shot conditions. At 25% scale, however, its performance is not statistically distinguishable from that of the baselines. Thus, the evidence supports improved robustness to moderate scale variation, but not uniformly superior performance under severe resolution reduction.

The paper attributes this behavior to the NCA’s local, translation-equivariant update rule. Since each cell communicates only with a local neighborhood and the same rule is applied throughout the grid, the model has limited ability to memorize absolute spatial coordinates. This inductive bias plausibly favors local geometric structures over pixel-level templates. However, the experiments do not isolate locality from other architectural properties, such as recurrence, stochastic developmental dynamics, or the particular channel allocation. Consequently, the causal contribution of each design choice remains unresolved.

Hidden-state injection as developmental bootstrapping

Temporary hidden-state injection improves the NCA’s performance across all evaluated few-shot conditions. The injected student crosses the 80% threshold between approximately three and four examples per class, improving on the already strong non-injected NCA. It also performs better on downscaled inputs, including a statistically significant, although modest, result at 25% scale where the unassisted NCA was not statistically distinguishable from the baselines.

Figure 3: Performance when the student receives a pretrained hidden state during an initial portion of training.

The training dynamics expose a distribution shift intrinsic to this protocol. During the injection phase, the student receives a mature hidden representation and rapidly drives training loss toward zero while validation loss stabilizes. When injection is removed, training loss temporarily increases because the student must generate the hidden state rather than consume it. Validation loss subsequently decreases substantially. The result indicates that the transfer mechanism does not simply provide a permanently available shortcut; it can alter the optimization trajectory such that the student later learns to reconstruct useful internal states.

Figure 4: One-shot training and validation losses before and after hidden-state injection is removed.

This interpretation requires qualification. The reported loss curve is from a single one-shot run, rather than an aggregate over the 100 random seeds. It therefore illustrates the mechanism but does not establish that the same transition occurs with identical timing or magnitude across runs. Moreover, the temporary-injection protocol exposes the student to a training distribution that differs from the post-injection training and testing distributions. The observed transient is consequently both an optimization effect and a consequence of covariate shift.

Permanent access to the teacher’s hidden channels produces a much larger improvement. Under this condition, the student receives the pretrained state throughout training and validation and substantially outperforms all comparison models in the few-shot experiments.

Figure 5: Few-shot and scale-variant performance when teacher hidden channels remain available throughout training and testing.

The interpretation is more limited than in the temporary-injection condition. With permanent access, the student is effectively a conditioned classifier whose input includes both the digit and a teacher-derived spatial representation. The result demonstrates the utility of hidden-state transfer, but it does not by itself show that the student independently acquires an equally powerful internal representation. The temporary-injection experiment is therefore more informative about learned reconstruction and autonomous developmental dynamics.

Transfer from incomplete class knowledge

The paper’s strongest transfer claim comes from the reduced-knowledge experiment. The teacher is trained on only digits 0–5, while the student is trained and evaluated on all ten MNIST classes. Despite the teacher’s restricted class coverage, its hidden channels support strong student generalization across the 1–100-shot range. The scale-dependent degradation is reported to be similar in magnitude to that obtained when the teacher has been trained on all MNIST digits.

Figure 6: Transfer performance when the teacher has been trained only on digits 0–5 and the student must classify all MNIST classes.

This result is consistent with the hypothesis that the hidden channels encode local topological primitives—such as edges, junctions, and intersections—rather than class-specific digit templates. If correct, the representation transferred from digits 0–5 contains geometric features applicable to digits 6–9. The claim is also compatible with the NCA’s local connectivity constraints, which make global coordinate memorization difficult.

The terminology in the experimental description should nevertheless be interpreted carefully: digits 0–5 comprise six classes, although the text refers to a “5-way MNIST dataset” in one place. More importantly, the experiment does not fully disentangle topological transfer from ordinary low-level visual transfer. Digits 0–5 and 6–9 share stroke statistics, local edge structures, and the same image-generation process. A stronger test would require systematically controlled structural dissimilarity, class-balanced morphology splits, or non-MNIST domains.

Internal division of labor

The paper analyzes the hidden channels through spatial variance over developmental time. For each channel subset, spatial variance is averaged across samples and spatial positions. Hidden channels 11–15 rapidly increase in spatial variance as the NCA forms spatially structured features and then approach a plateau. By contrast, the classification channels remain near-zero in spatial variance, indicating that they converge toward a nearly uniform grid-wide state.

Figure 7: Temporal evolution of spatial variance for teacher and student hidden channels.

Figure 8: Temporal evolution of spatial variance for the classification readout channels.

This establishes a clear functional asymmetry. The hidden channels retain morphological complexity in spatially heterogeneous maps, whereas the readout channels suppress spatial variation to distribute a common class decision across the grid. The NCA therefore separates feature construction from consensus formation: internal channels represent the input in a spatially structured manner, while output channels aggregate that representation into a robust global classification.

This division of labor is significant because it challenges the view that NCA hidden channels are merely temporary communication buffers. Their persistent spatial variance suggests that they store and transform task-relevant structure across developmental steps. At the same time, the analysis is correlational. Spatial variance alone does not prove that a channel is causally necessary for feature extraction, nor does it identify the specific computation performed by any individual channel.

The cross-channel cosine-similarity analysis further indicates that hidden channels become approximately orthogonal on average over time.

Figure 9: Cross-channel cosine similarity over developmental time for teacher and student hidden states.

The authors interpret the near-zero average similarity as evidence that the constrained NCA allocates distinct algorithmic roles to different channels and minimizes representational redundancy. This is a plausible account of how an approximately 9,800-parameter model maintains multiple complementary feature maps. It should not, however, be equated with statistical independence or functional disentanglement. Cosine orthogonality is a second-order geometric property of channel activations; it does not establish that channels encode semantically independent variables or that removing one channel selectively eliminates a particular computation.

The teacher and student dynamics also differ. The teacher exhibits comparatively smooth channel relationships from the initial developmental step, whereas the student undergoes a high-variance transient before converging toward its own configuration. This behavior supports the claim that the student does not simply copy the teacher state. Instead, it transforms the injected representation according to its own parameterization and local dynamics.

Scale-invariant spatial features

The hidden-state visualizations show qualitatively similar patterns for full-size and downscaled instances of the same digit. In particular, the paper identifies edge-like structures and topological intersections in distinct hidden channels.

Figure 10: Hidden-channel representations for a full-size digit and its 50% downscaled counterpart.

These observations support the proposed explanation for the NCA’s scale robustness: local feature maps preserve geometric relations while discarding some dependence on absolute image scale and position. The result is compatible with the quantitative scale experiments, especially the NCA’s advantage at 75% and 50% resolution.

The evidence remains suggestive rather than conclusive. The scale transformation is simple, uses nearest-neighbor downsampling, and preserves the original canvas through padding. It does not test arbitrary rescaling, stroke-width variation, rotations, affine transformations, or changes in image statistics. In addition, visual similarity of hidden maps does not establish formal scale equivariance. The strongest defensible conclusion is that the learned representations exhibit empirical robustness to the specific scale transformations used in the benchmark.

Limitations and open questions

The evaluation is restricted to monochromatic MNIST and transformed MNIST. This dataset is useful for controlled analysis of morphology and topology, but its limited visual complexity makes it insufficient to establish that hidden-state transfer will remain effective on high-resolution, multichannel, or natural-image data. The paper explicitly leaves open whether the same mechanisms extend to datasets such as CIFAR-10 or ImageNet.

The computational comparison is also asymmetric. Although the NCA uses only approximately 9,800 parameters, it performs many developmental updates and requires BPTT through those steps. The feed-forward baselines receive substantially less sequential computation, while the global recurrent model is the more appropriate computational comparator. Parameter matching therefore does not imply equal FLOP, memory, latency, or optimization budgets. The NCA’s sample efficiency should be reported alongside developmental-step counts and wall-clock costs before making broader efficiency claims.

Several mechanistic claims require additional intervention-based evidence. The near-zero cosine similarity does not establish channel specialization; the hidden-state trajectories do not identify attractor stability quantitatively; and the reduced-knowledge transfer experiment does not conclusively distinguish topological primitives from shared MNIST-specific statistics. Channel ablations, causal activation patching, controlled teacher–student architecture mismatches, and transfer across datasets would directly test these hypotheses.

The paper also leaves open how transfer quality depends on the developmental step at which the teacher state is extracted, the number and identity of injected channels, the degree of teacher–student parameter mismatch, and the spatial alignment between teacher and student grids. These questions are central to determining whether hidden-state transfer is a general NCA mechanism or a property of the particular MNIST implementation.

Conclusion

The paper presents evidence that NCA hidden channels encode persistent, spatially structured features that are functionally distinct from uniform classification readouts. In matched-parameter MNIST experiments, NCAs reach useful few-shot accuracy with fewer examples than the comparison models and show improved robustness to moderate scale reduction. Detached hidden-state injection further improves performance, and transfer from a teacher trained on digits 0–5 supports classification of the remaining MNIST classes.

The mechanistic analyses associate this behavior with a division of labor between morphologically variable hidden channels and spatially uniform readout channels, together with low average cross-channel similarity and developmental trajectories that depend on the student’s own dynamics. The results support hidden-state transfer as a parameter-efficient form of NCA bootstrapping, while the restricted benchmark, asymmetric computational budget, and primarily correlational mechanistic evidence limit the scope of the conclusions.