Stability and Geometry of Attractors in Neural Cellular Automata

Abstract: Throughout the literature on Neural Cellular Automata (NCAs), it is often taken for granted that the systems learn attractors. This is shown through evolving the system for many timesteps and noting visual similarity to the goal state. There remain many questions after such an analysis. Namely, what kind of attractors do we have? Is their behavior ordered or chaotic? Can we estimate stability over very long time horizons? What really happens in the attractor when perturbations are applied? In this paper, we present a case study to help answer these questions, with methods drawn from the literature on dynamical systems theory. We use the growing gecko NCA of Mordvintsev et al. (2020) with deterministic cell updates as a case study. To the best of the authors' knowledge, we present the first visualizations of NCA attractor dynamics. We also analyze them using the Lyapunov and Fourier spectra, to reveal that the NCA displays oscillatory, periodic and quasi-periodic behavior, and that these behaviors arise early during training. This challenges the belief that NCAs learn fixed point attractors. Finally, we show that large perturbations to the attractor states can throw the NCAs into a secondary mode separate from the original attractor. We hope that this initial foray into NCA attractor dynamics expands the toolkit for NCA researchers to analyze the robustness and stability of their systems.

• The paper demonstrates that well-trained NCAs exhibit oscillatory and quasi-periodic attractors rather than converging to fixed points.
• It employs Lyapunov spectra, PCA, and Fourier analysis to quantify stability and the low-dimensional geometry of high-dimensional attractor manifolds.
• Findings reveal that robust NCA training results in multistable attractor basins that respond differently to perturbations, informing design and application.

Stability and Geometry of Attractors in Neural Cellular Automata

Introduction

This paper investigates the long-term dynamics and stability properties of Neural Cellular Automata (NCAs) by applying techniques from nonlinear dynamics and the theory of dynamical systems. NCAs, which have found applications in domains requiring robust self-organizing computation, are generally assumed to learn fixed-point attractors that guide their observed macroscopic behavior. The analysis in this work, however, provides the first in-depth numerical and visual study of NCA attractor dynamics, challenging several standing assumptions about their limiting asymptotic states. Unlike prior work that focused on empirical stability or loss curves, this study employs Lyapunov spectrum analysis, Fourier spectra, principal component analysis (PCA), and visualization of basins of attraction to systematically characterize the dynamics, stability, and geometry of NCA attractors.

Methodological Approach

The authors use a deterministic variant of the "growing gecko" NCA, originally described by Mordvintsev et al. (2020), as a canonical example. The training regime ensures pool-training and perturbation-resilience, which is typical in robust NCA studies. Three independent NCA models (differing only in random initialization) were trained to maintain or regenerate a goal image from diverse perturbed states. After allowing for adequate burn-in periods, attractor dynamics were quantitatively and qualitatively probed using burn-in, PCA for dimensionality assessment, Lyapunov exponents for stability assessment, and Fourier analysis for discriminating among periodic, quasi-periodic, and chaotic attractors.

The high-dimensional state-space (over 25,600 dimensions) is systematically projected into lower-dimensional representations to inform manifold geometry and the effective dimensionality of the attractors, with careful attention paid to the pitfalls and variance explained by linear projections such as PCA.

Figure 1: The burn-in phase of the attractor, showing initial rapid convergence to the goal state, followed by slower, rotational convergence, and ultimate settling along the rotational axis.

Figure 3: Attractor geometry of the green gecko NCA for three model instantiations. Each demonstrates distinct oscillatory or quasi-periodic structure upon PCA projection.

Attractor Geometry and Dynamics

Contrary to the fixed-point hypothesis prevalent in the NCA literature, the results unequivocally show that NCA attractors for the growing gecko task are not fixed points—none of the three model instantiations converge to static states, even after extended simulation. Instead:

• Version 1 exhibits clear periodic (limit-cycle) behavior, confined to a two-dimensional manifold within the high-dimensional state space. This periodicity is confirmed both visually and via strong, isolated peaks in its Fourier spectrum.
• Version 2 demonstrates quasi-periodic motion along a 4-torus, with four independent incommensurate frequencies dominating its Fourier spectrum, and a higher intrinsic dimensionality (up to 35 PCA components to explain 95% of the variance).
• Version 3 forms an even more geometrically complex attractor (quasi-periodic, likely a 3-torus), as corroborated by both PCA variance and the multiplicity of frequencies in the Fourier transform.

  Figure 5: Unscaled, periodic attractor geometry for version 1, revealing minimal movement along the third principal component.

The attractor manifolds for all versions remain low-dimensional relative to the original state space, typically spanning between 2 to 33 PCA axes for 95% variance, despite the rigorous pool-training and noise resilience.

Stability Assessment via Lyapunov Spectrum

Estimation of the top Lyapunov exponents for all attractor modes reveals no evidence of chaotic instability. All leading exponents are negative or zero, indicating Lyapunov stability: small perturbations in the state decay and trajectories are predictable over long horizons. Specifically:

• Version 1: $\lambda_1 = 0$, $\lambda_2 = -0.002$, with subsequent exponents more negative. This is characteristic of periodic dynamics.
• Version 2: Two near-zero exponents ($\lambda_{1,2} \approx -0.005$), suggesting quasi-periodic behavior.
• Version 3: All leading exponents are slightly negative ($\lambda_1 = -0.009$), which, in conjunction with volume-proxy analyses demonstrating no contraction, argues for numerical artifacts rather than true dissipative decay.

No chaotic (positive Lyapunov) behavior is identified, in contrast to models such as the Lorenz attractor.

Characterization via Fourier Analysis

Fourier spectral analysis bolsters the above conclusions:

Figure 2: Fourier power spectra for the three attractors. Peaks indicate the number and character of dominant oscillatory modes: periodic in version 1, and quasi-periodic in versions 2 and 3.

• For version 1, a prominent single peak dominates, indicating limit-cycle periodicity.
• For versions 2 and 3, several nonharmonic independent frequencies persist, signifying high-order quasi-periodicity.

These results indicate that oscillatory and quasi-periodic attractors, not fixed points, are the norm for these trained NCA systems, and such behavior emerges early during learning.

Basins of Attraction and Perturbation Response

Basins of attraction were probed using large and small perturbations from states on the attractor manifold. The system robustly returned to the attractor under small perturbations, but large perturbations induced unexpected transitions to secondary oscillating modes (secondary limit cycles or quasi-periodic attractors), distinct in geometry and location in the reduced state space.

Figure 4: NCA trajectories recovering from small (top) and large (bottom) perturbations, returning to either the original or a secondary attractor, depending on perturbation magnitude.

These observations indicate that while the attractor basin is locally robust, it is partitioned—sufficiently large perturbations can induce phase switches to structurally distinct attractors, despite all sharing the same goal image in the output channels. This supports the notion of a multitude of accessible attractors within the large unconstrained subspaces of NCA hidden channels.

Attractor Emergence During Training

Attractor geometry and stability emerge surprisingly early in training. Oscillatory dynamics manifest well before perfect goal-image maintenance is achieved, and attractor structure stabilizes before the end of training.

Figure 9: Attractor evolution as a function of training epoch, indicating early emergence and stabilization of oscillatory modes.

Theoretical and Practical Implications

The findings contradict the fixed-point attractor assumption for well-trained NCAs, revealing that multistability and high-structured oscillatory or quasi-periodic modes are common. This has several implications:

• Practical: For robust engineering of NCAs in critical applications (e.g., medical imaging), understanding the attractor's class is essential to predict long-term behavior under perturbation and confirm stability.
• Theoretical: The prevalence of quasi-periodic/oscillatory attractors raises questions about the universal dynamical regimes accessible by high-dimensional, nonlinear, spatially coupled neural substrates. The role of training objectives and pool-based convergence remains ripe for further analytic exploration.
• Future Work: Extending this methodology to stochastic NCAs (random dynamical systems) requires new tools, potentially incorporating attractor theory for random systems. Additionally, detailed studies of local versus global (cell-wise vs. substrate-wise) attractor synchronization and the possible presence of strange nonchaotic attractors (SNAs) represent significant directions, as does formalizing emergent secondary attractor phenomena following large perturbations.

Conclusion

This study presents the first systematic, multi-perspective characterization of attractor geometry and stability in Neural Cellular Automata. Through dynamical systems analysis, it is shown that fixed-point attractors are generally not learned—oscillatory and quasi-periodic manifolds are instead typical, even under robust training. The attractor basins can partition under significant perturbations, supporting emergent multistability in the NCA hidden subspace. These insights have immediate consequences for robust design and interpretation of NCA-driven systems and lay groundwork for future theoretical advancements in understanding high-dimensional, distributed, self-organizing computation.