Continuous Attractors in Neural Networks

• Continuous attractors are invariant equilibrium manifolds that encode continuous variables via neutral stability along the manifold and contraction in transverse directions.
• They are characterized by zero eigenvalues along the attractor and strictly negative eigenvalues normal to it, ensuring robust analog memory and stability.
• Applications include neural network dynamics and machine learning, where methods like Jacobian rank analysis and spectral-gap regularization guide model design.

Continuous attractors are invariant manifolds of equilibria that encode a continuous variable by position along the manifold while contracting perturbations in the transverse directions. In the canonical setting $\dot x=f(x)$, an $l$-dimensional continuous attractor $M\subset\mathbb R^d$ consists of points satisfying $f(x)=0$ for all $x\in M$, with exactly $l$ zero eigenvalues tangent to $M$ and strictly negative eigenvalues normal to it; line attractors and ring attractors are the standard examples in neural dynamics (Ságodi et al., 2024). In both theoretical neuroscience and machine learning, continuous attractors are now treated as geometric objects—typically smooth submanifolds or their perturbative remnants—whose existence, stability, mobility, and learnability can be analyzed through Jacobian rank, spectral gaps, Lyapunov exponents, and slow–fast decomposition (Tian et al., 3 Sep 2025).

1. Equilibrium manifolds and local geometry

A continuous attractor can be defined as an equilibrium set endowed with Lyapunov stability at every point. In a differential-manifold formulation, one considers a smooth vector field

$\dot x=F(x)=\sigma(Wx+b)-Ax,$

with equilibrium set

$X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$

In this framework, $X$ is called a continuous attractor if each $l$0 is Lyapunov-stable. Under constant-rank conditions on the Jacobian $l$1, the equilibrium set is locally a smooth submanifold: if $l$2 for all $l$3 and $l$4 at some $l$5, then $l$6 is, near $l$7, an $l$8-dimensional $l$9 manifold (Tian et al., 3 Sep 2025).

The local geometry is especially transparent in adapted coordinates. The constant-rank formulation yields charts in which the attractor is represented as $M\subset\mathbb R^d$0, and the vector field can be written locally as

$M\subset\mathbb R^d$1

so that $M\subset\mathbb R^d$2 is invariant and attracting in the normal directions. This replaces an informal notion of a “flat valley” in state space with a precise manifold statement: the attractor is neutral along the coordinates $M\subset\mathbb R^d$3 and contracting along the coordinates $M\subset\mathbb R^d$4 (Tian et al., 3 Sep 2025).

In rate-model language, the same structure appears in recurrent networks of the form

$M\subset\mathbb R^d$5

A continuous manifold of fixed points $M\subset\mathbb R^d$6 satisfies $M\subset\mathbb R^d$7 for all $M\subset\mathbb R^d$8, and the tangent vector $M\subset\mathbb R^d$9 obeys $f(x)=0$0. The state therefore remembers a continuous variable not by selecting one isolated equilibrium, but by selecting one point on a neutrally stable equilibrium manifold (Park et al., 2023).

2. Spectral characterization and stability criteria

The local Jacobian spectrum determines whether an equilibrium manifold is a true continuous attractor rather than a merely degenerate set of fixed points. For a one-dimensional attractor, the required condition is

$f(x)=0$1

with

$f(x)=0$2

together with a spectral gap

$f(x)=0$3

More generally, if the zero eigenvalue has multiplicity $f(x)=0$4 and all remaining eigenvalues satisfy $f(x)=0$5, then the system has a $f(x)=0$6-dimensional neutral manifold with exponential stability in the normal directions (Tian et al., 3 Sep 2025).

This criterion can be read directly in terms of tangent and normal modes. Along the manifold, zero eigenvalues imply marginal stability: infinitesimal displacements neither decay nor grow. Transverse to the manifold, negative real parts imply rapid return. In neural terms, these two ingredients separate persistent analog memory from ordinary relaxation dynamics. The spectral gap is not merely technical; it quantifies the time-scale separation that makes the manifold function as a memory substrate rather than as a weakly unstable or weakly dissipative set (Tian et al., 3 Sep 2025).

Equivalent statements appear in classical neural-network formulations. With

$f(x)=0$7

the condition $f(x)=0$8 must hold along the whole manifold, while transverse modes must remain in the strictly negative-eigenvalue subspace. This is the standard linearized signature of continuous attractors in recurrent rate models (Park et al., 2023).

In feedforward or classification networks, the overall Jacobian need not be square, so eigenvalues are unavailable. A recent extension replaces eigenvalue analysis with singular-value analysis. If

$f(x)=0$9

then “singular-value stratification” refers to a plateau

$x\in M$0

This does not define an exact continuous attractor in the equilibrium-manifold sense, but it provides a spectral analogue for approximate neutral directions in non-square Jacobians (Tian et al., 3 Sep 2025).

3. Structural instability, bifurcation, and persistent slow manifolds

Exact continuous attractors are structurally fragile. They occupy a measure-zero subset of vector fields, and a generic small perturbation shifts one of the zero eigenvalues away from zero, thereby destroying the continuum of equilibria (Ságodi et al., 2024). In recurrent networks, this fragility appears as the fine-tuning problem: the condition

$x\in M$1

imposes infinitely many constraints on $x\in M$2, so an infinitesimal weight perturbation $x\in M$3 generically shifts the zero eigenvalue by $x\in M$4, replacing the continuum with a discrete set of fixed points or inducing drift along the former manifold (Park et al., 2023).

Two codimension-1 normal forms organize common bifurcations away from exact continuity. One is the saddle–node on a line of equilibria,

$x\in M$5

and the other is the pitchfork,

$x\in M$6

In both cases, the continuum present at the degenerate point disappears once $x\in M$7. The asymptotic dynamics then differs categorically from that of an exact continuous attractor, even though finite-time behavior can remain similar (Ságodi et al., 2024).

Persistent manifold theory explains this similarity. In local coordinates $x\in M$8, the unperturbed system can be written as

$x\in M$9

while a perturbation introduces

$l$0

If the original equilibrium manifold $l$1 is compact and normally hyperbolic, then for sufficiently small $l$2 there exists a locally invariant slow manifold $l$3, diffeomorphic to $l$4, at Hausdorff distance $l$5. Trajectories rapidly approach $l$6 and then drift slowly along it (Ságodi et al., 2024).

A central implication is that exact continuous attractors need not survive perturbation to remain functionally relevant. Approximate continuous attractors, or “ghost” slow manifolds, can preserve finite-time analog memory and near-neutral dynamics even after structural instability has removed the exact equilibrium continuum. This suggests why trained recurrent networks and biological circuits can exhibit behavior closely resembling continuous-attractor dynamics without satisfying the exact fine-tuning conditions (Ságodi et al., 2024).

A related contrast concerns periodic and quasi-periodic attractors. A stable limit cycle has one zero Lyapunov exponent in the phase direction and negative exponents transversely; a $l$7-torus has $l$8 zero exponents. These objects provide non-decaying directions while remaining structurally stable under parameter perturbation. This has motivated the view that periodic and quasi-periodic attractors can serve as robust alternatives when exact continuous attractors are too fragile (Park et al., 2023).

4. Neural-field realizations, bump dynamics, and computational regimes

The canonical neural implementation is the continuous attractor neural network (CANN), in which neurons are indexed by a continuous feature coordinate $l$9 and recurrent connectivity is translationally invariant. In one common formulation, synaptic input $M$0, firing rate $M$1, and adaptation current $M$2 obey

$M$3

$M$4

With $M$5 and $M$6, the translational symmetry of kernels such as a Gaussian or cosine yields a one-parameter family of bump states centered at arbitrary $M$7. The translation mode is a Goldstone mode with eigenvalue $M$8, while amplitude and width modes have negative eigenvalues. When adaptation is included, the same network supports static bumps, spontaneous traveling waves, smooth tracking, oscillatory tracking, anticipatory tracking, and runaway regimes; for example, the static-to-drift boundary occurs at $M$9, and the Hopf boundary for oscillatory tracking occurs at $\dot x=F(x)=\sigma(Wx+b)-Ax,$0 (Li et al., 2024).

Two-dimensional CANNs with short-term synaptic depression (STD) or spike-frequency adaptation (SFA) show qualitatively similar behavior. In both cases, Gaussian bump states can lose translational stability and begin moving spontaneously, and perturbative expansions in Hermite–Gaussian modes predict phase diagrams, dynamical variables, and speed of spontaneous motion (Fung et al., 2015). This suggests that mobility on continuous attractors is not an anomaly of one-dimensional ring models, but a recurrent consequence of slow negative feedback in translationally invariant neural fields.

External forcing imposes quantitative limits on how rapidly a representation stored on a continuous attractor can be updated. In a recurrent rate network with a continuum of bump states on a ring, projection of the full dynamics onto the bump center $\dot x=F(x)=\sigma(Wx+b)-Ax,$1 yields an overdamped Langevin equation

$\dot x=F(x)=\sigma(Wx+b)-Ax,$2

For a moving cup-shaped drive, the bump lags the minimum by a steady offset determined by the effective potential

$\dot x=F(x)=\sigma(Wx+b)-Ax,$3

A critical speed

$\dot x=F(x)=\sigma(Wx+b)-Ax,$4

marks the loss of continuous tracking, and an absolute upper bound takes the form

$\dot x=F(x)=\sigma(Wx+b)-Ax,$5

In place-cell-style models with multiple stored maps, quenched disorder acts as an effective temperature and yields a velocity-dependent non-equilibrium memory capacity

$\dot x=F(x)=\sigma(Wx+b)-Ax,$6

These results place continuous attractors within a non-equilibrium statistical-mechanical framework rather than a purely static one (Zhong et al., 2018).

Several perturbations modulate the trade-off between flexibility and stability. In a CANN with STD, reducing the variance of the release-probability parameter $\dot x=F(x)=\sigma(Wx+b)-Ax,$7 to model astrocytic NMDA blockade shifts the static-to-traveling boundary to larger $\dot x=F(x)=\sigma(Wx+b)-Ax,$8, reduces spontaneous traveling speed, increases reaction time to abrupt input shifts, and decreases the diffusion coefficient of the bump center by $\dot x=F(x)=\sigma(Wx+b)-Ax,$9 (Liu et al., 2024). Under oscillatory drive, a one-dimensional CANN can also exhibit discrete-attractor-like tracking: discontinuous jumps of the bump center phase-locked to slow-gamma oscillation emerge without any discreteness assumption in the network architecture (Fung et al., 2018).

5. Learning continuous attractors in artificial neural networks

Recent work has shifted from hand-designed CANNs to learned attractor structure. A reservoir-computing study trained a 1000-neuron RNN on isolated and shifted examples of either stable limit cycles or chaotic Lorenz attractors and reported that the network learned a continuum of attractors, quantified by an extra Lyapunov exponent equal to zero. The proposed abstraction mechanism combines differentiable generalized synchronization with feedback dynamics, and the resulting system was described as learning a continuous dynamical attractor memory from isolated examples of dynamical attractor memories (Smith et al., 2021).

A broader artificial-neural-network perspective treats continuous attractors through local differential geometry and singular-value structure. In common classification models and datasets—specifically MNIST, CIFAR-10, Fruits, MLP, CNN, and ResNet-18—a differential-manifold analysis argues for the universality of singular-value stratification. Empirically, for almost every training sample $X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$0,

$X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$1

and the typical pattern is

$X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$2

with a nontrivial spectral gap. The paper interprets this as evidence that approximate continuous-attractor manifolds of dimension $X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$3 may underlie learned representations and suggests that such structure may be ubiquitous in general neural networks (Tian et al., 3 Sep 2025).

The same framework proposes a practical detection procedure: compute the sample Jacobian $X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$4 by backpropagation, perform an SVD, compute the coefficient of variation $X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$5, and, if $X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$6 with $X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$7 exemplified by $X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$8, declare strong stratification and estimate the plateau dimension $X=\{x^*\in\mathbb R^n:F(x^*)=0\}.$9 from the dominant singular-value cluster (Tian et al., 3 Sep 2025). This is not a proof of exact equilibrium-manifold dynamics in feedforward classifiers, but it supplies an operational criterion for nearly neutral directions in local linearizations.

The same study also gives engineering prescriptions for inducing such behavior: Jacobian rank control, spectral-gap regularization, symmetric or normal weight constraints, layerwise attractor induction through slow–fast decomposition, and data augmentation along known low-dimensional manifolds such as rotations or translations (Tian et al., 3 Sep 2025). A plausible implication is that continuous-attractor behavior in modern neural networks may be better understood as an architectural and spectral design principle than as a narrowly biological motif.

6. Continuous attractors and the continuity of attractors

A persistent terminological confusion is that a “continuous attractor” is not the same object as an attractor that depends continuously on a parameter. In the latter setting, one studies a family $X$0 and asks whether

$X$1

for sufficiently small $X$2, where $X$3 is the Hausdorff distance. For maps $X$4, this notion concerns continuation of attractors in parameter space, not neutral directions in state space (Glendinning et al., 2019).

The distinction matters because the two theories answer different questions. Continuous-attractor theory concerns equilibrium manifolds, tangent zero modes, and memory of analog variables. Continuity-of-attractors theory concerns whether a family of attractors can be followed under perturbation. In semigroup language, equi-attraction is equivalent to continuity of the global attractor map on compact parameter spaces, while pointwise convergence without uniformity yields continuity only on a residual set by Baire category (Hoang et al., 2014). Closely related residual-continuity and equi-attraction results hold for pullback and uniform attractors of non-autonomous processes (Hoang et al., 2016) and for pullback $X$5-attractors in time-dependent phase spaces (Li et al., 2022).

Examples show that parameter continuity can succeed or fail independently of whether the attractor is “continuous” in the equilibrium-manifold sense. In smooth unimodal families, dense periodic windows force discontinuities of chaotic attractors on any non-trivial interval apart from trivial period-doubling cascades. In contrast, piecewise-smooth systems such as the tent map, coupled skew-tent maps, the Lozi map, and the border-collision normal form can possess Hausdorff-continuity domains within regions of robust chaos (Glendinning et al., 2019). For semilinear parabolic PDEs on $X$6-perturbed domains, global attractors have been shown to be continuous at the perturbation parameter $X$7 after pullback to a fixed reference domain and analysis via sectorial operators, Lyapunov functionals, and hyperbolicity assumptions on equilibria (Barbosa et al., 2016, Pereira et al., 2018).

This distinction clarifies a common misconception. A ring attractor in a neural field is a continuous attractor because it is a neutral manifold in state space. A chaotic or global attractor that varies continuously in the Hausdorff metric under parameter change is continuous in a different sense. The two notions can coexist, but neither implies the other.