Activation Manifolds in Deep Systems

• Activation manifolds are low-dimensional geometric representations of neural pre- or post-activation vectors, elucidating the intrinsic structure of data in deep networks and physical systems.
• They underpin architecture-agnostic transfer mechanisms via frameworks like CAST, which align model behaviors through learned linear projections between source and target manifolds.
• Topology-aware and low-rank techniques applied on activation manifolds enable efficient memory management, improved class separability, and principled statistical modeling in complex systems.

An activation manifold is a low-dimensional geometric structure formed by the set of neuron activation vectors or system states associated with a dynamical, computational, or physical process. In deep learning, an activation manifold at a given layer is the collection of pre- or post-activation vectors as the input varies across its domain; in physical or control systems, it may represent the submanifold of state-space compatible with particular constraints or modes. The concept provides a rigorous foundation for understanding representational geometry, transfer mechanisms across architectures, the topology-aware manipulation of latent spaces, efficient storage and transport of activity, and the mathematical treatment of activation-driven processes in artificial and physical systems.

1. Mathematical Foundations of Activation Manifolds

At each layer $\ell$ of a neural network, let $a_\ell(x)\in \mathbb{R}^{d_\ell}$ be the activation for input $x$. The set

$$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$

defines the layer's activation manifold. Typically, $\dim(M_\ell) \ll d_\ell$, as real-world data manifolds and the transformations performed by neural layers rarely utilize the full ambient space. In applications involving physical systems, such as cardiac electrophysiology, the term also applies to manifolds $\mathcal{S}$ (e.g., heart surfaces) on which dynamic fields (activation times, local conduction velocities) are defined and interpolated via Gaussian processes (Pezzuto et al., 2022, Coveney et al., 2020).

The geometry and topology of $M_\ell$—its dimension, the presence of nontrivial homology (loops, holes), curvature, and alignments—inform both the expressivity of the layer and the design of methods that act upon or exploit such manifolds. In sparse autoencoders and feature analysis, activation manifolds may be subspaces $S_i$ of dimension $d_i$ whose direct sums (modulo sparsity) give the complete activation vector: $x = \sum_i S_i f_i$, with $a_\ell(x)\in \mathbb{R}^{d_\ell}$0 parameterizing points on a manifold $a_\ell(x)\in \mathbb{R}^{d_\ell}$1 (Michaud et al., 2 Sep 2025).

2. Activation Manifolds in Model Transfer and Interoperability

A principal application of activation manifolds is in architecture-agnostic behavior transfer. The Cartridge Activation Space Transfer (CAST) framework defines direct maps between the activation manifolds $a_\ell(x)\in \mathbb{R}^{d_\ell}$2 and $a_\ell(x)\in \mathbb{R}^{d_\ell}$3 of source and target LLMs in order to translate frozen LoRA behavioral kernels across models of arbitrary structure (Kari, 19 Oct 2025):

• Define $a_\ell(x)\in \mathbb{R}^{d_\ell}$4, $a_\ell(x)\in \mathbb{R}^{d_\ell}$5 as learned (linear) projection heads.
• For $a_\ell(x)\in \mathbb{R}^{d_\ell}$6, compute

$a_\ell(x)\in \mathbb{R}^{d_\ell}$7

where $a_\ell(x)\in \mathbb{R}^{d_\ell}$8 define the LoRA low-rank kernel (frozen).

• The composite transformation $a_\ell(x)\in \mathbb{R}^{d_\ell}$9 aligns task-specific behavior by routing target stream activations through the source model's kernel manifold.
• Training aligns both outputs (via softened KL divergence loss) and hidden states (via MSE), and optionally enforces round-trip self-consistency. CAST achieves $x$0–$x$1 of retrain-from-scratch LoRA adapter performance across diverse architectures and dramatically outperforms weight-space alignment (Kari, 19 Oct 2025).

3. Topology-Aware and Algebraic Manipulation of Activation Manifolds

The topology of the activation manifold at a hidden layer directly affects class separability and downstream linear readout. Classical activations (ReLU, tanh, sigmoid) act homeomorphically or by gluing (collapsing regions), preserving or simplifying topology. Topology-aware activations such as SmoothSplit and ParametricSplit explicitly cut the manifold by introducing learnable "gaps" (discontinuities or strong non-linearities) along specific axes, thus breaking nontrivial cycles (e.g., splitting a circle at $x$2 into arcs) (Snopov et al., 17 Jul 2025). Their effect is particularly pronounced in low-dimensional layers, where such "cuts" can immediately enable linear separability of classes otherwise topologically entangled.

Bilinear autoencoders further provide a direct means of representing and extracting the underlying geometry of activation manifolds: each latent is a closed-form quadratic (or linear) function of $x$3, $x$4. Decoding and mixing these latents reveal explicit quadric surfaces (ellipses, cones, etc.) in activation space, and analyzing their clustering uncovers higher-dimensional manifold structure within network representations (Dooms et al., 19 Oct 2025).

4. Low-Rank and Subspace Structure in Activation Manifolds

Empirical studies consistently show that high-dimensional activations in large models concentrate on low-dimensional linear subspaces—a property leveraged for significant compression, efficient memory management, and improved numerical stability. Both OASIS and LASER frameworks build dynamic, online-tracked low-rank bases $x$5 via streaming PCA (e.g., Oja’s rule, power iteration) (Choudhary et al., 10 Apr 2026, Çakar et al., 19 Apr 2026):

• Activations at time $x$6, $x$7, are projected as $x$8 where $x$9 spans the activation manifold.
• All gradients and optimizer states are similarly compressed and "transported" across subspace updates using transition matrices $$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$0.
• LASER demonstrates that, in recursive models, $$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$1 of activation variance resides within a $$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$2 principal subspace (e.g., $$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$3 for $$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$4), enabling memory reductions exceeding $$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$5 with negligible accuracy impact (Çakar et al., 19 Apr 2026).

This subspace dynamic is not fixed but drifts over time, underscoring the necessity for continual tracking rather than static truncation (Choudhary et al., 10 Apr 2026).

5. Activation Manifolds in Physical and Control Systems

The activation manifold concept generalizes to constrained mechanical systems whose topology varies due to the activation of additional holonomic constraints. When new constraints $$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$6 engage at $$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$7, the activation manifold is the locus $$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$8 in configuration space; in state space, it is the tangent bundle restricted by both pre- and post-activation constraints (Mueller, 21 Apr 2026). At the switching instant, velocity jumps must satisfy both impulsive momentum balance and kinematic compatibility, leading to a linear system for the impulse and velocity jump, exactly enforcing the new constraints and preserving appropriately projected momenta.

Two principal formulations exist:

• Redundant coordinates: Uses projectors $$M_\ell = \{ a_\ell(x) \mid x\in \mathcal{X} \} \subset \mathbb{R}^{d_\ell}$$9 to handle old constraints and imposes the new constraint via a momentum-consistent update.
• Minimal coordinates: Re-expresses dynamics in a local coordinate chart $\dim(M_\ell) \ll d_\ell$0, then imposes the new constraint in this reduced space (Voronets form). Both guarantee that switching events move the system onto the activation manifold associated with the augmented constraint set, in a physically correct and computationally robust fashion (Mueller, 21 Apr 2026).

6. Statistical and Surrogate Modeling of Activation Manifolds

The need to model functions on non-Euclidean or unknown-geometry activation manifolds arises in cardiac electrophysiology and inverse modeling. Gaussian Process (GP) priors defined intrinsically on surface or activation manifolds use Laplace–Beltrami eigenfunctions for efficient reduced-rank expansion, enabling both interpolation and uncertainty quantification in, e.g., the estimation of local activation times (LAT) or conduction velocities (CV) on cardiac surfaces (Pezzuto et al., 2022, Coveney et al., 2020).

Manifold-aware surrogate models are also used in Bayesian optimization, e.g., when identifying the source of ectopic cardiac foci from ECG signals (Pezzuto et al., 2022). Surrogates based on spectral kernels or multi-fidelity co-kriging yield efficient acquisition and rapid convergence, exploiting the intrinsic geometric structure of the activation manifold (e.g., the heart surface) for physically plausible and clinically relevant inference.

7. Empirical Insights and Implications

Layerwise activation manifolds have been analyzed using unsupervised methodologies (NAP, clustering, UMAP embedding). In practice, activations in some domains (e.g., neural network-based radio receivers) do not discretize into separate clusters but rather form a smooth, continuous one-dimensional manifold parametrized by physical factors (e.g., SNR) (Tuononen et al., 21 May 2025). This structure enables the deployment of manifold-based OOD detection, interpretability, and model monitoring.

In sparse autoencoder scaling, the intrinsic dimension and curvature of feature manifolds govern the rate at which new features are discovered as the number of latents increases, producing either benign or pathological scaling regimes depending on the relationship between data complexity and feature sparsity (Michaud et al., 2 Sep 2025).

The significance of the activation manifold concept is thus pervasive: it is central to the architecture-agnostic transfer of learned functions, the manipulation of representation topology, memory- and compute-efficient architecture design, principled statistical modeling on complex geometries, and theoretically grounded analysis of emergent neural phenomena.