Latent Representation Drift in Neural Systems

• Latent representation drift is the temporal evolution of neural tuning that maintains stable, low-dimensional computations amidst individual variability.
• The framework introduces Latent Processing Units (LPUs) to formalize robust latent variables that govern collective neural dynamics.
• Linear decoding and manifold redundancy ensure stable behavioral outputs despite drifting tuning in high-dimensional neural systems.

Latent representation drift refers to the temporal evolution or variability of internal neural codes in a system—biological or artificial—such that the tuning of individual units (neurons or artificial neurons) changes over time, even as system-level computations and behaviors remain stable. The dynamical systems framework introduced in "Latent computing by biological neural networks: A dynamical systems framework" presents a principled, analytic account of how robust, low-dimensional latent computations can persist stably at the system level despite ongoing representational drift at the cellular or unit level. This framework is centered around the concept of Latent Processing Units (LPUs), which formalize latent variables as organizing centers of collective neural dynamics.

1. Latent Processing Units and the Dynamical Systems Perspective

The core of the framework is the separation between high-dimensional observable neural population activity and low-dimensional latent variables, or LPUs, which encode the essential variables used for cognition and behavior. The relationship is formalized as follows:

$$\kappa(t) = \phi(r(t)),\quad \tau \dot{r}(t) = -r(t) + \varphi(\kappa(t), u(t)).$$

Here,

• $r(t) \in \mathbb{R}^{N_{\mathrm{rec}}}$ denotes the population activity at time $t$,
• $\kappa(t) \in \mathbb{R}^{K}$ is the latent state ($K \ll N_{\mathrm{rec}}$),
• $\phi$ is the encoding map from neural activity to latent state (typically linear),
• $\varphi$ is a, possibly nonlinear, embedding back to neural activity,
• $u(t)$ represents inputs.

This systems view explains how neural population activity (possibly exhibiting complex, high-dimensional, and individually variable behavior) can stably encode and evolve low-dimensional, computationally meaningful latent variables.

2. High-Dimensional Redundancy and Manifold Geometry

A key attribute is that even when the latent computation is low-dimensional, the corresponding manifold traced in neural state space is highly curved and massively redundant. Multiple neural trajectories—or even entire high-dimensional subspaces—can implement the same latent trajectory. Such redundancy arises as a direct consequence of the universal computation capabilities of nonlinear dynamical systems and allows the neural code to be robust against variability in the tuning of individual units.

$r(t) = f( M \kappa(t) )$

where $M$ is a high-dimensional embedding matrix and $f$ is typically nonlinear (e.g., $\sin, \tanh$). Complex, high-dimensional manifolds result. This multiplicity of representations is what enables robust computation even as individual neuron tuning changes.

3. Linear Decoding and Behavioral Readout

The framework asserts that linear readout of population activity is sufficient to extract even complex, nonlinear functions of the underlying latent variables:

$$o(t) = W_{\mathrm{out}} r(t) \approx \psi(\kappa(t))$$

This is possible due to the structure of the neural manifolds, where latent variables are typically embedded linearly, and decoding can focus on projecting onto the relevant subspace. Behavioral and perceptual outputs can thus remain stably controlled by the system despite inner representational changes.

4. Observational Scaling and Timescale Separation

One major implication is that the number of neurons needed to stably decode computations depends on the timescale:

• For instantaneous decoding (e.g., at a single timepoint), sampling thousands of neurons suffices for near-optimal reconstruction of the latent variables and behavioral outputs.
• However, accurately predicting the correct trajectory of neural states—i.e., tracking detailed latent system dynamics—may require access to millions of neurons if the timescale of observation extends to seconds or longer.

This is because the redundancy in the high-dimensional embedding means that short-term decoding is robust to subsampling, but longer-term prediction requires sampling the redundant space more exhaustively.

5. Robustness to Representational Drift

The central result concerning latent representation drift is that the essential system-level computations—i.e., the trajectory of $\kappa(t)$—are robust to ongoing changes (drift) in the tuning of individual neurons. This is mathematically grounded in the framework by distinguishing between:

• Embedding weights ($m_i$): Control how each neuron's activity is related to the latent variables. These may drift with time.
• Encoding directions ($n^{(p)}$): Define the projection from neural activity to latent variables (used for decoding).

If drift in the embedding weights is orthogonal to the encoding directions, then the latent state and thus the output computation remain invariant to arbitrarily large drift in neuronal tuning:

$$\text{If}\quad E[n_i^{(p)} \Delta m_i] = 0,\quad \forall p,\quad \text{then latent computation is preserved.}$$

Consequently, population-level redundancy ensures computation is stable so long as the drift does not disrupt the (typically much smaller) coding subspace.

6. Biological and Empirical Implications

This theoretical result provides an explanation for empirical findings in neuroscience: even though the receptive fields or tuning properties of individual neurons may drift considerably over days or weeks, behavioral outputs (e.g., perception, memory, movement) remain stable. Experimental observations (in hippocampus, sensory cortices, etc.) show that recorded neuron codes change while function persists.

The framework predicts:

• Drift should primarily occur in the “null” space of the decoder, away from the coding directions.
• Large neural populations are needed to ensure drift robustness over long timescales.
• Behavior is selectively sensitive to perturbations in the coding subspace, not in the subspace orthogonal to encoding directions.

7. Summary Table: LPU Framework and Drift

Attribute	Role in Drift Robustness	Mathematical Basis
Low-dim latent computation ($\kappa$)	Core to system-level stability	Encoded by $\phi(r)$
High-dim embedding ($r$), nonlinear manifold	Provides redundancy for drift tolerance	$r(t) = f(M \kappa(t))$
Linear decoding/readout	Sufficient for behavioral outputs	$o(t) = W_{\rm out} r(t)$
Encoding vs embedding weights	Orthogonality prevents drift in embedding from affecting computation	$E[n^{(p)} \Delta m] = 0$
Population size	More neurons = higher drift robustness over time	Error scaling: $\propto N^{-1/2}$

Conclusion

The latent processing unit framework resolves the apparent paradox that robust, stable computation can coexist with ongoing representational drift in high-dimensional neural codes. By embedding critical computational variables in robust population-level latent variables, and maintaining an encoding subspace largely orthogonal to the drift of individual units' tuning, the system achieves stability and flexibility simultaneously. This dynamical perspective provides not only a theoretical foundation for future empirical research in systems neuroscience but also practical guidelines for the design of artificial neural systems that aspire to balance learning, robustness, and adaptability in the presence of inevitable internal change.