Behavior-dLDS: A decomposed linear dynamical systems model for neural activity partially constrained by behavior

Abstract: Brain-wide recordings of large-scale networks of neurons now provide an unprecedented view into how the brain drives behavior. However, brain activity contains both information directly related to behavior as well as the potential for many internal computations. Moreover, observable behavior is executed not only by the brain, but also by the spinal cord and peripheral nervous system. Behavior is a coarse-grained product of neural activity, and we thus take the view that it can be best represented by lower-dimensional latent neural dynamics. Capturing this indirect relationship while disambiguating behavior-generating networks from internal computations running in parallel requires new modeling approaches that can embody the parallel and distributed nature of large-scale neural populations. We thus present behavior-decomposed linear dynamical systems (b-dLDS) to disentangle simultaneously recorded subsystems and identify how the latent neural subsystems relate to behavior. We demonstrate the ability of b-dLDS to decouple behavioral vs. internal computations on controlled, simulated data, showing improvements over a state-of-the-art model that uses behavior to supervise all dynamics based on behavior. We then show that b-dLDS can further scale up to tens of thousands of neurons by applying our model to large-scale recording of a zebrafish hindbrain during the complex positional homeostasis behavior, wherein b-dLDS highlights behavior-related dynamic connectivity networks.

• The paper introduces behavior-dLDS, a decomposed linear dynamical systems model that sparsely links selected dynamics operators to behavior while leaving other operators free to capture internal neural processes.
• The paper shows that behavior-dLDS identifies behavior-generating subsystems and achieves lower dynamics reconstruction error than CLDS across all tested simulation settings, while avoiding CLDS’s more-than-1 TB memory requirement.
• Applied to calcium imaging from 13,000 zebrafish hindbrain neurons, the model achieved an overall reconstruction R² of 0.90 and produced behavior-related connectivity maps consistent with known self-location circuitry.

Motivation and problem statement

Brain-wide recordings now capture tens of thousands of neurons simultaneously, but such recordings contain a mixture of behavior-generating activity and internal computations running in parallel. Existing behavior-aware dynamical systems models occupy two extremes: fully supervised approaches such as dynamic causal modeling (DCM) and conditionally linear dynamical systems (CLDS), in which dynamics coefficients are entirely functions of external variables, and fully unsupervised decomposed linear dynamical systems (dLDS), which learn a dictionary of dynamics operators (DOs) with no reference to behavior. The paper introduces behavior-dLDS (b-dLDS) (2603.05612), a middle ground that ties only a subset of DOs to observed behavior while leaving the remainder free to model internal processes.

Model formulation

b-dLDS retains the dLDS generative structure: observations $\bm{y}_t = \bm{D}\bm{x}_t + \epsilon_y$ with latent states evolving as $\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$, where $\bm{F}_t = \sum_m \bm{f}_m c_{mt}$ is expanded over $M$ learned DOs weighted by time-varying dynamics coefficients. The key addition is a behavioral generative channel,

$\bm{b}_t = \bm{\Psi}\bm{c}_t + \bm{\epsilon}_b,$

mapping dynamics coefficients—not instantaneous latent states—to behavior. This choice reflects the authors' position that behavior is a coarse-grained product of which neural processes are active, rather than of moment-by-moment firing. Because only a subset $\Gamma \subset \{1,...,M\}$ of coefficients is assumed to generate behavior, the unobserved-regressor problem is handled with an $\ell_1$ (Laplacian) prior on $\bm{c}_t$, encouraging sparse activation so that behavior-tied and untied DOs are disambiguated.

Inference

Inference proceeds via expectation-maximization with a dirac-delta posterior approximation. The E-step uses basis pursuit denoising with dynamic filtering (BPDN-DF), jointly estimating $\bm{x}_t$ and $\bm{c}_t$ with terms for observation fit, dynamics consistency, sparsity of both states and coefficients, temporal smoothness of coefficients, and—new to b-dLDS—a behavior reconstruction term $\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$0. The M-step takes stochastic projected gradient steps on $\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$1, each $\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$2, and $\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$3, with optional constraint projections. Two practical refinements for $\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$4 are introduced for regimes where many behaviors are generated by few coefficients: a step size rescaled by the norm of the dynamics coefficients ("sparse suspected"), and a Frobenius-norm penalty that drives negligible columns of $\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$5 to zero. These options are used in simulations but deliberately disabled for neural data, where many coefficients are expected to contribute to low-dimensional behavior.

Simulation results

The authors evaluate recovery on a two-independent-systems simulation (following the dLDS benchmark), with 10 simulated behavior traces generated from ground-truth dynamics coefficients via a known $\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$6, under four conditions: one coefficient driving all behaviors; two overlapping coefficients; all six coefficients contributing; and one coefficient with added Gaussian noise ($\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$7 scaling). In the sparse case, the learned $\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$8 has exactly one nonzero column that perfectly matches the correct ground-truth column—the one tied to the first dynamics coefficient—demonstrating that b-dLDS identifies which subsystem generates behavior.

The comparison against CLDS is the strongest quantitative claim in the paper: across simulations varying the number of nonzero $\bm{x}_t = \bm{F}_t\bm{x}_{t-1} + \epsilon_x$9 columns from 1 to 6 (10 random seeds each), b-dLDS achieved lower relative MSE on reconstructed dynamics than CLDS in every setting, because CLDS's strong behavioral conditioning forces its learned dynamics to follow the single observed behavior rather than intrinsic dynamics. Notably, CLDS required more than 1 TB of RAM when run with 10 behavior traces, whereas b-dLDS ran without issue—an implication that strongly behavior-conditioned models are both less accurate and less scalable here. PSID was also tested: because it shares a single latent state between neural activity and behavior, it correctly detected switch times but incorrectly predicted oscillatory behavior outputs instead of smooth on-off states, illustrating the failure mode of tying behavior directly to the latent state.

A caveat applies to these simulations: they are simplified by construction ($\bm{F}_t = \sum_m \bm{f}_m c_{mt}$0, i.e., latent states are observed, block-diagonal independent systems, at most two active operators). Recovery guarantees under realistic observation matrices or overlapping subsystems are not established.

Application to zebrafish hindbrain

b-dLDS was applied to light-sheet calcium imaging of ~13,000 neurons (13,098 traces, 4,282 time points after preprocessing) from larval zebrafish hindbrain during positional homeostasis [Yang et al., Cell 2022], using a one-dimensional motor signal as behavior. The full model achieved overall reconstruction $\bm{F}_t = \sum_m \bm{f}_m c_{mt}$1 (and $\bm{F}_t = \sum_m \bm{f}_m c_{mt}$2 in the four-behavior variant including three trial types). Dynamic connectivity maps, computed as $\bm{F}_t = \sum_m \bm{f}_m c_{mt}$3 under isotropy and zero-mean assumptions on the latent states, were compared for DOs with the strongest versus weakest $\bm{F}_t = \sum_m \bm{f}_m c_{mt}$4 coefficients.

Several findings emerge. The strongest behavior-related coefficients either persisted throughout the session correlated with motor activity or partitioned the recording into early and late epochs, suggesting nonstationarity. Trial-type-related coefficients split the recording temporally, consistent with evolving internal state. The connectivity maps for strongly behavior-tied DOs implicated hindbrain layers containing the self-location encoding medulla oblongata neurons (SLO-MO) identified by Yang et al., corroborating the model against prior biological findings—one DO exhibited hindbrain-wide connectivity correlated with behavior, while others acted locally in space and time. Weakly tied DOs shared nodes with strongly tied ones but showed distinct, sharp, sparse episodes of recurrent connectivity along lateral and anterior-posterior axes. An open question the paper leaves is whether these weakly-tied sharp transients correspond to specific internal computations, e.g., displacement-response computations, which the authors flag as a target for future analysis comparing internal computations to active swimming.

Limitations

The authors are explicit about several limitations. First, regularization can bias inference: extreme settings of the behavior-reconstruction or sparsity parameters can push the model toward too many or too few behavior-tied coefficients, though moderate settings allow the two biases to counterbalance. Second, scalability claims rest on avoiding high-dimensional covariance inversions; CLDS and PSID both failed on the zebrafish-scale data due to memory constraints, but the paper does not provide controlled runtime benchmarks beyond this. Third, the dynamic connectivity formula relies on assumptions of isotropic, zero-mean latent states; corrections exist but are not exercised in the presented analyses. Fourth, the zebrafish result is a single fish and single session, so robustness of the identified connectivity across animals remains unverified. Finally, whether the sparse-behavior assumption underlying the simulation design generalizes to behaviors with dense distributed neural correlates is not tested.

Conclusion

b-dLDS extends the dLDS family with a regularized mapping from dynamics coefficients to behavior, occupying a principled midpoint between fully supervised (CLDS, DCM) and fully unsupervised (dLDS) decomposed dynamics models. Simulations show it recovers the correct behavior-generating subsystem and outperforms CLDS on dynamics reconstruction across all tested configurations, while its computational efficiency enables modeling of ~13,000-neuron zebrafish recordings at $\bm{F}_t = \sum_m \bm{f}_m c_{mt}$5, yielding behavior-aligned dynamic connectivity maps consistent with known SLO-MO circuitry. The framework's modularity suggests extensions to hierarchical relationships between sensory inputs and latent components, though validation across animals and denser behavior regimes remains open.