Closed-Form Continuous-Time Neurons

• CfCs are neural network architectures that employ closed-form integration to model continuously evolving hidden states, ensuring efficient and precise updates.
• Their design uses three gating mechanisms—time-constant, proposal, and output—that enable exponential stability and overcome limitations of discrete RNNs and neural-ODEs.
• Applications of CfCs include irregular time-series analysis, healthcare digital twins, and closed-loop control, offering up to 10^1–10^5× speed improvements over neural-ODE methods.

Closed-form Continuous-Time Neurons (CfCs) are a family of neural network architectures that implement continuous-time dynamics with exact, solver-free updates. Originating from the theory of Liquid Time-constant Networks, CfCs reformulate the hidden-state evolution into forms that admit closed-form integration over time intervals. This addresses both the expressivity limitations of discrete-time RNNs and the severe computational bottlenecks of neural-ODE-based models, enabling principled modeling of irregularly sampled and multimodal time-series data with state-of-the-art efficiency, performance, and robustness, notably in domains such as healthcare digital twins and closed-loop control.

1. Mathematical Foundations of CfC Units

The core of CfC design is a per-neuron continuous-time dynamical system: $$\frac{d}{dt}\,z(t) = -\alpha(t)\,\odot\,z(t) + \alpha(t)\,\odot\,\beta(t)$$ where $z(t)\in\mathbb{R}^d$ is the state vector, $\alpha(t)$ the learned (positive) time-constant gate, and $\beta(t)$ a learned target state. Inputs and previous states are concatenated into $\chi(t) = [x(t); z(t)]$, and the gates are defined as: $$\begin{aligned} \alpha(t) &= \mathrm{softplus}(W_\alpha\,\chi(t) + b_\alpha), \ \beta(t) &= \tanh(W_\beta\,\chi(t) + b_\beta), \ \gamma(t) &= \sigma(W_\gamma\,\chi(t) + b_\gamma) \end{aligned}$$ where $W_\alpha,W_\beta,W_\gamma$ are parameter matrices, $b_\alpha,b_\beta,b_\gamma$ are biases.

For each data update at times $t_k$, gates $\alpha_k, \beta_k, \gamma_k$ are evaluated with frozen input over $[t_k, t_{k+1}]$ and the ODE is integrated exactly: $$z(t_{k+1}) = z(t_k)\odot\exp(-\alpha_k \Delta t) + \beta_k\odot[1-\exp(-\alpha_k \Delta t)]$$

$h(t_{k+1}) = \gamma_k\odot z(t_{k+1})$

This closed-form solution is efficient, requiring only matrix multiplies, element-wise non-linearities, and exponentials per step (Nye, 2023).

Variants and alternative parameterizations exist, including the use of smooth sigmoidal gating in place of exponentials to address gradient flow and to interpolate between two candidate hidden-state updates (Hasani et al., 2021), yielding general CfC cell updates of the form: $x_k = z_k \odot u_k + (1-z_k) \odot v_k$ with $z_k = \sigma(-\tau_k\Delta t)$, and $u_k, v_k$ as MLP or affine transformations of state/input.

2. Gating Mechanisms and Update Principles

The three-gate architecture is central:

• Time-constant gate $\alpha$ ensures strictly positive rates, modulating speed of forgetting and stabilizing dynamics.
• Proposal gate $\beta$ sets the attractor toward which the state decays, bounded in $(-1, +1)$ by $\tanh$.
• Output gate $\gamma$ modulates the output exposure with a sigmoid.

The closed form arises because, once $\alpha, \beta$ are fixed over a step, the system is linear in the state, and its evolution can be integrated explicitly over arbitrary step sizes. This is in contrast to generic neural-ODEs where no such analytical solution exists and numerical solvers are required.

At scale, the gates are implemented as lightweight per-coordinate MLPs or affine projections, and explicit time input ($\Delta t$) enables native handling of irregular sampling (Hasani et al., 2021, Nye, 2023).

3. Comparison with Discrete RNNs and Neural-ODE Models

Expressivity and Practicality

• Discrete RNNs (e.g., LSTM, GRU): Evolve with fixed-step recurrence, $z_{k+1}=F(z_k, x_k)$. Continuous-time behavior can only be approximated via small steps or high-depth stacking.
• Neural-ODEs: Define $\dot{z}=f(z,x)$, solved numerically per sample, enabling fully continuous but computationally intensive trajectories.
• CfCs: Model "liquid time-constant" flows (state reversion with adaptive decay toward a target), capturing key dynamical structures with exact integration.

Computational Complexity

Model	Per-step complexity	Solver overhead	Typical speedup (vs ODE-RNN)
Discrete RNN	$O(d^2)$	None	Baseline
Neural-ODE	$>$ $O(d^2)$	High (adaptive solver)	-
CfC	$O(d^2) + O(d)$	None (closed-form)	$10^1$–$10^5\times$ faster

CfCs offer $1$–$5$ orders of magnitude faster training and inference than neural-ODE models—crucial for real-time and large-scale deployments (Nye, 2023, Hasani et al., 2021).

Stability

• RNNs: Prone to gradient vanishing/explosion.
• Neural-ODEs: Suffer from stiffness, leading to potential integration instability.
• CfCs: The explicit damping by $\alpha_i>0$ ensures exponential stability around $\beta_i$, aiding long-horizon credit assignment and robust forward passes.

4. Training, Numerical Stability, and Implementation

Key practices supporting CfC training:

• Losses: Choose per-task (MSE, cross-entropy, survival analysis, etc.).
• Optimization: Adam/RMSProp with modern LR scheduling; apply gradient clipping (e.g., norm $\le1.0$) to handle irregular sampling shocks.
• Handling multimodal time-series: Pre-embed each modality (e.g., small FFNs, 1D-CNNs), concatenate into $x(t)$, and feed into CfC.
• Initialization: Start $W_\alpha$ near zero with $b_\alpha>0$ for moderate initial time constants (avoid rapid decay).
• Efficient computation: Mixed FP16 computation, exploit vectorized exponentials. Layer normalization on preactivations stabilizes optimization.

For stacking, CfC layers propagate hidden states layerwise, and explicit time dependency introduces continuous-depth functionality: $$z^{(\ell+1)}_k = \text{CfC}^{(\ell+1)}(z^{(\ell)}_k, x_k)$$ (Hasani et al., 2021, Nye, 2023).

5. Architectural Extensions: Low-Rank and Sparse Connectivity

Recent work explores structural priors for improved robustness and efficiency in CfCs—particularly parameterizing the recurrent kernel $W_{hh}$ as low-rank plus sparse (Tumma et al., 2023). The formulation: $$W_{hh}(r,s) = (U_r\Sigma_r^{1/2})(\Sigma_r^{1/2}V_r^\top) \odot M_s$$ where $r$ is rank, $s$ sparsity, $M_s$ a random binary mask. Only factors $W_1, W_2$ are updated during training. Theoretical results:

• Spectral radius $\rho(W_{hh})$ and norm $\|W_{hh}\|$ can be tightly controlled by $r, s$.
• Low rank ($r\ll h$) produces vanishing-gradient regimes, reducing temporal attention span—beneficial for short-horizon or robust closed-loop tasks.
• Low parameter count: e.g., for $h=64$ and $r=1$, only $128$ recurrent parameters (compared to $4096$ for full) with equal or better out-of-distribution generalization.

Empirically, CfCs with very low rank ($r=1$ or $5$) and minimal sparsity ($s\le0.2$) outperform standard full-rank RNNs, LSTMs, GRUs, and feed-forward CNNs under distribution shift, yielding more robust and memory-efficient agents (Tumma et al., 2023).

6. Empirical Performance Characteristics

Multiple benchmarks demonstrate CfCs:

• Achieve state-of-the-art AUC, MSE, or classification metrics, matching or exceeding LSTMs, GRUs, ODE-RNNs, NCDEs, typically by $1$–$5\%$ on held-out testing (Nye, 2023, Hasani et al., 2021).
• Enable $10$–$10^5\times$ speedups over neural-ODE approaches, facilitating practical deployment for real-time analytics or embedded control.
• Maintain superior stability with irregular or multimodal data due to built-in exponential decay mechanisms.
• In digital twin frameworks for healthcare, CfCs are used for real-time risk scoring, trajectory forecasting, and simulation of interventions, operating with substantially reduced computational load compared to ODE-based analogues (Nye, 2023).

7. Design Guidelines, Applications, and Limitations

Design guidelines for CfC deployments include:

• For robustness under distribution shift, use low-rank ($r=1/5$) recurrent matrices with minimal sparsity.
• For tasks requiring short- or medium-term memory, the inherent vanishing-gradient bias of CfCs is a feature, not a limitation.
• Rank pruning is preferable to unstructured sparsity, as it better preserves spectral properties crucial to stability and robustness (Tumma et al., 2023).

Key applications: Irregularly sampled time-series analysis, healthcare digital twins, real-time closed-loop control, multimodal sequential tasks. CfCs have also been assembled into mixed-memory and continuous-depth architectures (Hasani et al., 2021, Nye, 2023).

Limitations: CfCs are tailored to "liquid time-constant" dynamics—arbitrary continuous-time flows cannot be modeled directly. For problems requiring highly nonlinear latent dynamics beyond the linear-ODE-with-attractor template, neural-ODEs retain a modeling advantage (at significant computational cost).

---

• "Digital Twins for Patient Care via Knowledge Graphs and Closed-Form Continuous-Time Liquid Neural Networks" (Nye, 2023)
• "Closed-form Continuous-time Neural Models" (Hasani et al., 2021)
• "Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control" (Tumma et al., 2023)