Multi-Rate Mixture of Experts for Accelerating Liquid Neural Network Training

Abstract: Multivariate time-series data often exhibit complex temporal dependencies, irregular sampling, and heterogeneous dynamics across multiple time scales, making accurate sequence modeling particularly challenging. Traditional recurrent neural networks (RNNs), such as Long Short-Term Memory (LSTM) networks, operate in discrete time and may struggle to effectively capture continuous and irregular temporal behaviors. Liquid Neural Networks (LNNs) address some of these limitations through continuous-time dynamics, but standard LNN architectures typically rely on a single dynamical system, limiting their ability to model heterogeneous temporal patterns. To address these challenges, we propose a Multi-Rate Mixture-of-Experts (MR-MoE) framework built on top of Liquid Neural Networks. In the proposed architecture, multiple LNN-based experts operate at distinct time scales, enabling the model to explicitly separate fast-changing dynamics from slow-evolving temporal trends. A gating network further enables adaptive expert specialization based on input conditions. In addition, we incorporate both feature-level and temporal attention mechanisms to improve robustness, interpretability, and long-range dependency modeling. Feature-level attention suppresses noisy or irrelevant variables, while temporal attention selectively focuses on informative historical states. We evaluate the proposed framework on a complex multivariate time-series prediction task and compare it against strong baselines, including LSTM, monolithic LNN, and standard MoE models. Experimental results demonstrate that the proposed MR-MoE framework consistently achieves improved AUROC and AUPRC performance while maintaining favorable computational efficiency. These results highlight the effectiveness of combining continuous-time dynamics, multi-scale expert decomposition, and adaptive attention mechanisms for time-series modeling.

• The paper introduces a Multi-Rate Mixture-of-Experts architecture that combines Liquid Neural Network experts with distinct time constants, gating, feature attention, and temporal attention to model fast and slow clinical dynamics.
• The method improves reported sepsis prediction from approximately 0.55 AUROC and 0.32 AUPRC for a monolithic LNN to 0.65–0.68 AUROC and 0.45 AUPRC with attention, while showing greater robustness to input noise.
• The paper uses quasi-steady-state approximations for fast experts to reduce memory demands, but its efficiency claims lack wall-clock training measurements and require validation across datasets, learnable time constants, and statistical testing.

Motivation and problem statement

Multivariate clinical time series exhibit irregular sampling, missing values, and heterogeneous dynamics that evolve on multiple time scales. Discrete-time recurrent models such as LSTM operate on a fixed step size and struggle to represent continuous, irregular temporal behavior, while standard Liquid Neural Networks (LNNs) — which model the hidden state via a continuous dynamical system $\frac{dx(t)}{dt} = f(x(t), u(t); \theta)$ with an input-dependent time constant $\tau$ — rely on a single dynamical system, limiting their capacity to capture processes that evolve at distinct rates. This paper, "Multi-Rate Mixture of Experts for Accelerating Liquid Neural Network Training" (2606.12240), proposes a Multi-Rate Mixture-of-Experts (MR-MoE) framework that decomposes the continuous-time model into several LNN experts, each assigned a distinct time constant, combined through a learned gating network and augmented with feature-level and temporal attention.

Architecture

The framework is constructed incrementally from four components. First, a gating network computes expert weights $\pi(t) = \mathrm{softmax}(g(z(t); \phi))$, and the prediction is a gated sum of expert outputs $y(t) = \sum_k \pi_k(t) y_k(t)$. Second, the multi-rate structure enforces an ordering of expert time constants, $\tau_1 \ll \tau_2 \ll \cdots \ll \tau_K$, so that fast experts capture short-term fluctuations and slow experts capture long-term trends. Drawing on singular perturbation theory (2606.12240), fast experts are approximated by quasi-steady-state mappings of the slow-scale hidden state rather than by full numerical integration of their dynamics — this is the mechanism the authors credit with reducing computational overhead relative to a monolithic LNN. Third, feature-level attention computes softmax-normalized weights over input variables and reweights the input $\tilde{u}(t) = \beta(t) \odot u(t)$, acting as soft feature selection that suppresses noisy clinical variables. Fourth, per-expert dot-product temporal attention aggregates historical hidden states into a context vector, which the readout maps to the expert output. The authors claim that their architecture is the only one combining MoE-based expert decomposition with both attention mechanisms simultaneously; this claim is asserted rather than substantiated by a systematic comparison against prior hybrid architectures.

Experimental results

The model is evaluated on sepsis onset prediction from ICU physiological time series, using AUROC and AUPRC as metrics appropriate for class-imbalanced clinical tasks. All models use a hidden dimension of 1500, $K=3$ experts for MoE variants, Adam with learning rate $10^{-3}$, and identical training settings across baselines. The reported results show a consistent monotonic improvement across the architectural progression:

Model	AUROC	AUPRC
LSTM	~0.53	~0.22
Monolithic LNN	~0.55	~0.32
MoE (LNN experts)	~0.58	~0.36
Multi-Rate MoE	~0.61	~0.42
Multi-Rate MoE + Attention	~0.65–0.68	~0.45

Two observations are notable. The LSTM baseline performs near chance on AUROC (~0.53), a weak result for a widely used clinical baseline that may indicate suboptimal tuning rather than an inherent limitation of discrete-time recurrence. The largest single gain comes from the multi-rate decomposition itself (AUPRC rising from ~0.36 to ~0.42), supporting the paper's central claim that explicitly separating time scales reduces interference between fast and slow dynamics. The authors attribute the gains to a bias–variance decomposition argument: expert specialization increases expressiveness (lowering bias) while conditional computation reduces variance.

Efficiency and robustness

On memory consumption, the paper reports that the LSTM baseline incurs the highest cost, the monolithic LNN is substantially more compact, and MR-MoE improves further because fast experts bypass full dynamic simulation via quasi-steady-state approximation. Attention adds moderate overhead — temporal attention requires storing historical hidden states, whereas feature-level attention is comparatively cheap and reduces effective input dimensionality. Under injected input noise of increasing standard deviation, MR-MoE and MR-MoE-Attention degrade more slowly than LSTM and monolithic LNN, with the attention variant maintaining the highest AUROC across all noise levels. The authors connect this robustness to the separation of transient noise (absorbed by fast experts) from stable physiological trends (captured by slow experts), and to attention-based suppression of irrelevant variables.

Limitations and open questions

The paper concedes several limitations directly. Statistical significance is not established: no error bars or significance tests are reported, so the margins between adjacent configurations (e.g., 0.65 vs. 0.68 AUROC) cannot be assessed for reliability. Time constants $\tau_k$ are manually specified and fixed during training, which may not reflect the true temporal scales of the data; making them learnable is left as future work. All experts are trained jointly in a single optimization process, which the authors note can introduce interference between time scales with different convergence behavior; decoupled or hierarchical training schemes remain unexplored. Evaluation is limited to a single sepsis prediction task, so generalization to other time-series domains is unverified. The efficiency claim in the title ("accelerating training") is supported only by memory-consumption comparisons, not by wall-clock training-time measurements. Code and data artifacts are not released.

Conclusion

The paper presents a structured integration of continuous-time dynamics, multi-rate expert decomposition via singular perturbation, and dual attention mechanisms, demonstrating monotonic improvements in AUROC and AUPRC on sepsis prediction, with favorable memory usage and improved robustness to input noise. The evidence for the multi-scale decomposition's contribution is the strongest part of the empirical case, while the attention gains, the absence of statistical testing, the fixed time constants, and the single-task evaluation leave the generality and reliability of the approach open to further validation.