Heterochronous-Speed RNN: Multi-Timescale Models

• Heterochronous-Speed RNNs are recurrent neural networks that explicitly model multiple timescales through modular state decomposition and adaptive computation.
• They employ techniques like subsampling, bandpass filtering, and dynamic gating to segregate and refine temporal signals for both short-term and long-range dependencies.
• Empirical evaluations indicate that these models outperform standard RNNs in tasks such as speech recognition, time series forecasting, and language modeling by efficiently processing diverse temporal dynamics.

A heterochronous-speed recurrent neural network (HS-RNN) refers, in the broad technical literature, to any recurrent neural network architecture that explicitly models, exploits, or adapts to multiple characteristic timescales or computational speeds within its hidden state dynamics. While the term "HS-RNN" itself does not appear as a defined architecture in the principal sources, several closely related classes of RNNs instantiate heterochronous processing via modularization, multi-rate memory, or adaptive-computation-time mechanisms. This article surveys foundational approaches, focusing on structurally modular, multi-timescale, and dynamically elastic recurrent models that enable heterochronous temporal processing.

1. Conceptual Foundation: Heterochronous and Multi-Scale Memory

Heterochronous-speed recurrence is motivated by the observation that real-world sequential signals often exhibit structure at multiple timescales—ranging from short-term fluctuations to long-range temporal dependencies. Standard RNNs, including LSTM and GRU, are intrinsically limited in their ability to capture dependencies spanning disparate time horizons, due to issues such as vanishing/exploding gradients and uniform update frequency across all hidden units. Explicit multi-timescale architectures modularize the state space or computational flow, with distinct modules or partitions updating at different rates, to better encode temporal heterogeneity (Carta et al., 2020, Bianchi et al., 2017, Park et al., 2017).

2. Structural Approaches: Modular Memory and Multi-Scale Updates

A prominent heterochronous design pattern is the decomposition of the hidden state into multiple modules, each responsible for processing information at a different temporal resolution or frequency. For instance:

• Multi-Scale Linear Memory Network (MS-LMN) partitions the hidden state into modules, each "subsampling" the activations at a different frequency. Modules updating at slower rates inherit, process, and refine representations from faster modules, supporting progressive extension of memory capacity. New modules can be iteratively added to capture progressively longer dependencies, with each module initialized by the subsampled activations of the preceding module (Carta et al., 2020).
• Temporal Overdrive RNN (TORNN) organizes hidden units into $K$ groups, each with $N$ neurons. Connections exhibit block-sparse structure: strong intra-group connectivity and weak inter-group coupling, with each group equipped with a learnable bandpass temporal filter that selects for a specific timescale (Bianchi et al., 2017). Only bandpass filter parameters and the readout weights are trained, enabling efficient multi-timescale adaptation.

Model	Modularization	Timescale Specialization
MS-LMN (Carta et al., 2020)	Hidden state, modules	Subsampling at increasing periods
TORNN (Bianchi et al., 2017)	Neuron groups	Bandpass filtering per group

Such modularization enables the network to selectively propagate and process features at appropriate temporal resolutions, thereby overcoming the limitations of homochronous (single-rate) recurrence.

3. Parameterization of Timescale Adaptation

Heterochronous-speed architectures typically parameterize timescale adaptation through either discrete subsampling ratios, learnable filter coefficients, or adaptive control mechanisms:

• MS-LMN uses explicit subsampling operators associated with each module, with frequency ratios increasing geometrically as modules are added. Each module's recurrence is trained only on the subsampled sequence, thereby focusing its memory on lower-frequency (longer-term) components (Carta et al., 2020).
• TORNN parameterizes each group's temporal selectivity with two cutoff parameters $(\gamma_{1,k}, \gamma_{2,k})$, one for the high-cut lowpass stage and one for the low-cut stage of the group-specific bandpass filter. These are optimized during training, subject to constraints (via sigmoid nonlinearity) to ensure they remain in $[0,1]$ (Bianchi et al., 2017).
• Early Improving Recurrent Elastic Highway Network (EI-REHN) adopts an adaptive computation depth approach, in which, for each time step, the number of micro–time-steps (intermediate recurrence layers) is dynamically determined via an elastic gate—an exponentially decreasing function of the previous state and current input. This "heterochronous dispatch" allows the network to process rapidly evolving sequences with shallow updates and allocate greater depth for challenging input-state combinations (Park et al., 2017).

4. Training Algorithms and Incremental Architectures

Incremental or adaptive training procedures are central to heterochronous-speed models:

• MS-LMN is trained incrementally: initially, a single module is trained on the raw sequence; when performance plateaus, a new module is added, trained on the subsampled output of the prior module, and the process repeats (Carta et al., 2020). Each new module thus specializes in capturing dependencies at a longer timescale.
• TORNN trains only the $2K$ cutoff parameters and the final output weights using backpropagation-through-time (BPTT), with fixed recurrent and input matrices. The separation of learned parameters sharply reduces training complexity and enhances stability when targeting multi-timescale tasks (Bianchi et al., 2017).
• EI-REHN applies truncated BPTT for parameter optimization, with a learnable hypernetwork generating intermediate weight updates for each dynamically chosen micro–time-step, thus supporting variable recurrence depth and richer representations (Park et al., 2017).

5. Empirical Results and Comparative Performance

Experimental evaluations demonstrate that heterochronous-speed RNNs can substantially outperform conventional RNNs, LSTM, GRU, and echo state networks in tasks characterized by multi-timescale dependencies:

• MS-LMN reports improved ability to capture long-term dependencies in speech recognition and handwritten character datasets, using modular architecture and incremental training (Carta et al., 2020).
• TORNN achieves the lowest normalized RMSE (NRMSE) in synthetic multi-sine time series forecasting as the number of spectral components increases. For $K=7$ timescales, TORNN obtains NRMSE=0.045 (clean) versus 0.413 (RNN), 0.164 (LSTM), and 0.084 (ESN); on noisy data, TORNN achieves NRMSE=0.297 compared to LSTM (0.407) (Bianchi et al., 2017).
• EI-REHN delivers higher accuracy and lower MSE with fewer parameters than RHN, LSTM, or RNN on synthetic regression, human activity recognition (best accuracy 92.48%), and Penn Treebank language modeling (test perplexity 66.2, outperforming LSTM and matching RHN) (Park et al., 2017).

6. Significance, Limitations, and Links to Broader Research

Heterochronous-speed RNNs establish foundational directions for modeling complex temporal dynamics in sequential data where information at diverse timescales must be efficiently integrated. Modular multi-rate networks and adaptive computation depth yield both expressive power and parameter efficiency, particularly under severe capacity or training budget constraints.

A plausible implication is that such designs may be extended or combined with transformer and attention-based models for further improvements in handling multimodal, irregular, or long-horizon sequences. A potential limitation is the requirement for careful selection of module parameters, frequency ratios, or appropriate gate decay profiles to best match the underlying dynamics of the data.

The development of heterochronous RNNs is interlinked with research on adaptive computation time, clockwork RNNs, bandpass filtering in neural sequence models, and neural ODEs, reflecting the field’s broadening interest in architectures that natively encode, rather than merely learn, temporal structure (Carta et al., 2020, Bianchi et al., 2017, Park et al., 2017).