Multi-Resolution Time Domain Encoder

• Multi-resolution time domain encoding is a strategy that decomposes time-series data into hierarchical representations across diverse temporal scales.
• These architectures employ parallel and hierarchical pathways to capture global, local, transient, and periodic components from real-world signals.
• They enhance tasks such as forecasting, anomaly detection, and speech enhancement while addressing challenges like computational overhead and fusion complexity.

A multi-resolution time domain encoder is an architectural strategy in time-series and dynamical systems modeling in which latent, intermediate, or output representations are constructed at multiple temporal scales. By explicit design, such encoders utilize features extracted or synthesized at various temporal resolutions (often via parallel or hierarchical pathways), allowing a given system to robustly capture global, local, fast, slow, transient, and stationary components present in real-world time domain signals. This approach is particularly effective in applications where conventional single-resolution representations are insufficient for capturing multi-scale variability, periodicities, or concise information from limited data.

1. Foundations and Methodological Principles

The theoretical foundation of multi-resolution time domain encoding draws from signal processing (wavelet decompositions, multi-resolution STFT), dynamical systems (DMD), and advances in deep representation learning. The goal is to decompose time series data into multiple, hierarchically organized representation streams, each specializing in a particular frequency band or temporal granularity.

Mathematically, a key representative is the multi-resolution dynamic mode decomposition (mrDMD), in which the state vector $x(t)$ of a system is expanded as:

$$x_{\mathrm{mrDMD}}(t) = \sum_{\ell=1}^L \sum_{j=1}^{2^{\ell-1}} \sum_{k=1}^{m_\ell} f_{(\ell,j)}(t) b_k^{(\ell,j)} \psi_k^{(\ell,j)} \exp(\omega_k^{(\ell,j)} t)$$

where each level $\ell$ covers a distinct temporal bin $j$, and slow background modes are recursively sifted out based on their temporal frequency $\omega_k^{(\ell,j)}$ (Kutz et al., 2015).

In neural architectures, this principle is instantiated via:

• Parallel encoders or filterbanks with diverse kernel lengths/strides (Grais et al., 2018, Myoung et al., 24 Sep 2025, Liu et al., 2020)
• Hierarchical decomposition with downsampling/upsampling stages (Singhania et al., 2021, Song et al., 2017)
• Branching transformers with adaptive patch lengths or salient period detection (Du et al., 2023, Zhang et al., 2023)
• Recursive or staged windowing for stepwise extraction/removal of slower modes (Kutz et al., 2015, Choi et al., 2022)

2. Deep Learning Realizations of Multi-Resolution Time Domain Encoding

Convolutional and Auto-Encoder Architectures

Multi-resolution convolutional auto-encoders (MRCAEs and MrCAEs) stack filterbanks of varying sizes per layer, yielding representations at different scales analogous to time-domain wavelets (Grais et al., 2018, Liu et al., 2020). The encoder in these models consists of multiple convolutional branches or filtergroups, each with a distinct kernel size $a_{ij}$. This allows the network to concurrently extract short- and long-duration motifs, facilitating successful source separation, denoising, or pattern recognition. MrCAE, in particular, employs a progressive, hierarchical training paradigm where lower-resolution weights are transferred and refined as the network is widened and deepened across higher resolutions (Liu et al., 2020).

Recurrent and LSTM-Based Designs

Multi-resolution LSTM networks for neural activity video prediction introduce either explicit temporal pyramids (via downsampling and interpolation of input sequences at two or more scales), or multi-resolution layers with dilated (skip) connections (Song et al., 2017). These designs are essential for combating the vanishing gradient problem in long-horizon prediction and excel at modeling both fine detail and extended temporal dependencies. The multi-resolution encoders in these LSTM architectures enable coherent forecasts several frames ahead, which is critical in real-time interventional settings such as seizure prevention.

Transformer-Based Adaptive Patch Encoders

Recent time-series forecasting models, including Multi-resolution Time-Series Transformer (MTST) and MultiResFormer, employ multi-branch transformer architectures where each branch operates on patches of differing size determined either heuristically (Zhang et al., 2023) or adaptively via explicit periodicity/frequency analysis (Du et al., 2023). The salient periodicities are detected using FFT, and patch sizes are selected so that each branch specializes in distinct frequency bands. The resultant representations, possibly enhanced by relative positional embedding, are then fused via weighted summation—often with weights derived from the salience of their associated frequencies.

3. Integration Strategies and Practical Implementations

Parallel Feature Integration

Systems such as short-segment speaker verification architectures (Myoung et al., 24 Sep 2025) combine multi-resolution encoder (MRE) outputs (where SREs operate at window shifts as fine as 25, 50, 100, 200 samples, matching 1.56–12.5 ms at 16 kHz) with pre-trained model (PTM) features and FBank representations. The MRE features are injected via adapters into backbone architectures (e.g., ECAPA-TDNN), modifying hidden representations using affine transforms conditioned on Z_cond from the MRE. PTM layerwise outputs are fused using learnable weights to optimize their complementarity with finer, data-driven features.

Hierarchical and Ensemble Decoding

Decoders are also designed to mirror the multi-resolution nature of the encoders, as seen in multi-resolution temporal convolutional networks (Singhania et al., 2021) and multi-resolution ensemble recurrent auto-encoders (RAE-MEPC) (Choi et al., 2022). Here, reconstruction or prediction is performed via sub-decoders corresponding to each encoding resolution, and their outputs are aggregated—sometimes with explicit coarse-to-fine fusion, or by leveraging weighted averaging derived from feature salience.

Loss Functions and Supervision Across Scales

Multi-resolution schemes often require tailored loss functions. For example, in multi-resolution speech enhancement (Shi et al., 2023), spectrogram losses are computed on stationary features derived from 8, 16, and 32 ms windows at various decoder outputs; these are combined with the waveform L₁ loss to promote global fidelity and resolution-specific enhancements. Similarly, multi-scale SI-SDR objectives are used in speaker extraction (Xu et al., 2020).

4. Comparative Advantages and Empirical Outcomes

Multi-resolution time domain encoders offer marked leverage over single-scale counterparts:

• Superior separation and identification of background/foreground or transient/slow components (e.g., sifting out El Niño events in ocean temperature data, or separating objects in video with different velocities via mrDMD (Kutz et al., 2015)).
• Lower error rates and more robust representations, especially in settings with limited or noisy data, as in short-segment speaker verification where fine resolution becomes essential (Myoung et al., 24 Sep 2025).
• Improved long-term forecasting and anomaly detection through parallel or hierarchical structures in transformers and auto-encoders (Zhang et al., 2023, Du et al., 2023, Choi et al., 2022).
• Reduced over-segmentation and improved temporal coherency in temporal action segmentation, as the ensembling across resolution levels smooths out spurious transitions (Singhania et al., 2021).

Empirical studies consistently show that systems leveraging multi-resolution time domain encoders achieve state-of-the-art or near state-of-the-art metrics:

• Word Error Rates (WER) reduced by 18–32% in MEMR end-to-end ASR (Li et al., 2018)
• Relative SDR, SI-SDR, and PESQ improvements of over 37% for speaker extraction (Xu et al., 2020)
• Up to 0.14 PESQ improvement for speech enhancement with multi-resolution frequency encoders (Shi et al., 2023)
• Lower Mean Squared Error (MSE) and Mean Absolute Error (MAE) against patch-based and CNN baselines for long-term forecasting (Zhang et al., 2023, Du et al., 2023)

5. Limitations, Open Problems, and Future Prospects

Despite their empirical success, multi-resolution time domain encoders present several technical challenges:

• Model complexity and computational overhead: As the number of branches or encoded resolutions increases, parameter count and latency may become significant for high-throughput or resource-constrained deployments.
• Hyperparameter selection: The choice of extraction windows, stride, and the number of scales directly influences model performance and may require problem-specific tuning.
• Fusion and aggregation strategies: Determining the optimal integration of multi-scale features, both at the architecture and supervision level, remains an open research area.
• Boundary artifacts: Recursive windowing and hard indicator functions can introduce discontinuities or artificial oscillations at segment boundaries (Kutz et al., 2015).
• Limited interpretability in deep encoders: While some approaches (e.g., multi-domain symbolic representation models (Nguyen et al., 2020)) are readily interpretable, most deep architectures render learned features opaque.

Future directions include:

• Smoother sifting functions and adaptive kernels to alleviate boundary issues and improve scale selectivity;
• Self-supervised learning of resolution choices and dynamic adaptation based on input signal characteristics;
• Cross-domain application to non-speech, non-audio time series, including financial data, sensor networks, and medical recordings, where multi-path or hierarchical temporal processes are prevalent.

6. Representative Applications

Domain	Encoder Approach	Application Example
Dynamical Systems	mrDMD (recursive DMD + time binning)	Ocean temperature, video background separation
Bio-Signal Prediction	Multi-scale LSTM, ConvLSTM	Long-term neural video prediction/inference
Audio/Speech Processing	Parallel CNN, TCN, multi-resolution LSTM	Speech separation, enhancement, and SV
Time-Series Forecasting	Multi-branch transformer, adaptive patch	Long-term prediction in finance, energy, health
Anomaly Detection	Multi-resolution recurrent autoencoder	Industrial/enterprise multivariate anomaly flag

Multi-resolution time domain encoders have thus become a foundational tool for a broad spectrum of temporal modeling tasks, integrating techniques from dynamical systems theory, signal processing, and deep learning to overcome the inherent multiscale challenge in complex real-world data.