ParallelTime Architecture: Adaptive Forecasting

• ParallelTime architecture is a neural framework that adaptively balances short- and long-term dependencies in multivariate time series forecasting.
• It integrates localized attention and efficient state-space modeling via a dynamic, token-specific weighting mechanism.
• Empirical evaluations show reduced computation and improved accuracy, setting new benchmarks for long-range forecasting performance.

The ParallelTime architecture is a neural framework developed for advancing long-range multivariate time series forecasting. It introduces a dual-path design combining localized attention mechanisms and efficient state-space modeling, augmented by a dynamic weighting scheme—the ParallelTime Weighter—that adaptively balances the influence of short-term and long-term temporal dependencies for each token in the input. This design is motivated by evidence that equally weighting short- and long-term dependencies, as done in prior approaches, is suboptimal for time series prediction (Katav et al., 18 Jul 2025).

1. Architectural Composition and Motivation

ParallelTime operates by first segmenting the input multivariate time series into non-overlapping “patches” or tokens. Each patch passes through two parallel processing modules:

• Local Windowed Attention Branch: Implements a causal multi-head windowed self-attention over a small window size, directly modeling short-term dependencies and localized temporal correlations typical in time series.
• Mamba Branch: Utilizes the Mamba block, a state-space modeling approach that can efficiently capture long-term dependencies while maintaining constant memory usage. The Mamba block processes sequences via state-space equations:

$h_t = A h_{t-1} + B x_t, \qquad y_t = C h_t$

where state and output are updated recurrently.

To enhance global representational capacity, global register tokens are concatenated to the input—these act as persistent memory, encoding domain-specific context across prediction windows.

The rationale behind this bifurcated design is to exploit the complementary strengths of localized attention for recent, short-term patterns and state-space modeling for persistent, long-term structures—both essential in multivariate time series, where the dynamics can exhibit regime switching and multi-scale behavior.

2. ParallelTime Weighter: Dynamic Balancing Mechanism

Central to ParallelTime is the “ParallelTime Weighter.” Rather than simply using a fixed or averaged blending of the two branches, the Weighter computes adaptive, token-specific weights that interdependently modulate the contributions from attention and Mamba modules. The process consists of:

1. Normalization: The raw outputs from both branches, $x_{\text{att}}$ and $x_{\text{mamba}}$, are first normalized via RMSNorm to standardize scale.
2. Dimensionality Compression: Each normalized vector is linearly projected from dimension $\text{dim}$ to $\sqrt{\text{dim}}$ via branch-specific matrices, producing $x_{\text{att}}'$ and $x_{\text{mamba}}'$.
3. Concatenation: The compressed representations are concatenated per patch:

$$x_{\text{cat}}' = \text{Concat}(x_{\text{att}}', x_{\text{mamba}}')$$

yielding a feature of shape $P \times 2\sqrt{\text{dim}}$, with $P$ the number of patches.

1. Nonlinear Transformation and Weight Calculation: A two-stage non-linear transformation is applied:

$$x_{\text{weights}} = \sigma \left( \operatorname{ReLU}(x_{\text{cat}}' W_1) W_2 \right)$$

where $W_1 \in \mathbb{R}^{2\sqrt{\text{dim}} \times d_h}$, $W_2 \in \mathbb{R}^{d_h \times 2}$, and $\sigma$ is the sigmoid function. $d_h$ is a hidden dimension greater than $2\sqrt{\text{dim}}$.

1. Splitting and Output Calculation: The resulting weight vector is split into two per-token weights $\mathbf{w}^{\text{att}}$, $\mathbf{w}^{\text{mamba}}$ and the output is computed as:

$$x_\text{out} = x_{\text{att}} \cdot \mathbf{w}^{\text{att}} + x_{\text{mamba}} \cdot \mathbf{w}^{\text{mamba}}$$

This dynamic mechanism enables the model to contextually shift reliance between short- and long-term memory per prediction, leveraging both the current input and learned context.

3. Empirical Performance and Efficiency

Extensive benchmarking was performed on several canonical long-range multivariate forecasting datasets, including Weather, Traffic, Electricity, Illness, and multiple ETT datasets, under various horizon lengths. The ParallelTime model consistently achieved state-of-the-art performance as measured by Mean Squared Error (MSE) and Mean Absolute Error (MAE).

Notably, compared to PatchTST (Transformer-based) and strong Mamba baselines:

• Forward and training FLOPs were reduced by 30–40%.
• The total parameter count was decreased.
• The model maintained or improved accuracy, with average MSE reductions of approximately 4–5% relative to strong baselines.
• The architecture proved robust for both short and long horizon forecasting, and scaled effectively to longer sequence lengths.

These results confirm that dynamically weighting the balance of dependency types outperforms previous strategies that statically average or otherwise fix the contribution of each path.

4. Comparative Evaluation and Analysis

ParallelTime was evaluated against a range of contemporary approaches, including:

• Transformer-style architectures (PatchTST)
• State-space models (Mamba)
• Linear models (DLinear)
• Frequency-enhanced models (FEDFormer)
• Prior hybrid approaches (e.g., simple averaging of attention and Mamba outputs).

Tables in the paper indicated that ParallelTime surpassed all baselines on the majority of metrics and benchmarks while achieving improvements in parameter and compute efficiency.

The dynamic weighting distinguishes ParallelTime from prior methods (such as those that assign equal weight to attention and Mamba, as in Hymba). This suggests that adaptivity in dependency blending is key to robust temporal modeling in diverse forecasting contexts.

5. Scalability, Robustness, and Computational Considerations

ParallelTime was explicitly designed for scalability and efficiency:

• Both the attention and Mamba modules operate on patches, enabling parallelism and consistent computational cost as the sequence length increases.
• The Weighter mechanism incurs minimal additional overhead due to its concise non-linear mapping.
• The modularity of the architecture allows it to be scaled to deeper models or larger dimensions if computational budgets allow.

Empirical studies showed that the computational savings did not compromise model robustness. The architecture maintained stability and accuracy at longer horizons and across varying data regimes, partly due to the tokenwise dynamic weighting.

6. Implications, Limitations, and Future Directions

The ParallelTime architecture introduces a flexible, extensible foundation for time series modeling. Potential future extensions include:

• Application of the dynamic weighting principle to tasks beyond forecasting, such as anomaly detection, time series classification, or imputation.
• Scaling the model with increased data or compute resources, possibly by stacking additional layers.
• Adaptation for tasks with richer global context needs, such as through additional global register tokens or domain-specific customization.
• Investigating the integration of additional memory pathways or mechanisms alongside attention and Mamba branches.

A plausible implication is that further research into per-token, data-dependent weighting and hybrid modeling could generalize to other domains with complex temporal and sequential dependencies, including multimodal time series or streaming environments.

7. Summary Table: Branch Comparison in ParallelTime

Pathway	Dependency Type	Key Mechanism	Role in Architecture
Windowed Attn	Short-term, Localized	Causal Multi-Head Windowed Self-Attention	Captures recent patterns
Mamba	Long-term, Global	Efficient State-Space Model (SSM) Operations	Models persistent trends
ParallelTime Weighted Output	Dynamic – input- and token-dependent	Tokenwise non-linear weighting of Attn and Mamba outputs	Blends local/global adaptively

In conclusion, the ParallelTime architecture demonstrates that adaptively balancing short- and long-term temporal dependencies through dynamic, token-specific weighting leads to substantially improved performance and efficiency in multivariate time series forecasting, setting a precedent for future "parallel Attention-Mamba" model research and practical deployment (Katav et al., 18 Jul 2025).