DeltaFormer Hybrid Architecture

• DeltaFormer Hybrid is a transformer-based architecture that employs a delegate-token attention mechanism to constrain inter-variable interactions via learnable low-dimensional tokens.
• It uses a three-stage pipeline—funnel-in, delegate-token self-attention, and funnel-out—to achieve linear scaling and robust performance on multivariate time series data.
• Empirical evaluations show significant forecasting accuracy improvements and reduced memory/computation costs compared to standard quadratic-scaling transformer models.

DeltaFormer Hybrid is a transformer-based architecture designed to address both the scalability and performance challenges inherent in modeling multivariate time series (MTS) data. The architecture introduces a delegate-token attention mechanism that constrains cross-variable interactions through learnable low-dimensional tokens, and integrates a three-stage pipeline—funnel-in, delegate-token self-attention, and funnel-out—that together enable linear scaling in the number of variables while simultaneously providing strong noise resilience and improved forecasting accuracy relative to standard quadratic-scaling transformer models (Lee et al., 23 Sep 2025).

1. Delegate-Token Attention Mechanism

DeltaFormer Hybrid departs from unconstrained joint modeling by introducing “delegate tokens” at each temporal patch position. For input $M \in \mathbb{R}^{N \times d \times T}$, where $N$ is the variable (channel) count, $d$ the patch embedding dimension, and $T$ the number of patches per channel, a set of learnable tokens $D_t \in \mathbb{R}^{d’}$ (for $t = 1, \ldots, T$) is introduced, where typically $d’ = d \cdot \alpha$ for some expansion factor $\alpha$.

The attention procedure unfolds in three stages:

• Funnel-In (Inter-Variable): Each delegate token $D_t$ (one per patch index) queries all variable embeddings at time $t$:
  • $N$0
  • $N$1
  • $N$2
  • Attention output: $N$3
• Delegate-Token Self-Attention (Inter-Temporal): The resulting $N$4 are stacked into $N$5 and processed by standard multi-head self-attention across $N$6 delegate tokens:
  • $N$7
• Funnel-Out (Propagating Back to Variables): Each variable embedding queries the updated per-patch delegate token to restore cross-variable context:
  • $N$8
  • $N$9
  • $d$0

All stages incorporate layer normalization and MLPs with residuals to ensure standard transformer block expressiveness. The delegate-token funnel enforces that all variable interactions are mediated through a single low-dimensional summary at each patch position.

2. Three-Stage Hybrid Pipeline and Module Integration

The DeltaFormer Hybrid stack comprises a sequence of transformer blocks, each organized as a three-stage module:

1. Funnel-In: Maps the variable-wise embeddings $d$1 at block $d$2 to the corresponding set of delegate tokens $d$3.
2. Delegate-Token Self-Attention: Models temporal dependencies among patches by applying self-attention on $d$4, yielding $d$5 for all $d$6 patches.
3. Funnel-Out: Broadcasts updated cross-variable context from $d$7 back to the variable representations, producing $d$8.

This modular decomposition allows for a clean separation of inter-variable and inter-temporal modeling, with inter-variable information bottlenecked through delegate tokens and inter-temporal dependencies captured with standard transformer self-attention.

3. Computational Complexity Analysis

DeltaFormer Hybrid is principally motivated by the desire to break the $d$9 complexity bottleneck of standard transformers on MTS data. Its key complexities are:

Operation	Complexity	Description
Funnel-in/out	$T$0	$T$1 attentions, each over $T$2 variables
Delegate self-attn	$T$3	Full-attention over $T$4 delegate tokens
Full transformer	$T$5	Quadratic in total token count
Variate-only baseline	$T$6	Temporal dimension collapsed

When $T$7 (common in long-horizon/long-channel contexts), DeltaFormer Hybrid operates at $T$8, which is asymptotically linear in $T$9 and substantially more scalable than quadratic-complexity alternatives (Lee et al., 23 Sep 2025).

4. Delegate Tokens as Implicit Regularizer

Delegate tokens enforce an information bottleneck that mitigates noisy or indiscriminate mixing across heterogeneous variables. Mathematically, by constraining all inter-variable interactions at a given patch to a single low-dimensional vector $D_t \in \mathbb{R}^{d’}$0, only the most predictive or salient cross-variable information passes the funnel-in softmax. Spurious correlations or non-informative channel interactions are thus suppressed.

Empirical observations demonstrate:

• In a synthetic key-retrieval task with sine-wave signals hidden among noisy variables, DeltaFormer dedicates approximately twice the normalized attention to genuine key variables compared to a standard transformer and retains $D_t \in \mathbb{R}^{d’}$186% of the attention mass with increasing channel noise, whereas standard transformers lose $D_t \in \mathbb{R}^{d’}$250% (Figure 1 in (Lee et al., 23 Sep 2025)).
• In real-world benchmarks (ECL, Solar, Traffic), under up to 80% injected random Gaussian noise, DeltaFormer's forecasting error degrades by only $D_t \in \mathbb{R}^{d’}$36% from its noise-free baseline, while iTransformer and Timer-XL errors rise by 12–14% (Figure 2 in (Lee et al., 23 Sep 2025)).

This bottleneck discourages “attention pollution,” enabling the architecture to focus on genuine inter-variable signals.

5. Empirical Performance and Benchmark Results

Comprehensive evaluations across 12 real-world datasets (8 long-term: ETT{h1,m1,h2,m2}, ECL, Traffic, Weather, Solar; 4 short-term: PEMS03–08), using identical 96-step look-back and multiple forecast horizons, establish several findings:

• State-of-the-art Long-Term Forecasting: DeltaFormer Hybrid achieves new best results on 6/8 long-horizon benchmarks:
  • ECL: MSE 0.165 vs prior best 0.173 (4.6% reduction)
  • Traffic: 0.418 vs 0.428 (2.3% reduction)
  • ETTm2: 0.227 vs 0.278 (18.3% reduction)
• Superiority Over Transformer Baselines: Outperforms full- and variate-only transformer baselines, with MSE gains of 41% over Crossformer and 16% over PatchTST.
• Short-Term Competitiveness: In PEMS (04–08), DeltaFormer is competitive, though MLP-based TimeMixer remains state-of-the-art on short-term tasks.
• Resource Efficiency: On ECL (321 variables), DeltaFormer uses 704 MB memory versus 1,195 MB for iTransformer and 8,107 MB for Timer-XL. On Traffic (862 variables), footprint is 1,099 MB versus 5,376 MB and 43,623 MB, respectively.

These results confirm that DeltaFormer Hybrid combines improved accuracy with substantially reduced computational and memory cost, while its delegate-token bottleneck mechanism empirically enhances robustness to noise and spurious variable interactions (Lee et al., 23 Sep 2025).