---
title: 'DELTAformer: Transformer with Delegate Tokens'
url: https://www.emergentmind.com/topics/transformer-with-delegate-token-attention-deltaformer
type: topic
---

# DELTAformer: Transformer with Delegate Tokens

Transformer with Delegate Token Attention (DELTAformer) is a transformer architecture for multivariate time series (MTS) that strategically constrains inter-variable modeling through delegate tokens and then uses those tokens to perform full, unconstrained, inter-temporal modeling. In the formulation reported for MTS, the central objective is to address two difficulties simultaneously: scalability with respect to the number of variables and degradation caused by indiscriminate inter-variable mixing in sparse and noisy settings. DELTAformer does so by imposing a bottleneck on cross-variable information flow, which functions as an implicit regularizer and yields a complexity profile that scales practically linearly with variable count while outperforming standard transformers across the reported benchmarks [2509.19471].

## 1. Problem setting and design rationale

In MTS, variable count is a central scalability bottleneck. Full attention over $V\times T$ tokens grows as $\mathcal{O}((VT)^2)$ in space and time and can soon exhaust GPU memory. The same setting also raises a representational issue: when informative signals are sparse, standard transformers tend to spread attention over irrelevant variables, which can induce spurious correlations and degrade forecasting accuracy. The description of DELTAformer therefore couples scalability with selectivity rather than treating them as separate optimization targets [2509.19471].

The architecture is motivated by three linked concerns. The first is the cost of modeling many variables jointly. The second is noise-accumulation from indiscriminate inter-variable mixing. The third is representational misalignment arising when heterogeneous variables are mixed at full capacity without sufficiently filtering irrelevant signals. DELTAformer’s response is to bottleneck all inter-variable interactions through delegate tokens. This design reduces complexity from quadratic to practically linear in the number of variables and forces the model to be highly selective about which cross-variable relationships are allowed to propagate.

A plausible implication is that DELTAformer treats sparsity of informative signals not merely as a nuisance but as an architectural prior. The reported results are explicitly framed as aligning model design with domain-specific challenges in MTS, with the delegate-token bottleneck serving both computational and statistical purposes.

## 2. Layer organization and delegate-token mechanism

DELTAformer operates after variate-wise patching. With $C$ variables, patch length $P$, sequence length $L$, and embedding dimension $d$, the number of patches per variable is $M=L/P$, and the patched representation is
$$
M\in\mathbb R^{\,C\times d\times M},
$$
where each slice $M_{[:,:,i]}\in\mathbb R^{\,C\times d}$ is the stack of $C$ variable-wise patches at patch position $i$. The model allocates one delegate token per patch position,
$$
D\in\mathbb R^{\,M\times d'}.
$$

Each DELTAformer layer consists of three stages: funnel-in, delegate-token self-attention, and funnel-out [2509.19471].

| Stage | Operation | Role |
|---|---|---|
| Funnel-in | All $C$ patches at position $i$ attend into delegate token $D_i$ | Gather inter-variable information |
| Delegate-token attention | Full self-attention over all updated delegates | Model inter-temporal structure |
| Funnel-out | Each patch stack attends from its position’s enriched delegate token | Scatter selected information back to variables |

In funnel-in, the delegate token at a given patch position receives information from all variables at that position. In delegate-token attention, the updated delegates are stacked and processed with full self-attention along the patch-index dimension. In funnel-out, the enriched delegate corresponding to each patch position redistributes information back to that position’s variable-wise patches. The architecture can therefore be read as alternating between a narrow inter-variable channel and a full temporal channel.

This separation is central. Inter-variable mixing is intentionally constrained, whereas inter-temporal modeling remains full and unconstrained at the delegate level. A common misunderstanding is to interpret DELTAformer as simply “sparser attention.” The reported mechanism is more specific: it is a structured bottleneck on cross-variable exchange, not a generic pruning rule.

## 3. Mathematical formulation

For each position $i\in\{1,\dots,M\}$, funnel-in is defined by a small attention from the $C$ patches at that position into the single delegate token. Let
- $Q_D = D_i W_Q\in\mathbb R^{1\times d_k}$,
- $K_M = M_{[:,:,i]}W_K\in\mathbb R^{C\times d_k}$,
- $V_M = M_{[:,:,i]}W_V\in\mathbb R^{C\times d_v}$.

Then
\[
A_i \;=\;\mathrm{softmax}\!\bigl(Q_D\,K_M^T/\sqrt{d_k}\bigr)\in\mathbb R^{1\times C},
\quad
D_i^{\text{(in)}} = A_i\,V_M \in \mathbb R^{1\times d_v}.
\]
A residual MLP and LayerNorm then produce
\[
D_i'
= \mathrm{LayerNorm}\bigl(D_i^{(\mathrm{in})} + \mathrm{MLP}(D_i^{(\mathrm{in})})\bigr).
\]

In the simplified notation reported for the paper, where $d_v=d_k=d$ and $d'=d$,
\[
D_i^{\mathrm{funnel\_in}}
= \mathrm{softmax}\bigl(D_i\,M_{[:,:,i]}^T/\sqrt d\bigr)\,M_{[:,:,i]},
\quad
D'_i = \mathrm{LayerNorm}\bigl(D_i^{\mathrm{funnel\_in}}
+ \mathrm{MLP}(D_i^{\mathrm{funnel\_in}})\bigr).
\]

The delegate-token stage applies full self-attention across the updated delegates:
\[
\widehat D
= \mathrm{softmax}\!\bigl(D'\,D'^T/\sqrt{d'}\bigr)\,D',
\quad
D^{\delta}
= \mathrm{LayerNorm}\bigl(\widehat D + \mathrm{MLP}(\widehat D)\bigr).
\]
This attention is quadratic in $M$, not in the total number of variable-time tokens.

Funnel-out then redistributes the enriched delegate at each patch position back to the corresponding variable-wise patches:
\[
B_i
= \mathrm{softmax}\!\bigl(M_{[:,:,i]}\,{D^{\delta}_i}^T/\sqrt{d'}\bigr)\in\mathbb R^{C\times 1},
\quad
M'_{[:,:,i]} = B_i\,D^{\delta}_i\in\mathbb R^{C\times d'}.
\]
Residual MLP and LayerNorm yield the final updated patches. The net effect is that each layer first compresses cross-variable evidence into a single token per patch position, then evolves those tokens temporally, then projects the result back to the variable domain [2509.19471].

## 4. Complexity and scaling behavior

The reported complexity comparison makes explicit why DELTAformer is described as scaling linearly with variable count in practical MTS regimes. A standard full transformer over $C\times M$ tokens has complexity
$$
\mathcal O\bigl((C\,M)^2\,d\bigr)=\mathcal O(C^2M^2 d).
$$
A variate-only transformer has complexity
$$
\mathcal O(C^2 d).
$$
DELTAformer has two funnel stages, each costing $\mathcal O(C\,M\,d)$, and delegate-token attention costing $\mathcal O(M^2d)$, so the total cost is
\[
\mathcal O\bigl(C\,M\,d + M^2d\bigr)=\mathcal O\bigl(d\,(C\,M + M^2)\bigr).
\]

Since the intended benchmark regime is one in which $M\ll C$, this scales practically as $\mathcal O(C\,M)$ rather than $\mathcal O(C^2M^2)$. The complexity claim is therefore conditional rather than absolute: DELTAformer does not eliminate the quadratic term in the temporal dimension of the delegate tokens, but it removes the quadratic dependence on jointly attending over all variable-time pairs [2509.19471].

This also clarifies a common misconception about the phrase “linear scaling.” In the reported formulation, linearity pertains to the dependence on variable count under the regime where $M\ll C$; it does not imply that every component of the model is linear-time.

## 5. Empirical performance, robustness, and ablations

The reported empirical evaluation covers 12 benchmarks, including ECL, Solar, Traffic, Weather, ETT variants, and PEMS datasets. On long-term forecasting, DELTAformer reduces MSE by 2–18% against prior transformers, with up to 41% MSE reduction vs Crossformer and 16% vs PatchTST on average. On high-dimensional series, the paper reports 2.3% and 4.6% MSE gains on Traffic $(C=862)$ and ECL $(C=321)$, respectively [2509.19471].

The architecture is also reported to exhibit superior noise-resilience. In a synthetic “key-retrieval” setup, described as sine-wave keys planted among random noise across variables, DELTAformer allocates roughly twice as much attention to true keys as full and variate transformers, and its focus degrades only 14% as irrelevant context grows, versus approximately 50% loss for baselines. With Gaussian noise injected at up to 80% of entries across three real datasets, DELTAformer’s error grows by approximately 6% from baseline, versus 12–14% growth for iTransformer and Timer-XL.

The ablation results are used to isolate which parts of the architecture matter. Replacing funnel-in and funnel-out attention by MLP or linear layers degrades MSE by up to 27%, which the paper interprets as evidence that dynamic weighting is crucial for selective inter-variable aggregation and propagation. Varying delegate-token size and patch length produces only small performance shifts, supporting the stated hypothesis that MTS signals are sparse and that large bottleneck capacity is not required.

Taken together, these results support two claims made jointly in the paper: that the delegate-token bottleneck improves scalability and that it can improve performance rather than merely preserving it.

## 6. Implicit regularization, selectivity, and related naming

DELTAformer is explicitly described as using delegate tokens as an implicit regularizer. Because all cross-variable information must pass through a small set of $M$ delegate tokens, the model is forced to assign high attention weights only to variables that help explain temporal patterns. Uninformative or noisy variables receive low attention during funnel-in and therefore cannot substantially pollute downstream temporal attention. The paper characterizes this as an information-selectivity constraint that discourages spurious correlations, penalizes attention dispersion across many variables, and empirically yields better generalization and stronger noise-robustness [2509.19471].

The acronym “DELTA” is also used in a distinct line of work on efficient long-context reasoning. That method, titled “DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning,” partitions transformer decoder layers into full-attention “Init” layers, a small set of “Δ-selection” layers, and sparse-attention layers, preserving the full KV cache while reducing attended tokens during decoding [2510.09883]. DELTAformer, by contrast, is a multivariate time-series architecture whose defining operation is delegate-token attention: funnel-in across variables, full temporal attention over delegates, and funnel-out back to variables. The overlap in acronym therefore does not indicate a shared mechanism.

This distinction matters because the two systems address different bottlenecks. DELTA targets decoding efficiency in large reasoning models by selectively reusing attention maps, whereas DELTAformer targets MTS scalability and noise-resilient cross-variable modeling. A plausible implication is that the shared name reflects a broad emphasis on selectivity in attention design, but the actual architectural commitments are different.

Source: https://www.emergentmind.com/topics/transformer-with-delegate-token-attention-deltaformer