SPADE-S: Forecasting & Sequence Modeling

• SPADE-S refers to two distinct models: a sparsity-robust deep forecaster for demand forecasting and a softmax-window transformer variant for long-sequence tasks.
• In demand forecasting, SPADE-S employs dual-arm routing with peak filtering and importance sampling to correct magnitude bias in low-demand series.
• For long-sequence tasks, SPADE-S integrates a global state-space model with windowed self-attention, enhancing efficiency in language modeling and classification.

Searching arXiv for SPADE-S and closely related entries to ground the article in the current literature. SPADE-S is an overloaded designation in the arXiv literature. In forecasting, it denotes “Split Peak Attention Decomposition – Sparsity‐Robust”, a deep quantile forecaster designed to correct the systematic under-performance of existing multi-horizon models on low-magnitude and sparse time series (Wolff et al., 24 Jul 2025). In long-sequence modeling, SPADE-S denotes the softmax-window local-attention instantiation of SPADE, the State sPace AugmenteD TransformEr, in which a single bottom-layer state-space model is combined with efficient local attention in the remaining layers (Zuo et al., 2022). This dual usage is important because the two systems address different problem classes, employ different architectures, and report different evaluation protocols.

1. Nomenclature and scope

The name SPADE-S is used in at least two distinct senses.

Usage	Expansion	Domain
SPADE-S	Split Peak Attention Decomposition – Sparsity‐Robust	Demand forecasting
SPADE-S	Softmax-window local-attention instantiation of SPADE	Long-sequence modeling

In the forecasting paper, SPADE-S is introduced as “a robust forecasting architecture that significantly reduces magnitude- and sparsity-based systematic biases and improves overall prediction accuracy” on large retail demand datasets (Wolff et al., 24 Jul 2025). In the long-sequence paper, SPADE-S refers to the “small” or windowed variant of SPADE used in Long Range Arena, language modeling, and encoder–decoder pre-training experiments (Zuo et al., 2022).

This suggests that SPADE-S should not be treated as a single canonical model family across machine learning. A related misconception is to conflate SPADE-S with SPADES, the “Stellar PArameters DEtermination Software” for spectroscopic stellar-parameter inference (Posbic et al., 2011), or with SPADE in spectroscopic photoacoustic denoising (Lin et al., 2024). Those systems share only partial acronymic overlap.

2. SPADE-S as a sparsity-robust forecaster

In the forecasting literature, SPADE-S is formulated for a multi-horizon quantile forecasting task with forecast creation dates $t \in [T] = \{1,\dots,T\}$, series indices $i \in \mathbb{I}$, and forecast horizons $h=(\ell,s)\in\mathcal{H}$, where $|\mathcal{H}|\approx240$ in multi-week retail use-cases (Wolff et al., 24 Jul 2025). The observed inputs comprise past covariates $x_t^{(p)}\in\mathbb{R}^{d_p}$, known future covariates $$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$ for span-1 horizons, and static features $x^{(s)}\in\mathbb{R}^{d_s}$, while the target demand is $y_{t,h}\in\mathbb{R}_+$.

The model learns a hypothesis $f_\theta$ that estimates, for each quantile $q\in Q=\{0.5,0.9\}$, the conditional distribution

$i \in \mathbb{I}$0

It outputs $i \in \mathbb{I}$1, thereby directly targeting quantile forecasts rather than point estimates (Wolff et al., 24 Jul 2025).

The architectural core is a SparsityMask routing module. Given a batch of series, the module routes each series to one of two arms based on its 52-week trailing sum: the “Non-Sparse” arm sends a series with nonzero demand in past 52 weeks or a new product to a Main Encoder → Quantile Heads, whereas the “Sparse” arm sends a series with zero sum and “not new” status to a SparseQuantileNetwork (Wolff et al., 24 Jul 2025). A series $i \in \mathbb{I}$2 is marked sparse if

$i \in \mathbb{I}$3

and is signaled as “not new.”

For non-sparse series, the model begins by peak-filtering the history $i \in \mathbb{I}$4 to isolate high-frequency “off-peak” dynamics as in SPADE [38], and then applies $i \in \mathbb{I}$5 parallel dilated causal convolutional heads:

$i \in \mathbb{I}$6

$i \in \mathbb{I}$7

Each $i \in \mathbb{I}$8 is a stack of dilated causal convolutions with kernel size $i \in \mathbb{I}$9 and dilation doubling each layer; $h=(\ell,s)\in\mathcal{H}$0 balances representation diversity with compute (Wolff et al., 24 Jul 2025). The resulting embedding is combined with future and static covariates via residual blocks and passed to separate quantile-prediction heads for each $h=(\ell,s)\in\mathcal{H}$1 and $h=(\ell,s)\in\mathcal{H}$2.

The SparseQuantileNetwork is introduced to avoid distributional collapse of CNNs on zero-inflated inputs. It fits a simple parametric distribution, specifically an exponential distribution whose scale grows linearly with predicted intensity:

$h=(\ell,s)\in\mathcal{H}$3

$h=(\ell,s)\in\mathcal{H}$4

$h=(\ell,s)\in\mathcal{H}$5

The paper states that this guarantees no collapse of the $h=(\ell,s)\in\mathcal{H}$6th quantile even if the history is all zeros, and ensure $h=(\ell,s)\in\mathcal{H}$7 (Wolff et al., 24 Jul 2025).

3. Loss, magnitude bias, and training methodology

SPADE-S is trained by minimizing the multi-quantile numerator of the Weighted Quantile Loss (WQL) and is evaluated with WQL:

$h=(\ell,s)\in\mathcal{H}$8

$h=(\ell,s)\in\mathcal{H}$9

Because $|\mathcal{H}|\approx240$0 with $|\mathcal{H}|\approx240$1 and $|\mathcal{H}|\approx240$2, the paper argues that series of larger $|\mathcal{H}|\approx240$3 carry proportionally more weight. It further states that a formal proposition shows that any loss of the form $|\mathcal{H}|\approx240$4 with strictly increasing $|\mathcal{H}|\approx240$5 results in a “magnitude bias” favoring high-volume series (Wolff et al., 24 Jul 2025).

To address this, SPADE-S combines architecture with a specialized sampling scheme. Training uses SGD with Adam, learning rate $|\mathcal{H}|\approx240$6, batch size $|\mathcal{H}|\approx240$7, and early stopping on validation WQL. Series $|\mathcal{H}|\approx240$8 are sampled with probability proportional to

$|\mathcal{H}|\approx240$9

where $x_t^{(p)}\in\mathbb{R}^{d_p}$0 is the 80th percentile of the per-series sum. The paper describes this as “importance sampling with uniform floor” and states that it ensures low-magnitude series receive at least weight $x_t^{(p)}\in\mathbb{R}^{d_p}$1, preventing their dropout from batches (Wolff et al., 24 Jul 2025).

The evaluation protocol uses three large retail forecasting tasks. D1 has $x_t^{(p)}\in\mathbb{R}^{d_p}$2 million series, $x_t^{(p)}\in\mathbb{R}^{d_p}$3, 260 weeks of training, and a 52-week backtest; 63.7 % have zero demand in past 52 weeks. D2 has $x_t^{(p)}\in\mathbb{R}^{d_p}$4 million series, up to 10-week horizons, 260 training weeks, and a 52-week backtest; 90 % are zero in trailing window. D3 has $x_t^{(p)}\in\mathbb{R}^{d_p}$5 million series, $x_t^{(p)}\in\mathbb{R}^{d_p}$6 daily lead/span pairs over 91 days, 730 training days, and a 365-day backtest; only 9.6 % zero (Wolff et al., 24 Jul 2025). Series are bucketed into six velocity categories: Super‐Fast (>10 000 units), Fast (365–10 000), Medium (52–365), Slow (2–52), Super-Slow (0–2), and Zero.

To interpret bias, the paper decomposes quantile loss into UnderBias and OverBias:

$x_t^{(p)}\in\mathbb{R}^{d_p}$7

$x_t^{(p)}\in\mathbb{R}^{d_p}$8

4. Empirical results and ablations in forecasting

The forecasting paper reports percent changes in WQL of SPADE-S relative to baselines, with SPADE [38] used for D1 and D2 and MQTransformer for D3 (Wolff et al., 24 Jul 2025). Negative values indicate improvement.

Category	D1	D2	D3
All, P50	–0.92 %	–0.77 %	–1.95 %
All, P90	–2.21 %	–6.58 %	–4.28 %
Zero, P50	–4.37 %	+0.40 %	–0.45 %
Zero, P90	–10.05 %	+0.20 %	+0.27 %

The paper’s stated highlights are that on D1’s zero-demand series, P90 QL drops by 10.05 %, P50 by 4.37 %; that on D2’s “Fast” series, P90 improves by 14.80 %; and that across all datasets, tail forecasts (P90) improve by up to 15 %, and overall by 2.21 % (D1), 6.58 % (D2), 4.28 % (D3) (Wolff et al., 24 Jul 2025). A plausible implication is that the combination of routing, sparse-arm modeling, and sampling is most consequential in the low-velocity and tail-forecast regimes emphasized by the paper.

Appendix C.1 reports an ablation on D1 comparing variants of SPADE with adjusted sampling cutoff quantile (V10), rule-based sparse override (V11), Mixture-of-Experts without sparse arm (V13), learned P90 layer via truncated normal or exponential (V15–V18), and full SPADE-S (V19). The paper states that the full SPADE-S (V19) consistently achieves the best P50 and P90 across all velocity strata (Wolff et al., 24 Jul 2025). It further reports that inserting the exponential-based SparseQuantileNetwork reduces P90 bias on zero series from +72 % (V9) to –35 % (V19), while the multi-head encoder reduces variance on “Super Fast” forecasts by another 1 %.

The comparison to prior universal forecasters is framed in terms of systematic under-bias on low-velocity series when trained on skewed data, attributed to magnitude bias in loss and representational collapse on zeros. The cited baselines are MQCNN, MQTransformer, and SPADE (Wolff et al., 24 Jul 2025). The paper attributes SPADE-S’s gains to architectural diversity (multi-head conv), sparse-arm routing, and importance sampling.

5. SPADE-S as the softmax-window SPADE variant for long sequences

In the long-sequence literature, SPADE denotes State sPace AugmenteD TransformEr, and SPADE-S denotes the Softmax-window local-attention instantiation presented in “Efficient Long Sequence Modeling via State Space Augmented Transformer” (Zuo et al., 2022). The overall design is hierarchical: the very bottom layer is replaced by a Single-layer State-Space Model (SSM) “global” block, and the remaining layers use efficient windowed self-attention. The paper’s stated intuition is that the SSM injects a strong, data-independent global bias into the representation, after which narrow-window attention suffices to capture fine-grained local patterns.

A single SPADE block is defined for inputs $x_t^{(p)}\in\mathbb{R}^{d_p}$9 as

$$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$0

Here $$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$1 is LayerNorm, $$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$2 is a learned $$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$3 fusion matrix, and $$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$4 is the usual two-layer MLP (Zuo et al., 2022).

The SSM layer uses the S4 instantiation of a linear state-space model:

$$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$5

which is discretized by the bilinear method to yield

$$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$6

and recurrence equations

$$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$7

Unrolled, this is equivalent to a 1-D convolution

$$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$8

For the SPADE-S experiments on Long Range Arena, the reported configuration is: embedding dimension $$x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$$9, feed-forward inner dimension $x^{(s)}\in\mathbb{R}^{d_s}$0, total depth 4 layers (1 global + 3 local), 4 attention heads in each local layer, SSM state dimension $x^{(s)}\in\mathbb{R}^{d_s}$1, and windowed self-attention with window size $x^{(s)}\in\mathbb{R}^{d_s}$2 for all but Path-X, where $x^{(s)}\in\mathbb{R}^{d_s}$3 (Zuo et al., 2022). An alternate SPADE-S variant uses chunk-based gating (MEGA-chunk) with chunk size 128, and 4096 on Path-X. The same paper also embeds SPADE-S in a 16-layer decoder for large causal language modeling and places SSM modules at layer-1 of encoder and decoder in an encoder–decoder pre-training setup.

6. Empirical behavior in long-sequence modeling and relation to adjacent names

For Long Range Arena, the reported average accuracies are 80.48% for Vanilla S4, 86.19% for SPADE-S (softmax-window), and 87.40% for SPADE-S (MEGA-chunk) (Zuo et al., 2022). For WikiText-103 causal language modeling, the reported test perplexities are 18.8 for full Transformer, 19.7 for window-only, 23.2 for S4, 20.9 for FLASH-chunk, 19.8 for MEGA-chunk, 18.5 for SPADE-S (softmax-window), 19.5 for SPADE-S (MEGA-chunk), and 19.9 for SPADE-S (FLASH-chunk). On GLUE, the reported dev-set average scores are 85.1 for published T5_base, 85.4 for SPADE_base, and 86.8 for SPADE_base++. For summarization, SPADE_base++ reports ROUGE-2 = 21.65 on arXiv, 20.40 on CNN/DailyMail, 19.03 on MediaSum, and 19.63 on MultiNews (Zuo et al., 2022).

The paper concludes that, across extremely long classification, large-context language modeling, and downstream NLU and NLG, SPADE-S improves over pure local or pure SSM baselines (Zuo et al., 2022). This suggests that, in this line of work, SPADE-S is best understood not as a sparsity-aware forecaster but as a specific efficient-transformer recipe built around one global SSM + many local windows.

A common source of confusion is the broader SPADE / SPADES namespace. “SPADES: a Stellar PArameters DEtermination Software” is a fully automatic, Java-based package for stellar-parameter estimation by direct profile fitting of selected spectral lines (Posbic et al., 2011). “SPADE: Spectroscopic Photoacoustic Denoising using an Analytical and Data-free Enhancement Framework” is a denoising framework for spectroscopic photoacoustic imaging that combines Spectral Domain Data Re-assembly, BM3D, and Zero-Shot Noise2Noise (Lin et al., 2024). Neither is the same system as either forecasting SPADE-S or sequence-modeling SPADE-S.

Within current arXiv usage, therefore, SPADE-S denotes either a sparsity-robust foundational forecaster for heterogeneous retail demand series (Wolff et al., 24 Jul 2025) or a softmax-window state-space augmented transformer variant for efficient long-sequence modeling (Zuo et al., 2022). The correct interpretation depends entirely on domain context, task definition, and accompanying notation.