--- title: 'SPADE-S: Forecasting & Sequence Modeling' url: https://www.emergentmind.com/topics/spade-s type: topic --- # SPADE-S: Forecasting & Sequence Modeling Searching arXiv for SPADE-S and closely related entries to ground the article in the current literature. SPADE-S is an overloaded designation in the arXiv literature. In forecasting, it denotes **“Split Peak Attention Decomposition – Sparsity‐Robust”**, a deep quantile forecaster designed to correct the systematic under-performance of existing multi-horizon models on low-magnitude and sparse time series [2507.21155]. In long-sequence modeling, **SPADE-S** denotes the **softmax-window local-attention instantiation** of **SPADE**, the **State sPace AugmenteD TransformEr**, in which a single bottom-layer state-space model is combined with efficient local attention in the remaining layers [2212.08136]. This dual usage is important because the two systems address different problem classes, employ different architectures, and report different evaluation protocols. ## 1. Nomenclature and scope The name **SPADE-S** is used in at least two distinct senses. | Usage | Expansion | Domain | |---|---|---| | SPADE-S | Split Peak Attention Decomposition – Sparsity‐Robust | Demand forecasting | | SPADE-S | Softmax-window local-attention instantiation of SPADE | Long-sequence modeling | In the forecasting paper, SPADE-S is introduced as **“a robust forecasting architecture that significantly reduces magnitude- and sparsity-based systematic biases and improves overall prediction accuracy”** on large retail demand datasets [2507.21155]. In the long-sequence paper, SPADE-S refers to the “small” or windowed variant of SPADE used in Long Range Arena, language modeling, and encoder–decoder pre-training experiments [2212.08136]. This suggests that SPADE-S should not be treated as a single canonical model family across machine learning. A related misconception is to conflate SPADE-S with **SPADES**, the **“Stellar PArameters DEtermination Software”** for spectroscopic stellar-parameter inference [1111.0474], or with **SPADE** in spectroscopic photoacoustic denoising [2412.12068]. Those systems share only partial acronymic overlap. ## 2. SPADE-S as a sparsity-robust forecaster In the forecasting literature, SPADE-S is formulated for a multi-horizon quantile forecasting task with forecast creation dates $t \in [T] = \{1,\dots,T\}$, series indices $i \in \mathbb{I}$, and forecast horizons $h=(\ell,s)\in\mathcal{H}$, where $|\mathcal{H}|\approx240$ in multi-week retail use-cases [2507.21155]. The observed inputs comprise past covariates $x_t^{(p)}\in\mathbb{R}^{d_p}$, known future covariates $x_{t,\mathcal{H}_1}^{(f)}\in\mathbb{R}^{d_f\times|\mathcal{H}_1|}$ for span-1 horizons, and static features $x^{(s)}\in\mathbb{R}^{d_s}$, while the target demand is $y_{t,h}\in\mathbb{R}_+$. The model learns a hypothesis $f_\theta$ that estimates, for each quantile $q\in Q=\{0.5,0.9\}$, the conditional distribution $$ P(y_{t,\mathcal{H}} \mid x_{1:t}^{(p)}, x_{1:t,\mathcal{H}_1}^{(f)}, x^{(s)}). $$ It outputs $\hat y_{t,h}^{(q)} \approx F^{-1}_{t,\ell,s}(q)$, thereby directly targeting quantile forecasts rather than point estimates [2507.21155]. The architectural core is a **SparsityMask** routing module. Given a batch of series, the module routes each series to one of two arms based on its **52-week trailing sum**: the **“Non-Sparse” arm** sends a series with nonzero demand in past 52 weeks or a new product to a **Main Encoder → Quantile Heads**, whereas the **“Sparse” arm** sends a series with zero sum and “not new” status to a **SparseQuantileNetwork** [2507.21155]. A series $i$ is marked sparse if $$ \sum_{t'=t-52}^{t} y_{i,t'} = 0 $$ and is signaled as “not new.” For non-sparse series, the model begins by **peak-filtering the history** $\widetilde x_{1:t}^{(p)}$ to isolate high-frequency “off-peak” dynamics as in SPADE [38], and then applies $G$ parallel dilated causal convolutional heads: $$ e_{t,g}^{(p)} = \mathrm{Conv}^{(g)}(\widetilde x_{1:t}^{(p)}; W^{(g)}), \qquad g=1\dots G $$ $$ e_t^{(p)} = \mathrm{Linear}([e_{t,1}^{(p)}, \dots, e_{t,G}^{(p)}]). $$ Each $\mathrm{Conv}^{(g)}$ is a stack of dilated causal convolutions with kernel size $k=3$ and dilation doubling each layer; $G\lesssim6$ balances representation diversity with compute [2507.21155]. The resulting embedding is combined with future and static covariates via residual blocks and passed to separate quantile-prediction heads for each $h\in\mathcal{H}$ and $q\in\{0.5,0.9\}$. The **SparseQuantileNetwork** is introduced to avoid **distributional collapse of CNNs on zero-inflated inputs**. It fits a simple parametric distribution, specifically an exponential distribution whose scale grows linearly with predicted intensity: $$ s_t = \mathrm{PatchMLP}(x_{1:t}^{(p)}) $$ $$ \theta_t = \mathrm{MLP}(s_t) \Rightarrow \mathrm{scale} = h(\mathrm{span},\theta_t) \coloneqq \mathrm{span}\cdot \zeta \cdot \theta_t $$ $$ \hat y_{t,\ell,s}^{(q)} = F^{-1}_{\mathrm{Exp}}(q;\mathrm{scale}) = -\mathrm{scale}\cdot \ln(1-q). $$ The paper states that this **guarantees no collapse of the $q$th quantile even if the history is all zeros, and ensure $\hat y^{(0.5)}=0$** [2507.21155]. ## 3. Loss, magnitude bias, and training methodology SPADE-S is trained by minimizing the multi-quantile numerator of the **Weighted Quantile Loss (WQL)** and is evaluated with WQL: $$ \mathrm{QL}_q(y,\hat y) \coloneqq q\cdot(y-\hat y)_+ + (1-q)\cdot(\hat y-y)_+ $$ $$ \mathrm{WQL}(\hat y; q) = \frac{\sum_{i,t,h} \mathrm{QL}_q(y_{i,t,h}, \hat y_{i,t,h}^{(q)})} {\sum_{i,t,h} y_{i,t,h}}. $$ Because $\mathrm{QL}_q(y,\hat y)=y\cdot f_q(r)$ with $r=(\hat y-y)/y$ and $f_q(r)=|q-1_{r<0}|\cdot|r|$, the paper argues that **series of larger $y$ carry proportionally more weight**. It further states that a formal proposition shows that any loss of the form $\ell(y,\hat y)=g(y)f(r)$ with strictly increasing $g$ results in a **“magnitude bias” favoring high-volume series** [2507.21155]. To address this, SPADE-S combines architecture with a specialized sampling scheme. Training uses SGD with **Adam**, learning rate $10^{-3}$, batch size $32\text{K}$, and early stopping on validation WQL. Series $i$ are sampled with probability proportional to $$ w_i = \max\bigl(\sum_{t'} y_{i,t'}, Q_{0.8}\bigr), $$ where $Q_{0.8}$ is the **80th percentile of the per-series sum**. The paper describes this as **“importance sampling with uniform floor”** and states that it ensures low-magnitude series receive at least weight $Q_{0.8}$, preventing their dropout from batches [2507.21155]. The evaluation protocol uses three large retail forecasting tasks. **D1** has $|\mathbb{I}|\approx700$ million series, $|\mathcal{H}|\approx240$, 260 weeks of training, and a 52-week backtest; **63.7 % have zero demand in past 52 weeks**. **D2** has $|\mathbb{I}|\approx100$ million series, up to 10-week horizons, 260 training weeks, and a 52-week backtest; **90 % are zero in trailing window**. **D3** has $|\mathbb{I}|\approx3$ million series, $|\mathcal{H}|=285$ daily lead/span pairs over 91 days, 730 training days, and a 365-day backtest; **only 9.6 % zero** [2507.21155]. Series are bucketed into six velocity categories: **Super‐Fast (>10 000 units), Fast (365–10 000), Medium (52–365), Slow (2–52), Super-Slow (0–2), and Zero**. To interpret bias, the paper decomposes quantile loss into **UnderBias** and **OverBias**: $$ \mathrm{UnderBias}_q = \sum_{\hat yy} (\hat y-y)\cdot|q-1_{\hat y