Exformer: Domain-Specific Transformer Variants

• Exformer is a set of transformer-derived models tailored for distinct domains, including hydrology, speech, and finance.
• It employs domain-specific modifications such as sparse attention, speaker conditioning, and dynamic variable selection to improve task performance.
• Empirical results show that each variant consistently outperforms generic models by addressing key challenges like extreme events, speaker extraction accuracy, and noisy financial data.

Exformer denotes a family of transformer-derived architectures proposed in distinct research areas rather than a single canonical model. In the supplied literature, the name appears in at least three domain-specific senses: Exformer, the Extreme-Adaptive Transformer for hydrologic multivariate time-series forecasting (Shrestha et al., 2 Jul 2026); Exformer, a time-domain target speaker extraction architecture conditioned on a pre-trained speaker embedder (Wang et al., 2022); and EXFormer, a multi-scale trend-aware Transformer with dynamic variable selection for foreign exchange return prediction (Liu et al., 14 Dec 2025). A closely related but differently spelled term, ExoFormer, refers to a language-model architecture based on exogenous anchors for attention projection mixing (Su, 13 Jan 2026). The shared nomenclature reflects a recurring pattern of transformer adaptation to domain-specific structural constraints, but the architectures, objectives, and empirical protocols are otherwise independent.

1. Nomenclature and scope

The term Exformer is not unique to one lineage. In hydrologic forecasting, it abbreviates Extreme-Adaptive Transformer and denotes an encoder-only forecasting model with decomposition and an extreme-aware sparse attention mask (Shrestha et al., 2 Jul 2026). In speech separation, it designates a time-domain target speaker extraction model that combines a frozen pre-trained speaker embedder with a dual-path separator based on transformer encoder blocks (Wang et al., 2022). In quantitative finance, the uppercase form EXFormer denotes a Transformer-based architecture for daily exchange-rate return prediction that integrates dynamic variable selection, multi-scale convolution, squeeze-and-excitation, and trend-aware self-attention (Liu et al., 14 Dec 2025).

A useful disambiguation is therefore by application domain rather than by name alone.

Variant	Domain	Defining mechanism
Exformer	Hydrologic streamflow forecasting	Extreme-Adaptive Attention with Local, Stride, and Extreme components
Exformer	Target speaker extraction	Dual-path transformer separator conditioned on a speaker embedder
EXFormer	Foreign exchange return prediction	Multi-scale trend-aware self-attention with dynamic variable selection
ExoFormer	Language-model architecture	Exogenous anchors for normalized projection mixing

The orthographic proximity between Exformer and ExoFormer can invite confusion. The latter is explicitly introduced to address a tension in Transformer design involving residualized attention projections and exogenous anchor modules, not forecasting or speech extraction (Su, 13 Jan 2026).

2. Extreme-Adaptive Transformer for hydrologic forecasting

In hydrologic forecasting, Exformer is designed for multivariate time series forecasting in settings where the target series is highly skewed and contains rare but important extreme events (Shrestha et al., 2 Jul 2026). The paper focuses on streamflow forecasting, where most observations are normal or low-flow values and a small number are extreme peaks caused by rainstorms or flood conditions. The forecasting problem is formulated as mapping historical multivariate observations $\mathbf{X} \in \mathbb{R}^{I \times D}$ to future values $\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$, and the experiments use a 3-day-ahead horizon with 15-minute sampling, yielding $h = 288$ prediction steps.

The central claim is that canonical self-attention is insufficient for such data because it is token-agnostic, treats all positions similarly, and may dilute rare extreme tokens among abundant normal tokens. Exformer therefore introduces Extreme-Adaptive Attention, a sparse mechanism with three components: Local, Stride, and Extreme (Shrestha et al., 2 Jul 2026). Local attention captures short-range temporal dependence among normal tokens, Stride attention captures periodic or seasonal dependencies among normal tokens, and Extreme attention allows extreme queries to attend to extreme keys. The final sparse mask is defined by combining a temporal sparse mask with an extreme-aware mask:

$$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$

where query type determines whether temporal sparsity alone is sufficient or whether extreme-aware routing is required.

Architecturally, the model is an encoder-only Transformer with decomposition. The input sequence is split into a seasonal component $\mathbf{X}_s$ and a trend component $\mathbf{X}_t$; the encoder models the seasonal part, a linear layer forecasts the trend part, and the final output is their sum. The model further employs a dimension-invariant embedding layer and non-overlapping temporal patches, so attention operates over patch tokens rather than raw time steps. Patch labels are binary, with 0 = normal and 1 = extreme, obtained by computing an outlier score $s_t$, thresholding it to produce $\ell_t \in \{0,1\}$, and aggregating labels within each patch (Shrestha et al., 2 Jul 2026). The details specify that outlier scores are estimated using a Gaussian Mixture Model (GMM), all series are log-transformed via $x_i \leftarrow \log(1+x_i)$ and standardized, and Kruskal-Wallis sampling is used to address class imbalance.

The experiments use four hydrologic datasets from Santa Clara County, California—Ross, Saratoga, UpperPen, and SFC—with training and validation data from January 1988 to August 2021 and forecast evaluation on September 2021 to May 2022 (Shrestha et al., 2 Jul 2026). Baselines include FEDformer, Informer, NLinear, DLinear, iTransformer, Attention-LSTM, NEC+, DAN, and PFformer. Evaluation uses RMSE and MAPE. For the $h=288$ setting, reported Exformer results are RMSE 4.20, MAPE 0.05 on Ross, RMSE 1.61, MAPE 0.07 on Saratoga, RMSE 0.96, MAPE 0.04 on UpperPen, and RMSE 15.12, MAPE 0.12 on SFC. The paper states that Exformer achieves the best MAPE on all four datasets and the best RMSE on 3 of 4 datasets.

The model is also presented as computationally efficient. Its complexity is stated as

$\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$0

with $\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$1 the number of local keys per query, $\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$2 the number of stride keys per query, $\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$3 the patch size, and $\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$4 the number of extreme patch tokens. On input length 1440 and horizon 288, the paper reports 31.60 GFLOPs, 14.30M parameters, and 240.85 MB memory. The ablations further indicate that each attention component contributes, with the Extreme-only branch usually the strongest single component, while the full combination performs best overall (Shrestha et al., 2 Jul 2026).

3. Exformer for target speaker extraction

In speech processing, Exformer refers to a time-domain target speaker extraction (TSE) architecture that couples a fixed speaker-conditioning pipeline with a transformer-based separator (Wang et al., 2022). The design consists of two principal parts: a pre-trained speaker embedder network that converts an enrollment utterance into a fixed speaker vector and a separator network based on transformer encoder blocks inside a dual-path framework inspired by SepFormer. The stated motivation is twofold: to replace earlier convolutional or recurrent separation blocks with transformer encoder blocks and to exploit unlabeled mixtures through a semi-supervised training procedure.

The speaker embedder maps an enrollment utterance $\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$5 to

$\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$6

Its input is a 40-dimensional log mel-spectrogram; the architecture uses 3 BLSTM layers, each with 768 hidden units, followed by a linear projection to a 256-dimensional embedding, and the final embedding is $\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$7-normalized (Wang et al., 2022). The embedder is pre-trained with GE2E loss on VoxCeleb1 and clean LibriSpeech training data, and it is fixed during training of the extraction network.

The extraction network follows an encoder-separator-decoder pattern. A 1D convolutional encoder with ReLU transforms the mixture waveform $\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$8 into latent features $\mathbf{X}_{pred} \in \mathbb{R}^{O \times D}$9. The masking network then segments $h = 288$0 into overlapping chunks of length $h = 288$1 with 50% overlap, producing

$h = 288$2

Within each dual-path block, speaker information is fused with the mixture representation using one of three tested strategies: concatenation, multiplicative, or additive fusion. The additive form is

$h = 288$3

The paper reports that additive fusion performed best (Wang et al., 2022).

The separator alternates between intra-chunk modeling and inter-chunk modeling, both implemented with transformer encoder blocks containing layer normalization, multi-head attention, feed-forward networks, residual connections, and sinusoidal positional encodings before each MHA. The system stacks $h = 288$4 such dual-path blocks. After the final block, a 2D convolution produces source masks, overlap-add reconstructs full-length masks, and a 1D transposed convolution decodes masked latent features back to waveforms. The setup is explicitly two-speaker extraction, with one mask for the target speech and one for the residual interferer.

A major contribution is the semi-supervised procedure. For unlabeled data, the model uses a speaker representation loss based on embeddings of the estimated target and residual signals:

$h = 288$5

with triplet-style loss

$h = 288$6

where $h = 288$7 (Wang et al., 2022). When supervised references are available, Exformer combines this term with SI-SDR reconstruction loss; when references are unavailable, it uses only the embedder loss. The full semi-supervised objective employs $h = 288$8 and $h = 288$9.

The paper reports that training from scratch with the embedder loss hurts optimization, so a two-stage training strategy is used: supervised pretraining on labeled mixtures with SI-SDR loss, followed by semi-supervised fine-tuning with the full objective (Wang et al., 2022). Evaluation is conducted on WSJ0-2mix-extr, with 20,000 training utterances, 5,000 development utterances, and 3,000 test utterances, at 8 kHz, with mixtures generated at 0 to 5 dB SNR. Semi-supervised experiments add over 100,000 VoxCeleb1 recordings as unlabeled data.

Reported supervised results on the test set are 19.05 dB SI-SDR / 3.78 PESQ for Exformer with concatenation, 19.55 dB / 3.82 with multiplicative fusion, and 19.85 dB / 3.82 with additive fusion, compared with 18.2 dB / 3.49 for SpEx+ and 14.73 dB / 3.14 for TseNet. With two-stage semi-supervised training and 10% unlabeled sampling, the best reported result is 20.49 dB SI-SDR and 3.85 PESQ, approximately a 0.6 dB SI-SDR gain over the supervised additive-fusion baseline (Wang et al., 2022).

4. EXFormer for foreign exchange return prediction

In finance, EXFormer is a Transformer-based architecture for daily foreign exchange return prediction (Liu et al., 14 Dec 2025). The model targets one-step-ahead returns for EUR/USD, USD/JPY, and GBP/USD, using a sliding window of recent history and 28 exogenous covariates spanning FX cross rates, equities, yields, commodities, and macro variables. The paper frames the problem as difficult because daily FX returns are noisy, regime-dependent, weakly autocorrelated, and influenced by time-varying covariate relevance. It further argues that vanilla self-attention is too brittle because pointwise matching ignores local slope or trend context.

EXFormer combines four named components: dynamic variable selection, multi-scale convolution, a squeeze-and-excitation (SE) block, and multi-scale trend-aware self-attention (Liu et al., 14 Dec 2025). For input sequence

$$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$0

each covariate series is first embedded separately:

$$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$1

A shared scoring layer computes raw importance scores $$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$2, which are normalized across variables using softmax to yield $$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$3. The weighted embeddings are concatenated into $$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$4. The paper emphasizes that this module provides time-varying, pre-hoc interpretability because variable importance is assigned before prediction.

The selected features are then processed by three 1D convolution branches with kernel sizes $$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$5, representing short-, medium-, and long-term temporal dependencies. Their outputs are concatenated, projected, and passed through an SE recalibration stage. The SE weights are produced from a temporal average-pooled summary

$$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$6

and a bottleneck MLP, after which channels are rescaled as $$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$7 (Liu et al., 14 Dec 2025).

The core novelty is multi-scale trend-aware self-attention. Instead of forming queries and keys directly from the input with standard linear projections, EXFormer uses parallel 2D convolution branches over the head-time dimensions:

$$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$8

The value projection is shared, and each branch computes a local-context attention output

$$M_{EA}(i,j)= \begin{cases} M_D(i,j)\land M_E(i,j), & e_i=0 \ M_E(i,j), & e_i=1 \end{cases}$$9

The branch outputs are concatenated and fused into $\mathbf{X}_s$0. According to the paper, this modification is intended to align observations by local slope/trend rather than only by instantaneous value. A GRU-based decoder then produces the next-day return forecast from the attention-enriched representation, so the model is Transformer-inspired rather than a standard encoder-decoder Transformer (Liu et al., 14 Dec 2025).

The study uses daily spot exchange rates from May 7, 2010 to August 29, 2024. Returns are computed as

$\mathbf{X}_s$1

and the forecasting task is 1-day-ahead prediction under five look-back windows $\mathbf{X}_s$2. Data are split 80% training, 10% validation, and 10% test. Baselines include Random Walk, ARIMA, MLP, GRU, LSTM, TCN, Transformer, Informer, LSTM-ARIMA, and Chronos (chronos-t5-base). Training uses mean squared error, dropout, early stopping, and automated hyperparameter tuning with Optuna in PyTorch 2.5.1 on an NVIDIA RTX 3090 GPU with an Intel Xeon Gold 6330 CPU (Liu et al., 14 Dec 2025).

Empirically, the paper reports that EXFormer consistently outperforms the random walk across all three currency pairs and all tested windows, with MSFE ratios generally in the mid-50s to 70s and up to about 98.8 in the hardest EUR/USD 30-day setting. Reported directional-accuracy improvements over the random walk peak at 22.8% for USD/JPY at $\mathbf{X}_s$3, 11.4% for EUR/USD at $\mathbf{X}_s$4, and 10.5% for GBP/USD at $\mathbf{X}_s$5. Using the best 15-day window, the backtests report cumulative returns of 18.572%, 25.782%, and 18.373% for EUR/USD, USD/JPY, and GBP/USD respectively, with Sharpe ratios 2.087, 1.891, and 1.903; under 7 bps total frictions—5 bps transaction cost and 2 bps slippage—the corresponding cumulative returns remain 7.655%, 19.937%, and 9.216% (Liu et al., 14 Dec 2025).

The ablation results are also explicit: removing dynamic variable selection causes the largest performance drop; replacing trend-aware attention with standard attention hurts substantially; and removing SE or multi-scale convolution also degrades performance. This suggests that, within the paper’s experimental frame, predictive gains are attributed to the joint effect of adaptive covariate weighting and trend-contextual attention rather than to generic Transformer capacity alone.

5. Relation to ExoFormer

A separate architecture, ExoFormer, should be distinguished from the models named Exformer or EXFormer (Su, 13 Jan 2026). ExoFormer addresses a specific language-model design problem: when early-layer attention projections are reused as residual anchors across many layers, the first layer must function both as a stable reusable reference and as an effective computational block. The paper proposes to resolve this by learning dedicated exogenous anchor projections outside the sequential layer stack:

$\mathbf{X}_s$6

The architecture then applies a unified normalized mixing rule across all attention pathways $\mathbf{X}_s$7:

$\mathbf{X}_s$8

The abstract reports that ExoFormer variants consistently outperform internal-anchor counterparts, that the dynamic variant achieves a 2.13-point gain in downstream accuracy over the baseline, matches baseline validation loss with 1.84x fewer tokens, and achieves a 2x reduction in attention sink relative to standard Gated Attention (Su, 13 Jan 2026). The same paper also reports that ExoFormer variants exhibit signs of representation collapse, which the authors interpret through an Offloading Hypothesis: external anchors preserve token identity, freeing the sequential layers to specialize in computational refinement.

The relevance of ExoFormer to the Exformer name is nominal rather than genealogical. The spelling differs, the target task is different, and the core mechanism concerns normalized projection reuse in attention rather than extreme-event forecasting, speaker-conditioned extraction, or financial covariate selection.

6. Cross-domain themes, distinctions, and interpretive significance

Despite the heterogeneity of these models, a few common design tendencies are visible. First, each architecture modifies transformer computation to encode a domain prior that standard full self-attention is claimed not to express adequately. In hydrology, this prior is the asymmetry between normal and extreme events and the need to preserve extreme-to-extreme dependencies (Shrestha et al., 2 Jul 2026). In target speaker extraction, it is the requirement that separation be explicitly conditioned on a target speaker embedding and optimized using both labeled and unlabeled mixtures (Wang et al., 2022). In foreign exchange prediction, it is the claim that financial sequences must be aligned by local trend context and filtered through time-varying variable relevance (Liu et al., 14 Dec 2025).

Second, all three variants incorporate a structured sparsity or conditioning mechanism rather than relying on unmodified token-token attention. Exformer for hydrology uses query-type-dependent sparse masks. Exformer for TSE injects speaker information through fusion operators and dual-path chunked processing. EXFormer for FX uses variable selection and convolutional trend-aware projections. This suggests that the shared name is associated less with a standard architecture than with the broader practice of transformer specialization.

Third, the empirical claims are domain-specific and therefore not directly comparable. Hydrologic Exformer reports best MAPE on all four datasets and best RMSE on 3 of 4 datasets in 3-day streamflow forecasting (Shrestha et al., 2 Jul 2026). Speech Exformer reports 20.49 dB SI-SDR and 3.85 PESQ under a two-stage semi-supervised protocol on WSJ0-2mix-extr (Wang et al., 2022). Financial EXFormer reports statistically significant directional-accuracy gains and positive trading performance even after conservative frictions (Liu et al., 14 Dec 2025). These results cannot be ranked across domains, but they consistently support the narrower claim that domain-informed transformer modifications may outperform generic baselines under the evaluation protocols chosen by each paper.

A likely misconception is that Exformer names a single standardized backbone analogous to BERT or ViT. The supplied literature does not support that interpretation. Instead, it supports the narrower encyclopedic conclusion that Exformer is a reused model name applied to multiple transformer-derived systems, each tailored to a different problem formulation and each grounded in a different architectural intervention. A plausible implication is that citations using the term alone require disambiguation by full title, task, or arXiv identifier to avoid conflating unrelated methods.