PMformer: Partial-Multivariate Transformer

• The paper introduces PMformer, a novel forecasting model that balances univariate and full-multivariate approaches by dynamically sampling feature subsets.
• It employs a dual-attention mechanism combining temporal and feature attention to reduce computational cost and mitigate noise accumulation.
• Empirical studies, including cryptocurrency prediction, show PMformer achieves lower MSE and improved robustness, highlighting the bias-variance tradeoff in feature dependency modeling.

Partial-Multivariate Transformer (PMformer) is a Transformer-based forecasting architecture designed for the regime between univariate and complete-multivariate time-series modeling. Rather than ignoring inter-feature structure, as in univariate models, or estimating dependencies across the full feature set, as in complete-multivariate models, PMformer models dependencies within dynamically sampled feature subsets of size $S$, where $1Lee et al., 2024) and was later applied to cryptocurrency return prediction in "Partial multivariate transformer as a tool for cryptocurrencies time series prediction" (Tokajuk et al., 22 Nov 2025).

1. Conceptual placement in multivariate forecasting

PMformer is motivated by a recurring empirical pattern in multivariate forecasting: more cross-feature information does not necessarily improve generalization. The original formulation distinguishes two extremes. Univariate models process each feature independently and therefore ignore potentially useful inter-feature dependencies. Complete-multivariate models estimate relationships over all $D$ features and can, in principle, exploit richer structure, but they may underperform because of overfitting, noise accumulation, and computational overhead. PMformer is defined as a middle ground that captures dependencies only within subsets of features (Lee et al., 2024).

In the cryptocurrency study, this motivation is restated in domain-specific terms. Daily returns for BTCUSDT and ETHUSDT are described as extremely volatile and often close to a random walk under univariate modeling. At the same time, full-multivariate modeling can amplify noise and overfit in high-dimensional, nonstationary crypto markets. PMformer is therefore used as a partial-multivariate strategy intended to balance signal extraction and noise suppression by training on strategically selected subsets of features rather than on price-only signals or the entire feature universe (Tokajuk et al., 22 Nov 2025).

A common misconception is to equate PMformer with a fixed external feature-selection pipeline. In the original PMformer formulation, subsets are not learned once and held fixed; they are dynamically generated by random sampling or random partitioning during training and inference. In the cryptocurrency study, the same principle appears as stochastic training on varied feature subsets per minibatch rather than as a separate filter, wrapper, or embedded selection procedure.

2. Formal definition and subset semantics

The general PMformer setup considers $D$ features and a subset size $S$. The subset pool is defined as

$$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$

Given a sampled subset $\mathbf{F}\in\mathcal{F}^{all}$, the forecaster predicts future values only for the selected subset: $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$ where $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$ are past observations and $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}}\in\mathbb{R}^{\tau\times S}$$ are the forecasts (Lee et al., 2024).

This formulation recovers the two limiting cases exactly. When $1

The cryptocurrency application adopts the same partial-subset semantics for one-step-ahead prediction of next-day logarithmic returns. Its objective is written as

$1

with loss

$1

In that setting, $1Tokajuk et al., 22 Nov 2025).

The dynamic-subset construction is essential. In the original PMformer paper, subsets vary at each iteration via either random sampling or random partitioning, and the same shared neural network is applied to every subset. The model does not learn a subset-assignment matrix. This design increases the effective variety of training instances while avoiding explicit global dependency estimation (Lee et al., 2024).

3. Architecture, tokenization, and training algorithm

PMformer is a dual-attention Transformer. In the original architecture, the $1segment $1

$1

where $D$0 and $D$1 are learnable time-wise and feature-wise embeddings. The token tensor for a subset is $D$2 (Lee et al., 2024).

Each encoder block combines temporal attention and feature attention. Temporal attention operates independently for each feature across segments, while feature attention operates independently for each segment across the $D$3 selected features. The block is expressed as

$D$4

$D$5

The decoder flattens or concatenates the final segmentwise representation for each feature and applies a linear projection to the forecast horizon (Lee et al., 2024).

The cryptocurrency paper presents the same architectural principle in a notation adapted to daily return forecasting. Its encoder is summarized by

$D$6

where temporal self-attention acts along time within each selected feature, feature-wise self-attention acts across the selected features at each time step, and a linear head maps the final representation to the one-step-ahead prediction (Tokajuk et al., 22 Nov 2025).

Training in the original PMformer paper uses mean squared error with a shared network across sampled subsets. Two subset-generation schemes are specified. Random sampling draws subsets independently from $D$7. Random partitioning, the default scheme, partitions the full feature index set into disjoint subsets of size $D$8, covering all features once per iteration; if $D$9 is not divisible by $D$0, repeated indices are added to the last subset and removed from the decoded outputs afterward. The optimizer is Adam, and the reported implementation uses lookbacks $D$1, segment counts $D$2, hidden sizes $D$3, heads $D$4, layers $D$5, feed-forward sizes $D$6, dropout $D$7, learning rate $D$8, and 100 epochs (Lee et al., 2024).

In the cryptocurrency study, the subset mechanism is explicitly described as randomly partitioned feature subsets in each minibatch, used to improve generalization and mitigate overfitting to particular cross-feature interactions. The study does not explicitly state the optimizer, learning-rate schedule, early stopping, or layer normalization, but it reports Bayesian hyperparameter search with validation selection by MSE, 100 Bayesian trials per model, averaging over seeds $D$9, execution on an NVIDIA T4 GPU, and tracking via Weights & Biases (Tokajuk et al., 22 Nov 2025).

4. Theoretical rationale, efficiency, and robustness

The original PMformer paper supplements the architectural proposal with a PAC-Bayes argument. Under assumptions including bounded outputs, near-zero empirical loss, and non-informative priors, it adapts McAllester’s bound as

$S$0

The analysis relates the subset size $S$1 to two quantities: the effective number of subset-instances $S$2, which grows with $S$3 up to $S$4, and the posterior entropy $S$5, which the paper argues decreases as $S$6 increases. The resulting rationale is that the best $S$7 typically lies between the univariate and complete-multivariate extremes, often in the range $S$8 (Lee et al., 2024).

The same paper reports empirical support for this argument. Sensitivity curves in $S$9 are U-shaped, with the poorest performance near $$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$0 and the best MSE often attained at intermediate subset sizes. Increasing the number of distinct subsets also lowers MSE. On standard benchmarks—ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, and Traffic, with horizons $$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$1—PMformer achieves top-1 MSE in 27 of 28 tasks and second place in the remaining task against the main baselines. Against concurrent baselines on selected settings, it attains top-1 in 10 and top-2 in 12 out of 12, with average rank approximately $$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$2 (Lee et al., 2024).

Efficiency gains arise from restricting feature attention to subsets. In the original segmented formulation, inter-feature attention scales as

$$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$3

rather than $$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$4 for complete-multivariate attention. The cryptocurrency paper reports the per-block dual-attention cost as approximately

$$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$5

plus MLP costs $$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$6, and notes that smaller $$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$7 reduces feature-attention cost relative to full multivariate modeling (Lee et al., 2024, Tokajuk et al., 22 Nov 2025).

Robustness under missing features is another explicit property of the original PMformer. At inference time, missing features are simply excluded from the partitioning, and forecasts are produced only for observed features; no global padding is required. The paper reports that dropping a fraction of features increases test MSE much less for PMformer than for the complete-multivariate variant with $$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$8. The cryptocurrency study does not explicitly discuss missing-data handling, so this robustness claim belongs to the general PMformer framework rather than to the crypto-specific evaluation (Lee et al., 2024).

5. Cryptocurrency return forecasting instantiation

The cryptocurrency study instantiates PMformer for forecasting next-day logarithmic returns

$$\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.$$9

for BTCUSDT and ETHUSDT using Binance daily data from Oct 5, 2017 to May 20, 2025. Preprocessing consists of log-return computation for the target, Min–Max normalization to $\mathbf{F}\in\mathcal{F}^{all}$0 for all numeric features fitted on the training split and applied to validation and test sets, and a chronological split of 70% train, 20% validation, and 10% test. Extreme movements are retained, explicit stationarity tests are not performed, and missing-data handling is not explicitly discussed (Tokajuk et al., 22 Nov 2025).

The feature universe is deliberately compact. It includes OHLC, base and quote volumes, trade count, and the technical indicators SMA(50), EMA(21), RSI(14), CCI(20), ATR(14), and MACD(12,26,9) components. The paper explicitly notes that it does not include on-chain metrics, cross-asset signals, or macro proxies. Strategic selection is implemented through random subset sampling during training rather than through an external feature-selection procedure (Tokajuk et al., 22 Nov 2025).

The experimental comparison comprises eleven baselines: Naive (Previous Result/No-change), ARIMA, Univariate LSTM, Multivariate LSTM, Transformer, Autoformer, Informer, FEDformer, PatchTST, iTransformer, and DLinear. Statistical evaluation uses MSE, RMSE, and MAE, with directional accuracy defined as

$\mathbf{F}\in\mathcal{F}^{all}$1

Practical trading utility is assessed through a simple sign-based strategy: long if $\mathbf{F}\in\mathcal{F}^{all}$2, short otherwise, with daily rebalancing. The paper reports ROI, daily Sharpe ratio, max drawdown, and directional accuracy; position sizing, slippage, and transaction costs are not specified (Tokajuk et al., 22 Nov 2025).

The best reported PMformer configurations differ materially by asset. For BTC, the selected configuration is approximately LR $\mathbf{F}\in\mathcal{F}^{all}$3, BS $\mathbf{F}\in\mathcal{F}^{all}$4, SL $\mathbf{F}\in\mathcal{F}^{all}$5, LL $\mathbf{F}\in\mathcal{F}^{all}$6, $\mathbf{F}\in\mathcal{F}^{all}$7, $\mathbf{F}\in\mathcal{F}^{all}$8, Dim $\mathbf{F}\in\mathcal{F}^{all}$9, $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$0, $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$1, $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$2. For ETH, it is approximately LR $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$3, BS $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$4, SL $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$5, LL $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$6, $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$7, $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$8, Dim $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),$$9, $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$0, $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$1, $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$2. No explicit ablations over subset size, depth, heads, or window length are reported, but the differing selected sequence lengths suggest asset-specific temporal dependencies (Tokajuk et al., 22 Nov 2025).

6. Empirical findings, interpretive issues, and limitations

The cryptocurrency study reports a strong statistical performance for PMformer on both assets, but also emphasizes a disconnect between forecasting error and simulated trading utility. The central empirical pattern is summarized below (Tokajuk et al., 22 Nov 2025).

Asset	Statistical result	Trading result
BTCUSDT	Lowest MSE $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$3, RMSE $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$4, MAE $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$5	ROI 20.62%, Sharpe 3.83, MDD $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$6, DA 59.8%; highest DA and lowest MDD
ETHUSDT	Lowest MSE $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$7, RMSE $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$8, MAE $$\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}$$9	ROI $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}}\in\mathbb{R}^{\tau\times S}$$0, Sharpe $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}}\in\mathbb{R}^{\tau\times S}$$1, MDD $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}}\in\mathbb{R}^{\tau\times S}$$2, DA 47.9%

For BTCUSDT, PMformer attains the lowest MSE and the best directional accuracy and drawdown profile, yet FEDformer exceeds it in ROI and Sharpe, with ROI 38.08%, Sharpe 4.54, and MDD $$\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}}\in\mathbb{R}^{\tau\times S}$$3. For ETHUSDT, PMformer again attains the lowest MSE, RMSE, and MAE, but its trading simulation is negative; across models, ROI is generally negative, while Autoformer is slightly positive with ROI 0.59% and Sharpe 0.68 (Tokajuk et al., 22 Nov 2025).

This result directly challenges the assumption that minimizing conventional pointwise error is sufficient for financial deployment. The paper attributes the gap to properties of trading objectives: payoffs are nonlinear and asymmetric; regime shifts and fat tails can break the mapping from small error improvements to positive PnL; calibration and sign stability matter; and unmodeled frictions such as transaction costs and slippage may erase weak statistical gains. The cryptocurrency paper therefore recommends utility-aware objectives such as directional loss, asymmetric penalties, differentiable Sharpe or Sortino, cost-aware training, or reinforcement learning for policy optimization (Tokajuk et al., 22 Nov 2025).

Several limitations are explicit. In the general PMformer paper, subset discovery is random rather than learned, which may be suboptimal when very strong global dependencies dominate. The PAC-Bayes analysis relies on proxy assumptions and does not compute posterior entropy exactly. In the cryptocurrency application, feature coverage is restricted to technical and microstructure variables; on-chain, cross-asset, and macro features are omitted; formal significance tests such as Diebold–Mariano are not reported; regime-robustness checks are not explicitly discussed; and the model is optimized for MSE rather than for trading utility (Lee et al., 2024, Tokajuk et al., 22 Nov 2025).

A second common misconception is that partial-multivariate modeling is simply a weaker version of complete-multivariate forecasting. The reported evidence suggests a more specific interpretation: PMformer seeks a bias-variance trade-off in which incomplete cross-feature modeling can generalize better than both extremes. A plausible implication is that the relevant design question is not whether to use inter-feature information, but how much of it to expose to the model at once, and under what inference and evaluation regime.