---
title: 'PMformer: Partial-Multivariate Transformer'
url: https://www.emergentmind.com/topics/partial-multivariate-transformer-pmformer
type: topic
---

# PMformer: Partial-Multivariate Transformer

Partial-Multivariate Transformer (PMformer) is a Transformer-based forecasting architecture designed for the regime between univariate and complete-multivariate time-series modeling. Rather than ignoring inter-feature structure, as in univariate models, or estimating dependencies across the full feature set, as in complete-multivariate models, PMformer models dependencies within dynamically sampled feature subsets of size \(S\), where \(1<S<D\). The formulation was introduced as a general forecasting framework in "Partial-Multivariate Model for Forecasting" [2408.09703] and was later applied to cryptocurrency return prediction in "Partial multivariate transformer as a tool for cryptocurrencies time series prediction" [2512.04099].

## 1. Conceptual placement in multivariate forecasting

PMformer is motivated by a recurring empirical pattern in multivariate forecasting: more cross-feature information does not necessarily improve generalization. The original formulation distinguishes two extremes. Univariate models process each feature independently and therefore ignore potentially useful inter-feature dependencies. Complete-multivariate models estimate relationships over all \(D\) features and can, in principle, exploit richer structure, but they may underperform because of overfitting, noise accumulation, and computational overhead. PMformer is defined as a middle ground that captures dependencies only within subsets of features [2408.09703].

In the cryptocurrency study, this motivation is restated in domain-specific terms. Daily returns for BTCUSDT and ETHUSDT are described as extremely volatile and often close to a random walk under univariate modeling. At the same time, full-multivariate modeling can amplify noise and overfit in high-dimensional, nonstationary crypto markets. PMformer is therefore used as a partial-multivariate strategy intended to balance signal extraction and noise suppression by training on strategically selected subsets of features rather than on price-only signals or the entire feature universe [2512.04099].

A common misconception is to equate PMformer with a fixed external feature-selection pipeline. In the original PMformer formulation, subsets are not learned once and held fixed; they are dynamically generated by random sampling or random partitioning during training and inference. In the cryptocurrency study, the same principle appears as stochastic training on varied feature subsets per minibatch rather than as a separate filter, wrapper, or embedded selection procedure.

## 2. Formal definition and subset semantics

The general PMformer setup considers \(D\) features and a subset size \(S\). The subset pool is defined as
\[
\mathcal{F}^{all} \coloneqq \{\mathbf{F} \subset [0:D] \mid |\mathbf{F}|=S\}.
\]
Given a sampled subset \(\mathbf{F}\in\mathcal{F}^{all}\), the forecaster predicts future values only for the selected subset:
\[
\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}} = f\big(\mathbf{x}_{[0:T],\mathbf{F}},\, \mathbf{F}\big),
\]
where \(\mathbf{x}_{[0:T],\mathbf{F}}\in\mathbb{R}^{T\times S}\) are past observations and \(\hat{\mathbf{x}}_{[T:T+\tau],\mathbf{F}}\in\mathbb{R}^{\tau\times S}\) are the forecasts [2408.09703].

This formulation recovers the two limiting cases exactly. When \(S=1\), the model becomes univariate. When \(S=D\), it becomes complete-multivariate. PMformer is therefore not a separate forecasting family in the sense of changing the basic supervision problem; it changes the granularity at which cross-feature structure is estimated.

The cryptocurrency application adopts the same partial-subset semantics for one-step-ahead prediction of next-day logarithmic returns. Its objective is written as
\[
\hat{x}_{T+1,\mathcal F} = f_\theta\!\left(X_{0:T,\mathcal F},\,\mathcal F\right),
\]
with loss
\[
\mathcal{L}(\theta) = \mathbb{E}_{t,\mathcal F}\big[\,\|\hat{x}_{t+1,\mathcal F}-x_{t+1,\mathcal F}\|^2\,\big].
\]
In that setting, \(X_{t-L+1:t,\mathcal F}\in\mathbb{R}^{L\times S}\) denotes a lookback slice over the selected feature subset, and the target is the next-day return rather than a multi-step trajectory [2512.04099].

The dynamic-subset construction is essential. In the original PMformer paper, subsets vary at each iteration via either random sampling or random partitioning, and the same shared neural network is applied to every subset. The model does not learn a subset-assignment matrix. This design increases the effective variety of training instances while avoiding explicit global dependency estimation [2408.09703].

## 3. Architecture, tokenization, and training algorithm

PMformer is a dual-attention Transformer. In the original architecture, the \(T\)-length input for each selected feature is segmented into \(N_S\) equal segments. For feature \(\mathbf{F}_i\) and segment \(b\), tokenization is defined by
\[
\mathbf{h}^{(0)}_{b,i} = \mathrm{Linear}\!\big(\mathbf{x}_{[\frac{bT}{N_S}:\frac{(b+1)T}{N_S}],\,\mathbf{F}_i}\big) + \mathbf{e}^{Time}_b + \mathbf{e}^{Feat}_{\mathbf{F}_i},
\]
where \(\mathbf{e}^{Time}\in\mathbb{R}^{N_S\times d_h}\) and \(\mathbf{e}^{Feat}\in\mathbb{R}^{D\times d_h}\) are learnable time-wise and feature-wise embeddings. The token tensor for a subset is \(\mathbf{h}^{(0)}\in\mathbb{R}^{N_S\times S\times d_h}\) [2408.09703].

Each encoder block combines temporal attention and feature attention. Temporal attention operates independently for each feature across segments, while feature attention operates independently for each segment across the \(S\) selected features. The block is expressed as
\[
\bar{\mathbf{h}}^{(\ell-1)} = \mathbf{h}^{(\ell-1)} + \mathrm{Feature\text{-}Attention}\!\left(\mathbf{h}^{(\ell-1)},\, \mathrm{Temporal\text{-}Attention}\!\left(\mathbf{h}^{(\ell-1)}\right)\right),
\]
\[
\mathbf{h}^{(\ell)} = \bar{\mathbf{h}}^{(\ell-1)} + \mathrm{MLP}\!\left(\bar{\mathbf{h}}^{(\ell-1)}\right).
\]
The decoder flattens or concatenates the final segmentwise representation for each feature and applies a linear projection to the forecast horizon [2408.09703].

The cryptocurrency paper presents the same architectural principle in a notation adapted to daily return forecasting. Its encoder is summarized by
\[
H^{(l)} = H^{(l-1)} + \mathrm{MLP}\!\Big(\mathrm{FA}\big(\mathrm{TA}(H^{(l-1)})\big)\Big),
\]
where temporal self-attention acts along time within each selected feature, feature-wise self-attention acts across the selected features at each time step, and a linear head maps the final representation to the one-step-ahead prediction [2512.04099].

Training in the original PMformer paper uses mean squared error with a shared network across sampled subsets. Two subset-generation schemes are specified. Random sampling draws subsets independently from \(\mathcal{F}^{all}\). Random partitioning, the default scheme, partitions the full feature index set into disjoint subsets of size \(S\), covering all features once per iteration; if \(D\) is not divisible by \(S\), repeated indices are added to the last subset and removed from the decoded outputs afterward. The optimizer is Adam, and the reported implementation uses lookbacks \(T\in\{512,1024\}\), segment counts \(N_S\in\{8,16,32,64\}\), hidden sizes \(d_h\in\{32,64,128,256,512\}\), heads \(H\in\{2,4,8,16\}\), layers \(L\in\{1,2,3\}\), feed-forward sizes \(d_{ff}\in\{32,64,128,256,512\}\), dropout \(r\in\{0.1,0.2,0.3,0.4,0.7\}\), learning rate \(1\mathrm{e}{-3}\), and 100 epochs [2408.09703].

In the cryptocurrency study, the subset mechanism is explicitly described as randomly partitioned feature subsets in each minibatch, used to improve generalization and mitigate overfitting to particular cross-feature interactions. The study does not explicitly state the optimizer, learning-rate schedule, early stopping, or layer normalization, but it reports Bayesian hyperparameter search with validation selection by MSE, 100 Bayesian trials per model, averaging over seeds \(\{42,1337,2025\}\), execution on an NVIDIA T4 GPU, and tracking via Weights & Biases [2512.04099].

## 4. Theoretical rationale, efficiency, and robustness

The original PMformer paper supplements the architectural proposal with a PAC-Bayes argument. Under assumptions including bounded outputs, near-zero empirical loss, and non-informative priors, it adapts McAllester’s bound as
\[
l(\mathbf{Q}) \le \sqrt{ \frac{-H(\mathbf{Q}) + \log \frac{1}{\delta} + \frac{5}{2}\log m + 8 + C}{2m - 1} }.
\]
The analysis relates the subset size \(S\) to two quantities: the effective number of subset-instances \(m\), which grows with \(|\mathcal{F}^{all}|=\binom{D}{S}\) up to \(D/2\), and the posterior entropy \(H(\mathbf{Q})\), which the paper argues decreases as \(S\) increases. The resulting rationale is that the best \(S\) typically lies between the univariate and complete-multivariate extremes, often in the range \(1<S<D/2\) [2408.09703].

The same paper reports empirical support for this argument. Sensitivity curves in \(S\) are U-shaped, with the poorest performance near \(S\in\{1,D\}\) and the best MSE often attained at intermediate subset sizes. Increasing the number of distinct subsets also lowers MSE. On standard benchmarks—ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, and Traffic, with horizons \(\tau\in\{96,192,336,720\}\)—PMformer achieves top-1 MSE in 27 of 28 tasks and second place in the remaining task against the main baselines. Against concurrent baselines on selected settings, it attains top-1 in 10 and top-2 in 12 out of 12, with average rank approximately \(1.17\) [2408.09703].

Efficiency gains arise from restricting feature attention to subsets. In the original segmented formulation, inter-feature attention scales as
\[
\mathcal{O}(N_S \cdot S \cdot D \cdot d),
\]
rather than \(\mathcal{O}(N_S \cdot D^2 \cdot d)\) for complete-multivariate attention. The cryptocurrency paper reports the per-block dual-attention cost as approximately
\[
\mathcal{O}(S\cdot L^2\cdot d + L\cdot S^2\cdot d)
\]
plus MLP costs \(\mathcal{O}(L\cdot S\cdot d\cdot d_{ff})\), and notes that smaller \(S\) reduces feature-attention cost relative to full multivariate modeling [2408.09703] [2512.04099].

Robustness under missing features is another explicit property of the original PMformer. At inference time, missing features are simply excluded from the partitioning, and forecasts are produced only for observed features; no global padding is required. The paper reports that dropping a fraction of features increases test MSE much less for PMformer than for the complete-multivariate variant with \(S=D\). The cryptocurrency study does not explicitly discuss missing-data handling, so this robustness claim belongs to the general PMformer framework rather than to the crypto-specific evaluation [2408.09703].

## 5. Cryptocurrency return forecasting instantiation

The cryptocurrency study instantiates PMformer for forecasting next-day logarithmic returns
\[
r_t = \ln(P_t/P_{t-1})
\]
for BTCUSDT and ETHUSDT using Binance daily data from Oct 5, 2017 to May 20, 2025. Preprocessing consists of log-return computation for the target, Min–Max normalization to \([0,1]\) for all numeric features fitted on the training split and applied to validation and test sets, and a chronological split of 70% train, 20% validation, and 10% test. Extreme movements are retained, explicit stationarity tests are not performed, and missing-data handling is not explicitly discussed [2512.04099].

The feature universe is deliberately compact. It includes OHLC, base and quote volumes, trade count, and the technical indicators SMA(50), EMA(21), RSI(14), CCI(20), ATR(14), and MACD(12,26,9) components. The paper explicitly notes that it does not include on-chain metrics, cross-asset signals, or macro proxies. Strategic selection is implemented through random subset sampling during training rather than through an external feature-selection procedure [2512.04099].

The experimental comparison comprises eleven baselines: Naive (Previous Result/No-change), ARIMA, Univariate LSTM, Multivariate LSTM, Transformer, Autoformer, Informer, FEDformer, PatchTST, iTransformer, and DLinear. Statistical evaluation uses MSE, RMSE, and MAE, with directional accuracy defined as
\[
\mathrm{DA} = \frac{1}{N} \sum_{t=1}^{N} \mathbf{1}\{\operatorname{sgn}(r_t) = \operatorname{sgn}(\hat{r}_t)\}.
\]
Practical trading utility is assessed through a simple sign-based strategy: long if \(\hat{r}_{t+1}>0\), short otherwise, with daily rebalancing. The paper reports ROI, daily Sharpe ratio, max drawdown, and directional accuracy; position sizing, slippage, and transaction costs are not specified [2512.04099].

The best reported PMformer configurations differ materially by asset. For BTC, the selected configuration is approximately LR \(=1.06\times10^{-4}\), BS \(=128\), SL \(=48\), LL \(=12\), \(e=4\), \(d=3\), Dim \(=64\), \(H=16\), \(d_{ff}=128\), \(D=0.7\). For ETH, it is approximately LR \(=3.76\times10^{-4}\), BS \(=64\), SL \(=192\), LL \(=96\), \(e=3\), \(d=3\), Dim \(=512\), \(H=16\), \(d_{ff}=512\), \(D=0.4\). No explicit ablations over subset size, depth, heads, or window length are reported, but the differing selected sequence lengths suggest asset-specific temporal dependencies [2512.04099].

## 6. Empirical findings, interpretive issues, and limitations

The cryptocurrency study reports a strong statistical performance for PMformer on both assets, but also emphasizes a disconnect between forecasting error and simulated trading utility. The central empirical pattern is summarized below [2512.04099].

| Asset | Statistical result | Trading result |
|---|---|---|
| BTCUSDT | Lowest MSE \(6.5798\times10^{-4}\), RMSE \(\approx 2.5651\times10^{-2}\), MAE \(\approx 1.8919\times10^{-2}\) | ROI 20.62%, Sharpe 3.83, MDD \(-13.8\%\), DA 59.8%; highest DA and lowest MDD |
| ETHUSDT | Lowest MSE \(9.6063\times10^{-4}\), RMSE \(\approx 3.0994\times10^{-2}\), MAE \(\approx 2.1711\times10^{-2}\) | ROI \(-0.67\%\), Sharpe \(-0.84\), MDD \(-75.7\%\), DA 47.9% |

For BTCUSDT, PMformer attains the lowest MSE and the best directional accuracy and drawdown profile, yet FEDformer exceeds it in ROI and Sharpe, with ROI 38.08%, Sharpe 4.54, and MDD \(-15.1\%\). For ETHUSDT, PMformer again attains the lowest MSE, RMSE, and MAE, but its trading simulation is negative; across models, ROI is generally negative, while Autoformer is slightly positive with ROI 0.59% and Sharpe 0.68 [2512.04099].

This result directly challenges the assumption that minimizing conventional pointwise error is sufficient for financial deployment. The paper attributes the gap to properties of trading objectives: payoffs are nonlinear and asymmetric; regime shifts and fat tails can break the mapping from small error improvements to positive PnL; calibration and sign stability matter; and unmodeled frictions such as transaction costs and slippage may erase weak statistical gains. The cryptocurrency paper therefore recommends utility-aware objectives such as directional loss, asymmetric penalties, differentiable Sharpe or Sortino, cost-aware training, or reinforcement learning for policy optimization [2512.04099].

Several limitations are explicit. In the general PMformer paper, subset discovery is random rather than learned, which may be suboptimal when very strong global dependencies dominate. The PAC-Bayes analysis relies on proxy assumptions and does not compute posterior entropy exactly. In the cryptocurrency application, feature coverage is restricted to technical and microstructure variables; on-chain, cross-asset, and macro features are omitted; formal significance tests such as Diebold–Mariano are not reported; regime-robustness checks are not explicitly discussed; and the model is optimized for MSE rather than for trading utility [2408.09703] [2512.04099].

A second common misconception is that partial-multivariate modeling is simply a weaker version of complete-multivariate forecasting. The reported evidence suggests a more specific interpretation: PMformer seeks a bias-variance trade-off in which incomplete cross-feature modeling can generalize better than both extremes. A plausible implication is that the relevant design question is not whether to use inter-feature information, but how much of it to expose to the model at once, and under what inference and evaluation regime.

Source: https://www.emergentmind.com/topics/partial-multivariate-transformer-pmformer