HTMformer: Hybrid Time & Multivariate Forecasting

• HTMformer is a Transformer-based framework that integrates hybrid temporal and multivariate embeddings to improve time series forecasting.
• The model employs an inverted input design by treating variables as tokens, reducing complexity while enhancing inter-variable dynamics.
• Empirical results demonstrate state-of-the-art accuracy and efficiency, outperforming baselines on various high-dimensional forecasting benchmarks.

Searching arXiv for the primary paper and closely related works to ground the article in published sources. Searching arXiv for (Wang et al., 8 Oct 2025). HTMformer denotes a Transformer-based framework for multivariate time series forecasting introduced in "HTMformer: Hybrid Time and Multivariate Transformer for Time Series Forecasting" (Wang et al., 8 Oct 2025). It is designed around the claim that standard Transformer forecasters tend to overemphasize temporal dependencies and rely heavily on the embedding layer to learn effective representations. The model addresses this by replacing conventional embeddings with Hybrid Temporal and Multivariate Embeddings (HTME), a two-branch extractor that jointly mines temporal dynamics and inter-variable structure before attention, and by using an inverted input design in which variables rather than time steps are treated as tokens. In this formulation, HTMformer is an encoder-only, lightweight forecaster that aims to improve both accuracy and efficiency on high-dimensional multivariate forecasting benchmarks (Wang et al., 8 Oct 2025).

1. Conceptual motivation and problem formulation

HTMformer is motivated by three observations. First, many Transformer-based forecasters add increasingly complex mechanisms to model long-range temporal dependencies, yet temporal information is described as intrinsically limited: as the input length $L$ grows, loss reductions plateau or even degrade, while compute grows superlinearly (Wang et al., 8 Oct 2025). Second, forecasting accuracy is reported to be highly dependent on the embedding method used to learn effective representations. Third, real-world systems such as traffic networks, electricity grids, and weather stations exhibit strong inter-variable correlations, and modeling them is crucial.

The formal problem is multivariate time series forecasting with historical input $X \in \mathbb{R}^{L \times C}$, forecast horizon $H$, and ground truth $Y \in \mathbb{R}^{H \times C}$. A timestamp channel $T$ is appended to $X$ after normalization, so the effective input has $N = C + 1$ channels. The model outputs $\hat{Y} \in \mathbb{R}^{H \times C}$ (Wang et al., 8 Oct 2025).

The paper’s central position is that simply increasing the lookback length often adds computational overhead without yielding commensurate gains. HTMformer therefore shifts much of the representational burden from the attention stack to the embedding layer. This suggests a design philosophy different from approaches that primarily redesign attention mechanisms: feature extraction prior to attention is treated as the main bottleneck rather than as a peripheral preprocessing step.

2. Hybrid Temporal and Multivariate Embeddings

HTME is a two-branch embedding layer that replaces conventional embeddings and outputs $Y_{in} \in \mathbb{R}^{N \times D}$, where each variable, together with the timestamp channel, becomes a token with a $D$-dimensional embedding (Wang et al., 8 Oct 2025). A learnable fusion coefficient $X \in \mathbb{R}^{L \times C}$0 balances temporal and multivariate features.

The temporal feature extractor uses patch tokenization and local convolutions. The time series is segmented into temporal patches of length $X \in \mathbb{R}^{L \times C}$1 with stride $X \in \mathbb{R}^{L \times C}$2, and per-channel convolution is applied to capture short-term patterns within each patch’s local receptive field. Treating each variate independently is stated to reduce cross-channel interference while mining temporal context more accurately. Convolution outputs are then flattened and linearly projected, after which a second linear projection aggregates across patches and reshapes the result into the final per-variable embeddings:

$X \in \mathbb{R}^{L \times C}$3

$X \in \mathbb{R}^{L \times C}$4

$X \in \mathbb{R}^{L \times C}$5

This branch is summarized as $X \in \mathbb{R}^{L \times C}$6 (Wang et al., 8 Oct 2025).

The multivariate feature extractor uses the same patching scheme, but each patch is flattened across variables:

$X \in \mathbb{R}^{L \times C}$7

A linear projection across the variate dimension models inter-variable correlations within each patch, temporal convolution aggregates the resulting sequence, and a GRU applies gating to selectively emphasize or suppress inputs. A final 1D convolution expands the representation along the temporal dimension to match the output shape of the temporal branch:

$X \in \mathbb{R}^{L \times C}$8

$X \in \mathbb{R}^{L \times C}$9

This branch is summarized as $H$0 (Wang et al., 8 Oct 2025).

The complementary outputs are fused through

$H$1

with $H$2. The stated purpose of this fusion is to combine temporal features extracted by patching, convolution, and linear aggregation with multivariate features extracted by linear mixing, temporal convolution, and GRU gating, while keeping the embedding lightweight (Wang et al., 8 Oct 2025).

3. Architecture and computational design

HTMformer integrates HTME into an encoder-only Transformer pipeline. The full sequence is:

• RevIN normalization per batch.
• Concatenation of timestamps as an extra channel, $H$3, with $H$4.
• Hybrid embedding, $H$5, with $H$6.
• Encoder-only Transformer, $H$7, with $H$8.
• Linear projection head, $H$9, with $Y \in \mathbb{R}^{H \times C}$0 (Wang et al., 8 Oct 2025).

A defining element is the inverted input design: tokens correspond to variables rather than time steps. Since temporal features are largely extracted in HTME, self-attention in the encoder directly models global channel correlations. The paper states that this avoids forced alignment of variables at specific timestamps and reduces complexity from $Y \in \mathbb{R}^{H \times C}$1 to $Y \in \mathbb{R}^{H \times C}$2, where $Y \in \mathbb{R}^{H \times C}$3 is fixed by the number of variables plus the timestamp channel. Under this design, complexity is independent of lookback length $Y \in \mathbb{R}^{H \times C}$4 (Wang et al., 8 Oct 2025).

The attention block uses standard multi-head self-attention:

$Y \in \mathbb{R}^{H \times C}$5

$Y \in \mathbb{R}^{H \times C}$6

The encoder includes residual connections, layer normalization, and a position-wise feed-forward network. No positional encoding is used because timestamp information is introduced as an explicit channel and temporal structure is embedded by HTME (Wang et al., 8 Oct 2025).

The projection head is decoder-free. Following iTransformer, the paper states that a linear projection suffices to map variate-level representations to the forecast horizon, reducing parameters and runtime (Wang et al., 8 Oct 2025). A plausible implication is that HTMformer treats representation quality at the embedding and encoder stages as sufficient for forecasting without an autoregressive or sequence-to-sequence decoder.

4. Training objective, implementation, and experimental protocol

HTMformer is evaluated with both mean squared error and mean absolute error:

$Y \in \mathbb{R}^{H \times C}$7

$Y \in \mathbb{R}^{H \times C}$8

Models are evaluated on both metrics (Wang et al., 8 Oct 2025).

The experimental setup uses a fixed lookback $Y \in \mathbb{R}^{H \times C}$9. For long-term forecasting, horizons are $T$0. For short-term forecasting on PEMS, horizons are $T$1. RevIN is applied to each batch before HTME, and timestamps are appended as an additional variable (Wang et al., 8 Oct 2025).

Optimization uses Adam with learning rate $T$2, dropout $T$3, and batch size $T$4. The model configuration is $T$5, $T$6, $T$7, and $T$8. Training proceeds for 10 epochs with early stopping of patience 3. The implementation is in PyTorch, and experiments are conducted on an NVIDIA Quadro RTX 6000 GPU. Inference consists of an encoder-only forward pass followed by linear projection to $T$9, with no teacher forcing and no autoregressive decoding (Wang et al., 8 Oct 2025).

Eight real-world benchmarks are used: Electricity (ECL), Weather, Traffic, ETTh2, Solar-Energy, and PEMS03/04/08. The datasets span hourly, 15-minute, 10-minute, and 5-minute resolutions, with variable counts ranging from 7 to 862. Splits are ETT/PEMS 6:2:2 and others 7:1:2, following Time-Series-Library protocols (Wang et al., 8 Oct 2025).

5. Empirical performance and efficiency

On long-term forecasting with $X$0 and $X$1, HTMformer reports the following average MSE/MAE values across horizons (Wang et al., 8 Oct 2025):

Dataset	Average MSE / MAE	Setting
Electricity	0.185 / 0.272	Long-term
Weather	0.254 / 0.277	Long-term
Traffic	0.467 / 0.312	Long-term
ETTh2	0.379 / 0.399	Long-term
Solar-Energy	0.235 / 0.265	Long-term
PEMS03	0.289 / 0.369	Long-term
PEMS04	0.284 / 0.376	Long-term
PEMS08	0.508 / 0.448	Long-term

The paper states that HTMformer attains state of the art or second-best in most scenarios, with 26 out of 40 best MSE results and 29 out of 40 best MAE results (Wang et al., 8 Oct 2025).

On short-term forecasting for PEMS with $X$2 and $X$3, the reported averages are 0.138 / 0.245 for PEMS03, 0.138 / 0.249 for PEMS04, and 0.187 / 0.278 for PEMS08. HTMformer is described as consistently outperforming competing baselines and as improving over MultiPatchFormer by approximately 21.7% MSE and 12.0% MAE on average in short-term PEMS tasks (Wang et al., 8 Oct 2025).

The baseline set includes iTransformer, PatchTST, FEDformer, DLinear, WPMixer, MultiPatchFormer, TimeMixer, and SegRNN. In addition, HTME is presented as a drop-in module for several Transformer families, yielding average MSE gains of 35.8% for Transformer, 34.6% for Reformer, 43.6% for Informer, 31.9% for Flowformer, and 33.1% for Flashformer across datasets and horizons (Wang et al., 8 Oct 2025).

Efficiency is a major empirical theme. On eight datasets, HTMformer’s training time is reported as about one-third of MultiPatchFormer’s, and GPU memory usage as about 20%–45% of MultiPatchFormer’s. Parameters are described as roughly half of MultiPatchFormer’s, while accuracy is superior or comparable. Representative runtime and memory figures are:

• Electricity (ECL): 0.1004 vs 0.3319 s/iter; 1623 MB vs 7472 MB.
• Traffic: 0.2682 vs 0.9648 s/iter; 6221 MB vs 20860 MB.
• Solar: 0.0502 vs 0.1435 s/iter; 647 MB vs 3219 MB.

The average across datasets is reported as 0.101 s/iter and 1614 MB for HTMformer (Wang et al., 8 Oct 2025). These results are consistent with the model’s stated $X$4 encoder complexity and its emphasis on lightweight embedding design.

6. Ablation findings, limitations, and relation to adjacent work

The ablation study compares three variants: HTMformerV1, which uses only the temporal extractor; HTMformerV2, which uses only the multivariate extractor; and iTransV3, which augments iTransformer with HTME’s multivariate extractor in its embedding (Wang et al., 8 Oct 2025). The temporal branch alone already outperforms several baselines and often surpasses iTransformer, which the paper interprets as confirmation that patch+conv+linear provides effective temporal modeling. The multivariate branch alone is weaker on most datasets, although it is competitive on Solar and certain PEMS settings, indicating that multivariate features remain indispensable sources of information. Hybrid fusion in HTMformer and embedding-level multivariate augmentation in iTransV3 consistently improve results across datasets.

Hyperparameter sensitivity results indicate that the best settings are typically learning rate around $X$5, $X$6, $X$7, and $X$8, while increasing capacity beyond these values does not guarantee improvements (Wang et al., 8 Oct 2025).

The paper also identifies several limitations. HTMformer does not explicitly model spatiotemporal graphs or learn adjacency matrices; instead, multivariate correlations are described as being “weakly learned” via linear mixing, temporal convolution, GRU gating in the embedding, and channel-attention in the encoder. It further states that simply adding multivariate features to temporal embeddings cannot fully capture complex spatiotemporal dependencies, and that efficiently modeling dynamic inter-variable relations remains an open challenge (Wang et al., 8 Oct 2025). Future work is suggested in the direction of efficient and scalable methods such as learnable dynamic graphs or hybrid attention.

In the context of related work, HTMformer is positioned against prior Transformers such as Informer, FEDformer, Crossformer, PatchTST, and iTransformer, which mainly redesign attention mechanisms or input strategies to capture temporal dependencies (Wang et al., 8 Oct 2025). Its novelty is defined as the combination of a hybrid embedding, inverted input with variable tokens, a learnable fusion coefficient $X$9, and an encoder-only plus linear-head design.

The acronym also requires disambiguation. A distinct paper, "Human MotionFormer: Transferring Human Motions with Vision Transformers" (Liu et al., 2023), uses the alias “HTMformer” for a hierarchical ViT framework for one-shot human motion transfer rather than for time series forecasting. The two models are unrelated in domain, objectives, and architecture. In the forecasting literature, HTMformer refers specifically to the Hybrid Time and Multivariate Transformer introduced for multivariate time series forecasting (Wang et al., 8 Oct 2025).