---
title: Hybrid Temporal & Multivariate Embeddings
url: https://www.emergentmind.com/topics/hybrid-temporal-and-multivariate-embeddings-htme
type: topic
---

# Hybrid Temporal & Multivariate Embeddings

Searching arXiv for the named HTME paper and closely related hybrid temporal/multivariate embedding work.
Hybrid Temporal and Multivariate Embeddings (HTME) denotes an embedding paradigm for multivariate time-series and spatiotemporal learning in which temporal structure and cross-variable, cross-channel, or cross-region structure are modeled through distinct but coordinated representational pathways, then fused into a single latent representation. In the literature, the term is used explicitly by HTMformer, which defines HTME as an embedding extractor that combines a temporal feature extraction module and a multivariate feature extraction module before Transformer encoding [2510.07084]. Closely related work applies the same underlying principle without always using the exact term: InvDec separates temporal encoding from variate-level decoding for forecasting [2510.20302], B-TGAT combines ConvLSTM2D-based local spatiotemporal encoding with a graph-attention temporal bottleneck for clustering climate sequences [2509.13202], and HiTime constructs hierarchical temporal embeddings that are then aligned with semantic text embeddings for multivariate time-series classification [2410.18686]. Taken together, these works establish HTME as a design pattern for learning latent spaces that are temporally informative, multivariate-aware, and often task-adaptive.

## 1. Conceptual definition and scope

HTME emerged from a recurring limitation in sequence models for multivariate data: architectures that are strong at temporal modeling often underuse cross-variate structure, while architectures that emphasize inter-variable relations may weaken temporal encoding. InvDec states this trade-off directly: channel-independent models such as PatchTST are strong at temporal modeling but ignore variable correlations, whereas inverted, variate-attention models such as iTransformer model variable relationships well but sacrifice temporal attention [2510.20302]. HTMformer makes an analogous argument at the embedding level, asserting that existing Transformers for time series “tend to overemphasize temporal dependencies,” while inter-variable correlations in traffic, weather, electricity, and related settings are also highly informative [2510.07084].

Within this framing, HTME is not a single architecture but a family of representational strategies with a common principle: one component learns temporal dynamics, another learns multivariate structure, and a fusion mechanism combines them. In HTMformer, the temporal and multivariate branches are explicit modules fused by a learnable scalar $\alpha$ [2510.07084]. In InvDec, the separation is staged rather than parallel: temporal embeddings are formed first, then delayed variate embeddings and variate-wise self-attention create a distinct multivariate representation, and the two streams are combined through adaptive residual fusion [2510.20302]. In B-TGAT, ConvLSTM2D and U-Net-style compression learn localized spatiotemporal structure, while a graph-attention bottleneck transforms the deepest latent tensor into regime embeddings that capture broader spatial and temporal dependencies [2509.13202].

A plausible implication is that HTME is best understood as a representational doctrine rather than a single formal model class. The common requirement is that temporal information and multivariate information are not collapsed prematurely into one undifferentiated token stream.

## 2. Core architectural pattern

The most explicit formulation appears in HTMformer, where HTME is defined as an embedding extractor with two independent modules: a Temporal Feature Extractor and a Multivariate Feature Extractor [2510.07084]. After RevIN normalization and timestamp concatenation, the pipeline is

$$
X_{\text{in} = \text{CAT}(X,T), \qquad Y_{\text{in} = \text{HTMEE}(X_{\text{in}), \qquad Y_{\text{out} = \text{Encode}(Y_{\text{in}), \qquad Y = \text{Project}(Y_{\text{out})
$$

with $X \in \mathbb{R}^{L \times C}$, $Y_{\text{in} \in \mathbb{R}^{N \times D}$, and fusion defined as

$$
Y_{\text{in} = \alpha D_{\text{out} + (1-\alpha) V_{\text{out}
$$

where $D_{\text{out}$ is the temporal branch output and $V_{\text{out}$ is the multivariate branch output [2510.07084]. This is the clearest direct specification of HTME in the current literature.

InvDec implements the same principle sequentially rather than symmetrically. For each variable $c$, the series $\mathbf{x}_c \in \mathbb{R}^L$ is segmented into patches, projected into $D$-dimensional tokens, and encoded with self-attention over reshaped patch tokens, yielding a temporal representation $\mathbf{E} \in \mathbb{R}^{C \times P \times D}$ [2510.20302]. Only after temporal encoding does the model aggregate to one token per variable by global pooling,

$$
\mathbf{g}_c = \frac{1}{P} \sum_{i=1}^{P} \mathbf{z}_{c,i} \in \mathbb{R}^D, \qquad \mathbf{G}\in \mathbb{R}^{C\times D}
$$

and apply variate-wise self-attention to model cross-variable dependencies [2510.20302]. This “delayed variate embedding” design is central to its interpretation as HTME because variable identity is injected after temporal feature extraction rather than at the input.

B-TGAT generalizes the pattern to spatiotemporal tensors. Its encoder consumes a climate tensor $\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C}$ through stacked TimeDistributed ConvLSTM2D blocks and pooling, then reshapes the deepest tensor into graph node features for each time slice, $Z_t \in \mathbb{R}^{N \times F}$ with $N = H_4W_4$ [2509.13202]. The bottleneck is therefore both temporally indexed and spatially graph-structured. This suggests a broader interpretation of HTME in which “multivariate” includes not only channel relations but also graph-based region interactions.

## 3. Temporal embedding mechanisms

HTME methods differ substantially in how they construct temporal embeddings, but they share the assumption that temporal structure should be learned with dedicated operators rather than delegated entirely to a generic downstream attention stack.

HTMformer uses patching, convolution, flattening, and linear projection as its temporal feature extraction module. The paper states that the series is split into patches; each patch is treated as a token; a convolution extracts short-term temporal patterns; the output is flattened and linearly projected; and another projection along the temporal dimension models long-range correlations [2510.07084]. The branch is formalized as

$$
N_1^{4 \times k},\, N_2^{4 \times k},\, \ldots,\, N_{(L-k)N}^{4 \times k} = \operatorname{Conv}\bigl( \operatorname{Patch}(X_{\text{in}) \bigr)
$$

$$
E_1^{\text{dim},\, E_2^{\text{dim},\, \ldots,\, E_N^{\text{dim} = \operatorname{Linear}\bigl( \operatorname{Flatten}\bigl( N_1^{4 \times k},\, N_2^{4 \times k},\, \ldots,\, N_{(L-k)N}^{4 \times k} \bigr) \bigr)
$$

$$
D_{\text{out} = \operatorname{Linear}\bigl( E_1^{\text{dim},\, E_2^{\text{dim},\, \ldots,\, E_N^{\text{dim} \bigr)
$$

[2510.07084].

InvDec instead inherits a patch-based temporal encoder from PatchTST. For each variable, non-overlapping patches are projected as

$$
\mathbf{e}_{c,i} = \mathbf{W}_p \mathbf{p}_{c,i} + \mathbf{b}_p \in \mathbb{R}^D
$$

and standard self-attention and FFN blocks operate over the reshaped patch tokens [2510.20302]. The paper emphasizes that this preserves local temporal context while reducing sequence length and computational cost.

HiTime uses a hierarchical feature encoder with two Transformer-based branches: a data-specific encoder trained self-supervised through tokenization, masking, and reconstruction, and a task-specific encoder trained with labels to capture task-relevant long-term and short-term dependencies [2410.18686]. Their concatenation,

$$
\mathbf{Z} = \operatorname{Concat}[\mathbf{Encoder}_d(X), \mathbf{Encoder}_s(X)]
$$

is explicitly presented as the main feature-level hybrid embedding [2410.18686]. Here the temporal split is not between short-term and long-term modules in a single encoder, but between general temporal structure and task-discriminative temporal structure.

A more specialized antecedent is MEANTIME, which uses six temporal embeddings—three absolute and three relative—and assigns different temporal embeddings to different attention heads [2008.08273]. Its absolute embeddings are Day, Pos, and Con; its relative embeddings are Sin, Exp, and Log [2008.08273]. Although the paper is in sequential recommendation rather than general forecasting, it is a clear precursor to HTME-style multi-view temporal encoding because it rejects the sufficiency of a single positional embedding.

## 4. Multivariate embedding mechanisms

The multivariate component of HTME is the modeling of correlations across variables, channels, sensors, spatial nodes, or regions. In HTMformer, the multivariate feature extraction module is deliberately lightweight. The input is patched, flattened, passed through a linear layer to model inter-variable correlations, aggregated through GRU, and expanded by convolution:

$$
\left( P_1^{C_k}, P_2^{C_k}, \ldots, P_{L-k}^{C_k} \right) = \operatorname{Flatten}\bigl( \operatorname{Patch}(X_{\text{in}) \bigr)
$$

$$
M^N = \operatorname{GRU}\bigl( \operatorname{Linear}\bigl( P_1^{C_k}, P_2^{C_k}, \ldots, P_{L-k}^{C_k} \bigr) \bigr)
$$

$$
V_{\text{out} = \operatorname{Conv}\bigl( M^N \bigr)
$$

[2510.07084]. The paper explicitly characterizes this as a “weak learning” strategy for multivariate correlation modeling.

InvDec adopts a more direct variate-attention approach. After pooling patch-level temporal encodings into one token per variable, the decoder computes learnable variate-wise self-attention:

$$
\mathbf{Q}_v^{(\ell)}, \mathbf{K}_v^{(\ell)}, \mathbf{V}_v^{(\ell)} = \mathbf{H}^{(\ell-1)} \mathbf{W}_{Q_v}^{(\ell)}, \mathbf{H}^{(\ell-1)} \mathbf{W}_{K_v}^{(\ell)}, \mathbf{H}^{(\ell-1)} \mathbf{W}_{V_v}^{(\ell)}
$$

$$
\mathbf{A}_v^{(\ell)} = \text{softmax}\!\left(\frac{\mathbf{Q}_v^{(\ell)} (\mathbf{K}_v^{(\ell)})^\top}{\sqrt{d_k}\right)\mathbf{V}_v^{(\ell)}
$$

[2510.20302]. The decoder output is then projected and broadcast back across the patch dimension. This makes the variate dimension the primary axis of attention in the decoder.

B-TGAT uses graph structure rather than channel attention to encode multivariate or spatial interactions. At the bottleneck, a graph $\mathcal{G}_t = (\mathcal{V},\mathcal{E}_t)$ is built over spatial nodes using spatial adjacency or $k$-NN structure, and graph attention is applied as

$$
\tilde{Z}_t = \mathrm{softmax}\!\left(\phi(Z_t)\,\mathbf{a}\,\phi(Z_t)^\top\right) Z_t
$$

[2509.13202]. This allows adaptive weighting of neighbors rather than fixed spatial connectivity. The paper’s interpretation is that the bottleneck should not be treated as a mere vector; it should be interpreted as a spatiotemporal graph.

An even broader extension appears in HSMGNN, which embeds multivariate time series jointly in Euclidean space and on the SPD manifold through SCS embedding, then learns graph structure in both spaces before weighted fusion [2512.14023]. This suggests that HTME can extend beyond temporal-versus-variate separation to hybrid geometric representations of multivariate dependencies.

## 5. Fusion strategies and learning objectives

A defining property of HTME methods is that temporal and multivariate information are not merely extracted independently; they are recombined through explicit fusion rules, and training objectives are chosen so that the fused latent space remains useful for the target task.

HTMformer uses the simplest fusion rule: a learnable scalar $\alpha$ combines the two branches,

$$
Y_{\text{in} = \alpha D_{\text{out} + (1-\alpha) V_{\text{out}
$$

with the stated interpretation that datasets with stronger temporal structure can rely more on $D_{\text{out}}$, while datasets with richer inter-variable coupling can benefit more from $V_{\text{out}}$ [2510.07084].

InvDec uses adaptive residual fusion,

$$
\mathbf{Z}_{\text{fused} = \mathbf{Z}_{\text{enc} + \lambda \cdot \tilde{\mathbf{Z},
$$

where $\lambda \in [0,1]$ is a learnable or preset weight [2510.20302]. The paper explicitly states that smaller $\lambda$ is preferable for low-dimensional datasets, while larger $\lambda$ is more useful for high-dimensional datasets. This makes dimensionality a central condition governing temporal-versus-variate balance.

B-TGAT fuses representational aims through a joint objective that combines reconstruction and deep clustering. The decoder reconstructs the input sequence under

$$
\mathcal{L}_{\mathrm{rec} = \frac{1}{T}\sum_{t=1}^{T}\|X_t-\widehat{X}_t\|_2^2,
$$

while the clustering head applies Student’s $t$-distribution soft assignments $q_{ij}$, target distribution $p_{ij}$, and KL divergence

$$
\mathcal{L}_{\mathrm{clus} = \frac{1}{T}\sum_{i=1}^{T}\sum_{j=1}^{k} p_{ij}\log\frac{p_{ij}{q_{ij}.
$$

The total objective is

$$
\mathcal{L}_{\mathrm{total} = \mathcal{L}_{\mathrm{rec} + \lambda \mathcal{L}_{\mathrm{clus}, \quad \lambda>0.
$$

[2509.13202]. In this formulation, HTME is not only about representation fusion but also about latent-space shaping: the embedding must remain reconstructive and cluster-friendly.

HiTime adds a multimodal alignment dimension. After hierarchical temporal embedding generation, it aligns time-series and text embeddings via fine-grained and coarse-grained objectives, with total loss

$$
\mathcal{L} = \alpha \mathcal{L}_{\text{coarse} + \beta \mathcal{L}_{\text{fine}.
$$

[2410.18686]. This suggests that HTME can be extended into a tri-partite scheme: temporal embedding, multivariate-aware latent structure, and semantic alignment.

## 6. Empirical results and application domains

The empirical literature around HTME spans forecasting, clustering, classification, recommendation, and multimodal geospatial vision. The reported results consistently support the claim that separate temporal and multivariate embedding pathways are beneficial, especially as dimensionality or structural complexity increases.

| Method | Domain | Reported result |
|---|---|---|
| HTMformer | Time series forecasting | First in 26/40 MSE metrics and 29/40 MAE metrics on eight datasets [2510.07084] |
| InvDec-PatchTST | Multivariate forecasting | 20.9% MSE reduction on Electricity, 4.3% improvement on Weather, 2.7% gain on Traffic over PatchTST [2510.20302] |
| B-TGAT | Climate clustering | Best reported Silhouette, DB, I-CD, and RMSE on ERA5; best reported Silhouette, DB, CH, RMSE, and Variance on CARRA; best reported Silhouette, CH, RMSE, and I-CD on NCEP/NCAR [2509.13202] |
| HiTime | MTSC with LLMs | Average accuracy 0.8764 versus 0.8452 for the next best reported competitor, Reformer [2410.18686] |
| MEANTIME | Sequential recommendation | Best on all datasets and metrics, with up to 16.03% relative gain in NDCG@5 on Game [2008.08273] |

HTMformer is evaluated on Electricity, Weather, Traffic, ETTh2, Solar-Energy, PEMS03, PEMS04, and PEMS08, and is reported to be especially strong on high-dimensional datasets such as Traffic and Solar-Energy [2510.07084]. It also functions as a plug-in embedding strategy for Transformer, Reformer, Informer, Flowformer, and Flashformer, with average reported improvements of 35.8%, 34.6%, 43.6%, 31.9%, and 33.1%, respectively [2510.07084].

InvDec’s dimensionality analysis is particularly significant for the HTME concept. The paper states that the relative improvement over PatchTST increases with variable count, that cross-variate modeling becomes more valuable as the number of variables increases, and that gains are small on low-dimensional ETT datasets with $C=7$ [2510.20302]. This is one of the clearest empirical statements that the need for HTME is data-regime dependent rather than universal.

B-TGAT demonstrates that the same design principle applies beyond forecasting. On ERA5 it reports Silhouette 0.3268, DB 1.5009, I-CD 7.4839, and RMSE 13.2158 as best reported values; on CARRA it reports Silhouette 0.2767, DB 1.5089, CH 69.7729, RMSE 5.5424, and Variance 0.0105; on NCEP/NCAR it reports Silhouette 0.6541, CH 868.7555, RMSE 3.118, and I-CD 0.9098 [2509.13202]. The paper interprets these results as evidence of superior cluster separability, temporal stability, and alignment with known climate transitions.

## 7. Interpretation, related ideas, and open questions

The current literature supports several interpretations of HTME, while also revealing unresolved design questions.

First, HTME is not synonymous with “adding variable embeddings.” InvDec’s delayed variate embeddings are explicitly motivated by preserving temporal feature integrity, in contrast to injecting variable embeddings at the input stage as in TimeXer [2510.20302]. This suggests that the order in which temporal and multivariate signals are introduced matters. A common misconception is that any model with time tokens and channel tokens is already hybrid; the cited work instead emphasizes principled separation, then controlled fusion.

Second, HTME is not restricted to Transformer architectures. B-TGAT uses ConvLSTM2D, U-Net skip connections, graph attention, BiLSTM, and DEC-style clustering [2509.13202]. The 2023 “Temporal Embeddings” work for geospatial computer vision transforms activity time series to the frequency domain, compresses them with a contractive autoencoder, and outputs image-like channels for multimodal fusion with RGB, SAR, and graph embeddings [2401.08581]. This suggests that HTME can also be instantiated through spectral encoders, autoencoders, or graph pipelines rather than only through token self-attention.

Third, there is an unresolved question about the optimal fusion mechanism. Current methods use scalar weighting, residual addition, concatenation, or objective-level coupling. HTMformer uses a learnable scalar $\alpha$ [2510.07084]; InvDec uses residual fusion with $\lambda$ [2510.20302]; B-TGAT relies on the interaction of reconstruction and clustering losses [2509.13202]; HSMGNN uses weighted dual-space fusion with $w_s$ and $w_e$ [2512.14023]. This suggests that fusion remains method-specific rather than standardized.

Fourth, empirical evidence repeatedly indicates that hybridization is most beneficial in structurally rich settings. InvDec’s advantage grows with dataset dimensionality [2510.20302]. HTMformer is especially strong on high-dimensional datasets and can reduce the complexity of attention from $\mathcal{O}(L^2)$ to $\mathcal{O}(N^2)$ by using inverted input after HTME extraction [2510.07084]. B-TGAT is designed for high-dimensional multivariate spatiotemporal climate tensors where both local and global temporal dependencies matter [2509.13202]. A plausible implication is that HTME is most valuable when neither pure temporal encoding nor pure variate encoding can dominate.

Finally, the literature indicates several converging trajectories for future HTME research. One direction is multimodal semantic alignment, exemplified by HiTime’s dual-view alignment between time-series and text embeddings [2410.18686]. Another is hybrid geometry, exemplified by HSMGNN’s Euclidean-SPD manifold formulation [2512.14023]. A third is stronger task-dependent latent shaping, as in B-TGAT’s reconstruction-plus-clustering objective [2509.13202]. These lines of work suggest that HTME may continue to evolve from a two-stream embedding idea into a broader framework for structured latent-space design across time, variables, space, modality, and geometry.

Source: https://www.emergentmind.com/topics/hybrid-temporal-and-multivariate-embeddings-htme