Bi-directional Temporal Graph Attention Transformer

• The paper introduces B-TGAT, a bottleneck module that fuses graph-based spatial attention with bidirectional temporal summarization for unsupervised clustering of climate data.
• B-TGAT is embedded in a ConvLSTM U-Net autoencoder, where each time step is transformed into a spatial graph refined by attention before temporal encoding via a BiLSTM.
• Empirical evaluations on datasets like ERA5, CARRA, and NCEP/NCAR demonstrate enhanced cluster separability, temporal stability, and improved internal metrics such as Silhouette and RMSE.

Searching arXiv for the cited papers and related B-TGAT/TGAT works. Bi-directional Temporal Graph Attention Transformer (B-TGAT) is a climate-oriented spatiotemporal representation module introduced as the bottleneck of a deep clustering framework for multivariate spatial fields evolving over time. In the reported formulation, it is embedded inside a time-distributed hybrid U-Net autoencoder with ConvLSTM2D encoder–decoder blocks, and its purpose is to produce latent embeddings that are spatially aware, temporally discriminative, and suitable for unsupervised clustering of climate regimes. The model is presented as combining graph-based spatial modeling with attention-driven temporal encoding and bidirectional temporal context, but the explicit methodology specifies graph attention over per-time-step spatial graphs followed by bidirectional temporal summarization via a BiLSTM, rather than a fully formalized transformer stack in the canonical sequence-modeling sense (Nji et al., 16 Sep 2025).

1. Concept and problem setting

B-TGAT is designed for clustering high-dimensional multivariate spatiotemporal climate data whose structure is shaped by nonlinear dependence, evolving spatial interactions, non-stationarity, and both short- and long-range temporal dependencies. The target dataset is written as

$\{X_1, X_2, \dots, X_T\},$

where each observation $X_i$ is a stack of $n$ variables over a spatial grid $L \times W$. In the model specification, the input is also represented as a 5D tensor

$$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$

with batch size $B$, temporal length $T$, spatial dimensions $H,W$, and $C$ channels. The output is a partition of temporal observations into clusters $\{C_1,\dots,C_k\}$ with $X_i$0, intended to correspond to coherent climate regimes or transitions (Nji et al., 16 Sep 2025).

The method is motivated by limitations of conventional clustering and deep sequence models in this setting. The paper contrasts it with k-means, Gaussian mixture models, DBSCAN, hierarchical clustering, linear dimensionality reduction followed by clustering, CNN-only approaches, recurrent approaches such as LSTM or ConvLSTM used alone, and deep clustering methods such as DEC, DTC, and DSC. In that framing, B-TGAT is not a standalone clustering algorithm but the representational core of a broader end-to-end clustering system whose embeddings are optimized jointly for reconstruction and cluster separability (Nji et al., 16 Sep 2025).

A recurring theme in the formulation is that flattening spatiotemporal climate fields into tabular vectors discards spatial context, while purely local recurrent operators may not adaptively capture region-to-region interactions or long temporal dependencies. This motivates a bottleneck module that first treats each compressed time slice as a graph over spatial locations and then imposes bidirectional temporal encoding over the resulting sequence of graph-level representations. This suggests that B-TGAT is aimed primarily at offline temporal clustering, where whole observed sequences are available, rather than at strictly causal forecasting.

2. Position inside the hybrid autoencoder

The full architecture is a time-distributed hybrid U-Net autoencoder. Its high-level pipeline is: input spatiotemporal sequence, ConvLSTM2D encoder with downsampling, B-TGAT bottleneck, latent embedding projection, mirrored ConvLSTM2D decoder with U-Net skip connections, reconstruction head, and clustering head. Encoder features at level $X_i$1 are denoted

$X_i$2

with ConvLSTM and normalization written as

$X_i$3

$X_i$4

Spatial downsampling is performed with $X_i$5, while encoder channel capacities are reported as $X_i$6 (Nji et al., 16 Sep 2025).

At the deepest encoder level, the feature tensor

$X_i$7

is reshaped per time step into node features

$X_i$8

Each node corresponds to a spatial location in the compressed bottleneck feature map. For each time $X_i$9, a graph $n$0 is constructed, with sparse adjacency $n$1 formed by connecting spatial nodes through $n$2NN in feature space (Nji et al., 16 Sep 2025).

The decoder mirrors the encoder. At decoder level $n$3, skip-connected fusion is given by

$n$4

followed by

$n$5

The final reconstruction head is a ConvLSTM2D layer with $n$6 activation that outputs $n$7. The latent embedding used for clustering is reported as $n$8, so the effective bottleneck dimension used for clustering is 256 (Nji et al., 16 Sep 2025).

This architecture makes B-TGAT one component of a larger multiscale system. ConvLSTM2D supplies localized spatiotemporal feature extraction, skip connections preserve multiscale spatial detail, and B-TGAT is responsible for graph-based spatial reasoning plus bidirectional temporal abstraction at the compressed bottleneck. The paper explicitly presents these roles as complementary rather than interchangeable.

3. Internal mechanics of the B-TGAT bottleneck

The graph-attention part of B-TGAT is specified by first constructing a graph per time step from $n$9, then applying an attention-based spatial refinement:

$L \times W$0

where $L \times W$1 is a learned linear map and $L \times W$2 parameterizes pairwise compatibility. The intended semantics are adaptive weighting of spatial neighbors before aggregating node information. The paper defines the graph and adjacency explicitly, but it does not write the adjacency masking inside the attention equation, so the exact operational role of $L \times W$3 in the attention computation is not fully formalized (Nji et al., 16 Sep 2025).

After graph refinement, node-wise outputs are globally pooled to obtain a temporal sequence of graph-level representations,

$L \times W$4

The paper does not specify the exact pooling operator, but it states that the graph-refined node embeddings are pooled over nodes before temporal encoding. The temporal encoder is then given explicitly as

$L \times W$5

This is the clearest mathematical statement of the module’s bidirectionality: the time axis is encoded in both forward and backward directions via a bidirectional recurrent operator (Nji et al., 16 Sep 2025).

The module is described conceptually as a “Bi-directional Temporal Graph Attention Transformer,” and the conclusion refers to “bi-directional multihead attention with time encodings.” However, the methodology does not provide canonical transformer equations such as temporal query/key/value projections, feed-forward transformer sublayers, explicit positional or time encodings, or the number of attention heads. What is mathematically specified is a single graph-attention equation and a BiLSTM temporal summarizer. A common misconception is therefore to equate this B-TGAT directly with a fully specified temporal transformer stack; the paper itself supports a narrower reading in which the bottleneck combines graph attention with bidirectional temporal sequence modeling, and the transformer terminology is partly conceptual rather than fully algebraically instantiated (Nji et al., 16 Sep 2025).

This distinction matters for interpreting the model’s novelty. The graph component is explicit and spatial; the bidirectional component is explicit and temporal; the transformer label is justified by attention-driven encoding, but the exact transformer internals are only partially specified. A plausible implication is that the paper’s central contribution lies more in architectural composition for spatiotemporal clustering than in a new fully formalized transformer operator.

4. Clustering objective and optimization

The clustering mechanism follows the DEC-style template. After the encoder and B-TGAT bottleneck produce latent embeddings $L \times W$6, k-means is used to initialize centroids:

$L \times W$7

Soft assignments are then computed using the Student-$L \times W$8 kernel:

$L \times W$9

The target distribution is sharpened as

$$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$0

and the clustering loss is the KL divergence

$$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$1

The reconstruction loss is mean squared error,

$$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$2

and the total objective is

$$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$3

The implementation is described in terms of mse and kld loss heads, with $$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$4 realized through loss weighting (Nji et al., 16 Sep 2025).

Parameter updates are written using SGD with momentum:

$$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$5

where $$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$6 is the learning rate and $$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$7 the momentum coefficient. Training alternates between minibatch optimization and periodic refresh of the target distribution $$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$8, and convergence is monitored by the fraction of changed hard assignments,

$$\mathcal{X} \in \mathbb{R}^{B \times T \times H \times W \times C},$$9

Training halts when $B$0 falls below a tolerance threshold (Nji et al., 16 Sep 2025).

The paper also states in its conclusion that the joint loss couples reconstruction fidelity, cluster separability, and graph smoothness. However, no explicit graph smoothness term is provided in the mathematical formulation. That omission is significant: graph smoothness is part of the conceptual narrative, but not part of the stated objective as written.

5. Empirical behavior, interpretability, and limitations

The model is evaluated on three climate datasets: ERA5 Global Reanalysis, C3S Arctic Regional Reanalysis (CARRA), and NCEP/NCAR Reanalysis 1. Reported inputs include daily fields over one year, with examples such as CARRA at $B$1, ERA5 over a $B$2 grid with 7 variables, and NCEP/NCAR Reanalysis 1 represented both as $B$3 and as $B$4. Missing values are imputed using the overall mean, features are scaled with Min–Max normalization to $B$5, and the number of clusters is chosen with the distortion-score elbow method (Nji et al., 16 Sep 2025).

Evaluation uses six internal clustering metrics: Silhouette, Davies–Bouldin (DB), Calinski–Harabasz (CH), RMSE, Variance, and average inter-cluster distance (I-CD). On ERA5 with 7 clusters, B-TGAT reports Silhouette $B$6, DB $B$7, CH $B$8, RMSE $B$9, Variance $T$0, and I-CD $T$1. The paper notes that these are the best listed values for Silhouette, DB, RMSE, and I-CD, while CH is not the best among compared methods. On CARRA, the table reports 5 clusters, although another part of the text says $T$2 across experiments; B-TGAT reports Silhouette $T$3, DB $T$4, CH $T$5, RMSE $T$6, Variance $T$7, and I-CD $T$8, with best values on Silhouette, DB, RMSE, and Variance but not on CH or I-CD. On NCEP/NCAR Reanalysis 1 with 7 clusters, B-TGAT reports Silhouette $T$9, DB $H,W$0, CH $H,W$1, RMSE $H,W$2, Variance $H,W$3, and I-CD $H,W$4, with best values on Silhouette, CH, RMSE, and I-CD, while DB is not the best because DEC attains $H,W$5 (Nji et al., 16 Sep 2025).

The baseline set includes KMeans, HAC, DEC, DTC, and DSC. Across the three datasets, the paper argues that B-TGAT usually achieves the best or near-best values on the most emphasized internal metrics, particularly Silhouette, RMSE, and inter-cluster separation-related measures. It also claims superior temporal stability and alignment with known climate transitions, although the provided text does not include a separate quantitative table for those two properties (Nji et al., 16 Sep 2025).

Interpretability is framed around graph attention and evolving spatial interactions. The paper claims that the architecture can provide insights into complex spatiotemporal variability, teleconnections, regions undergoing rapid transitions, and nonlinear climate processes. Yet the supplied text does not include attention maps, interaction visualizations, or case-study analyses of specific climate regimes. The interpretability argument is therefore architecturally motivated rather than exhaustively demonstrated in the reported material.

Several limitations are explicit. The methodology omits learning rate, batch size, dropout, number of B-TGAT layers, number of attention heads, exact graph neighborhood size $H,W$6, target refresh interval, validation strategy, and early-stopping details. The individual contribution of bidirectionality, graph attention, ConvLSTM, and skip connections is not isolated by a formal ablation table in the provided text. The cluster-count reporting contains an inconsistency for CARRA. Most importantly, the mathematical specification of the “transformer” aspect is incomplete relative to the conceptual claims (Nji et al., 16 Sep 2025).

6. Relation to TGAT and adjacent temporal graph models

B-TGAT is best understood against the broader temporal graph learning literature. The original TGAT introduced continuous-time node embeddings $H,W$7, temporal neighborhood attention over timestamped interactions, and functional time encoding derived from Bochner’s theorem, establishing a causal, past-to-present framework for temporal graph representation learning (Xu et al., 2020). Relative to that foundation, the climate-oriented B-TGAT differs in task, data type, and formalism: it operates on spatial grids compressed into per-time-step graphs, uses bidirectional temporal summarization for offline clustering, and is embedded inside a ConvLSTM U-Net autoencoder rather than being a standalone event-based temporal graph encoder (Nji et al., 16 Sep 2025).

Several recent models are adjacent but not equivalent. TransformerG2G uses transformer self-attention over per-node snapshot histories to learn temporal graph embeddings with uncertainty quantification, but its attention is temporal-only and not graph-neighborhood attention (Varghese et al., 2023). TF-TGN reformulates TGAT-like temporal neighbor aggregation as causal Transformer-decoder sequence modeling and is explicitly unidirectional (Huang et al., 2024). TIDFormer is a continuous-time dynamic graph transformer with interaction-level self-attention, mixed-granularity temporal encoding, and bidirectional interaction encoding, but not bidirectional temporal attention over future and past events (Peng et al., 31 May 2025). AIS-TGNN applies spatial GAT-style message passing over daily AIS-derived graphs with forward temporal carry-over and is explicitly not a true bidirectional temporal graph attention transformer (Xue et al., 5 Mar 2026). TS-GATR for online signature verification combines graph attention and GRU-based temporal modeling, but its temporal module is not bidirectional and its transformer character is confined mainly to the graph side (Yuan et al., 22 Oct 2025).

Within that landscape, the term “Bi-directional Temporal Graph Attention Transformer” refers most directly to the climate-clustering architecture described above (Nji et al., 16 Sep 2025). Its distinguishing feature is not continuous-time temporal message passing in the TGAT sense, nor causal sequence reformulation in the TF-TGN sense, but a bottleneck that composes graph-based spatial attention with bidirectional temporal encoding inside a deep unsupervised spatiotemporal autoencoder. The paper therefore occupies a specific niche: not a canonical general-purpose event-based temporal graph transformer, but a domain-specialized architecture for clustering multivariate spatiotemporal climate data while preserving spatial context and temporal regime structure.