CT-PatchTST: Channel-Time Transformer

• The paper demonstrates that CT-PatchTST reduces MSE by approximately 7.6% over PatchTST for long-horizon renewable energy forecasting.
• CT-PatchTST is a deep learning model that integrates channel and time attention to effectively capture both interdependencies and temporal patterns in multivariate data.
• Empirical evaluations on Danish wind and solar datasets show improved forecast accuracy and robustness across varying patch configurations and forecast horizons.

The Channel-Time Patch Time-Series Transformer (CT-PatchTST) is a deep learning architecture specifically designed for long-term forecasting of multivariate renewable energy generation time series. It extends the Patch Time-Series Transformer (PatchTST) by introducing an inter-channel self-attention mechanism prior to the standard patch-wise temporal Transformer, enabling effective modeling of both channel interdependencies and temporal patterns. The method is evaluated on large-scale Danish offshore wind, onshore wind, and solar generation datasets, exhibiting superior predictive performance and robustness across a range of settings and ablation studies (Huo et al., 15 Jan 2025).

1. Multivariate Time Series Representation and Patch Extraction

CT-PatchTST processes a multivariate time series

$$\mathbf{X} = [\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_L] \in \mathbb{R}^{L \times M}$$

where $L$ is the look-back window and $M$ is the number of channels (here: offshore wind, onshore wind, solar power). The forecasting target is the next $T$-length sequence $\{\mathbf{x}_{L+1}, \dots, \mathbf{x}_{L+T}\}$.

Each input channel $c=1,\dots, M$ is segmented into $N$ overlapping patches, each of length $P$ and stride $S=\frac{P}{2}$:

$$x_{\mathrm{patch}}^{(c, i)} = \bigl[x_{c,\,t + (i{-}1)S}, \dots, x_{c,\,t + (i{-}1)S + P - 1}\bigr] \in \mathbb{R}^P, \quad i=1,\dots,N, \quad N = \Bigl\lfloor\frac{L - P}{S}\Bigr\rfloor + 1.$$

After reversible instance normalization (RevIN), this yields a three-dimensional tensor $L$0.

Each $L$1 is linearly embedded:

$L$2

stacked as $L$3.

2. Channel-Time Dual Self-Attention Transformer Encoder

The CT-PatchTST encoder introduces two consecutive Transformer-style self-attention mechanisms: channel attention and time attention, each followed by a pointwise feed-forward network (FFN) with residual connections and layer normalization.

a) Channel Attention (Inter-Channel)

For a fixed patch index $L$4, the model attends over channels:

$L$5

Each attention head $L$6 computes: \begin{align*} Q_{h}^{\mathrm{ch},i} &= \mathbf{Z}^{:,i,:} W_{h}^{Q,\mathrm{ch}}, \ K_{h}^{\mathrm{ch},i} &= \mathbf{Z}^{:,i,:} W_{h}^{K,\mathrm{ch}}, \ V_{h}^{\mathrm{ch},i} &= \mathbf{Z}^{:,i,:} W_{h}^{V,\mathrm{ch}}, \ \text{Attention}h^{\mathrm{ch},i} &= \operatorname{Softmax}!\Bigl(\tfrac{Q{h}^{\mathrm{ch},i}(K_{h}^{\mathrm{ch},i})^{T}}{\sqrt{d_k}}\Bigr) V_{h}^{\mathrm{ch},i} \in \mathbb{R}^{M \times d_k}. \end{align*} Multi-head channel attention is concatenated and projected, yielding $L$7 for each $L$8.

b) Time Attention (Intra-Channel)

For each channel $L$9, time attention attends over the patch dimension:

$M$0

with each head $M$1 computing \begin{align*} Q_{h}^{\mathrm{t},c} &= \mathbf{Z}{\mathrm{ch}}^{c,:,:} W{h}^{Q,\mathrm{t}}, \ K_{h}^{\mathrm{t},c} &= \mathbf{Z}{\mathrm{ch}}^{c,:,:} W{h}^{K,\mathrm{t}}, \ V_{h}^{\mathrm{t},c} &= \mathbf{Z}{\mathrm{ch}}^{c,:,:} W{h}^{V,\mathrm{t}}, \ \text{Attention}{h}^{\mathrm{t},c} &= \operatorname{Softmax}!\Bigl(\tfrac{Q{h}^{\mathrm{t},c} (K_{h}^{\mathrm{t},c})^T}{\sqrt{d_k}}\Bigr)V_{h}^{\mathrm{t},c}\in\mathbb{R}^{N\times d_k}. \end{align*} Concatenation and projection yield $M$2. Stacked Channel-Time encoder blocks produce the final tensor $M$3.

3. Output Head and Loss Formulation

The output head aggregates encoded representations for each channel, typically by flattening:

$M$4

and then linearly projecting to $M$5-step forecasts:

$M$6

Stacking the predictions across channels gives $M$7.

Model training minimizes mean squared error (MSE):

$M$8

Mean absolute error (MAE) is also used for evaluation:

$M$9

4. Empirical Evaluation Settings

The model is evaluated on Danish renewable energy datasets (2014–2019, approximately 500,000 hourly samples), spanning three channels: OffshoreWindPower, OnshoreWindPower, and SolarPowerProd.

Key experimental parameters are as follows:

Parameter	Value(s)
Look-back window $T$0	336 (and 512 in variant)
Forecast horizon $T$1	96, 192, 336, 720
Patch length $T$2	16
Stride $T$3	8
Encoder depth	4
Channel-attention heads	1
Time-attention heads	16
Model dimension $T$4	256
Feed-forward dimension	512
Batch size	128
Learning rate	$T$5
Optimization	Adam, 50 epochs

Performance is benchmarked against PatchTST and a GRU-based Seq2Seq model. For forecasting window $T$6 and $T$7:

Model	MSE	MAE
Seq2Seq (GRU)	0.0187	0.0561
PatchTST	0.0066	0.0540
CT-PatchTST	0.0061	0.0504
CT-PatchTST-512	0.0060	0.0515

Relative to PatchTST, CT-PatchTST reduces MSE by approximately 7.6% at $T$8 and consistently outperforms all baselines across the examined forecast horizons.

5. Ablation Studies and Model Robustness

Ablation studies investigate the impact of patch length $T$9, forecast horizon $\{\mathbf{x}_{L+1}, \dots, \mathbf{x}_{L+T}\}$0, and other hyperparameters. For all tested $\{\mathbf{x}_{L+1}, \dots, \mathbf{x}_{L+T}\}$1, CT-PatchTST consistently outperforms PatchTST, demonstrating stability across patching granularities. With $\{\mathbf{x}_{L+1}, \dots, \mathbf{x}_{L+T}\}$2 varying from 96 up to 720, CT-PatchTST maintains superior forecasting accuracy, indicating robustness for long-range predictions.

Visualization of learned channel-attention weights reveals that CT-PatchTST discovers and employs cross-channel dependencies (such as correlations between offshore and onshore wind time series) prior to temporal processing, supporting its observed gains over channel-independent approaches.

6. Significance and Implications for Renewable Energy Forecasting

CT-PatchTST addresses the primary limitation of standard PatchTST—the oversight of inter-channel dependencies in multivariate settings—by integrating a lightweight channel-wise self-attention block before temporal modeling. This dual attention approach enables the architecture to leverage both cross-variable context and long-term temporal structure, thereby improving the predictability of renewable energy systems. The demonstrated gains in MSE and MAE suggest benefits for operational decision-making and integration of renewable sources into energy grids. A plausible implication is that similar channel-time dual attention strategies could be effective for other multivariate time series applications exhibiting nontrivial channel dependencies.

7. Context Within the Literature

By extending patch-based time-series Transformers to incorporate explicit inter-channel modeling, CT-PatchTST builds upon both attention mechanisms and multi-horizon forecasting research. It maintains channel-independent advantages, such as increased data granularity, while resolving a key weakness through a principled dual-attention design. The model’s success in large-scale real-world renewable generation scenarios highlights its practical relevance and points towards further applications in environmental and resource forecasting tasks (Huo et al., 15 Jan 2025).