CT-PatchTST: Channel-Time Transformer

• The paper demonstrates that CT-PatchTST reduces MSE by approximately 7.6% over PatchTST for long-horizon renewable energy forecasting.
• CT-PatchTST is a deep learning model that integrates channel and time attention to effectively capture both interdependencies and temporal patterns in multivariate data.
• Empirical evaluations on Danish wind and solar datasets show improved forecast accuracy and robustness across varying patch configurations and forecast horizons.

The Channel-Time Patch Time-Series Transformer (CT-PatchTST) is a deep learning architecture specifically designed for long-term forecasting of multivariate renewable energy generation time series. It extends the Patch Time-Series Transformer (PatchTST) by introducing an inter-channel self-attention mechanism prior to the standard patch-wise temporal Transformer, enabling effective modeling of both channel interdependencies and temporal patterns. The method is evaluated on large-scale Danish offshore wind, onshore wind, and solar generation datasets, exhibiting superior predictive performance and robustness across a range of settings and ablation studies (Huo et al., 15 Jan 2025).

1. Multivariate Time Series Representation and Patch Extraction

CT-PatchTST processes a multivariate time series

$$\mathbf{X} = [\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_L] \in \mathbb{R}^{L \times M}$$

where $L$ is the look-back window and $M$ is the number of channels (here: offshore wind, onshore wind, solar power). The forecasting target is the next $T$-length sequence $\{\mathbf{x}_{L+1}, \dots, \mathbf{x}_{L+T}\}$.

Each input channel $c=1,\dots, M$ is segmented into $N$ overlapping patches, each of length $P$ and stride $S=\frac{P}{2}$:

$$x_{\mathrm{patch}}^{(c, i)} = \bigl[x_{c,\,t + (i{-}1)S}, \dots, x_{c,\,t + (i{-}1)S + P - 1}\bigr] \in \mathbb{R}^P, \quad i=1,\dots,N, \quad N = \Bigl\lfloor\frac{L - P}{S}\Bigr\rfloor + 1.$$

After reversible instance normalization (RevIN), this yields a three-dimensional tensor $$\tilde{\mathbf{X}} \in \mathbb{R}^{M \times N \times P}$$.

Each $x_{\mathrm{patch}}^{(c, i)}$ is linearly embedded:

$$Z^{(c,i)} = W_{\mathrm{emb}} x_{\mathrm{patch}}^{(c,i)} + E_{\mathrm{pos}}^{(i)} \in \mathbb{R}^d,$$

stacked as $\mathbf{Z} \in \mathbb{R}^{M \times N \times d}$.

2. Channel-Time Dual Self-Attention Transformer Encoder

The CT-PatchTST encoder introduces two consecutive Transformer-style self-attention mechanisms: channel attention and time attention, each followed by a pointwise feed-forward network (FFN) with residual connections and layer normalization.

a) Channel Attention (Inter-Channel)

For a fixed patch index $i$, the model attends over channels:

$\mathbf{Z}^{:, i, :} \in \mathbb{R}^{M \times d}.$

Each attention head $h=1,\dots,H_{\mathrm{ch}}$ computes: \begin{align*} Q_{h}^{\mathrm{ch},i} &= \mathbf{Z}^{:,i,:} W_{h}^{Q,\mathrm{ch}}, \ K_{h}^{\mathrm{ch},i} &= \mathbf{Z}^{:,i,:} W_{h}^{K,\mathrm{ch}}, \ V_{h}^{\mathrm{ch},i} &= \mathbf{Z}^{:,i,:} W_{h}^{V,\mathrm{ch}}, \ \text{Attention}h^{\mathrm{ch},i} &= \operatorname{Softmax}!\Bigl(\tfrac{Q{h}^{\mathrm{ch},i}(K_{h}^{\mathrm{ch},i})^{T}}{\sqrt{d_k}}\Bigr) V_{h}^{\mathrm{ch},i} \in \mathbb{R}^{M \times d_k}. \end{align*} Multi-head channel attention is concatenated and projected, yielding $$\mathbf{Z}_{\mathrm{ch}}^{:,i,:} \in \mathbb{R}^{M \times d}$$ for each $i$.

b) Time Attention (Intra-Channel)

For each channel $c$, time attention attends over the patch dimension:

$$\mathbf{Z}_{\mathrm{ch}}^{c,:,:} \in \mathbb{R}^{N \times d}$$

with each head $h=1,\dots,H_t$ computing \begin{align*} Q_{h}^{\mathrm{t},c} &= \mathbf{Z}{\mathrm{ch}}^{c,:,:} W{h}^{Q,\mathrm{t}}, \ K_{h}^{\mathrm{t},c} &= \mathbf{Z}{\mathrm{ch}}^{c,:,:} W{h}^{K,\mathrm{t}}, \ V_{h}^{\mathrm{t},c} &= \mathbf{Z}{\mathrm{ch}}^{c,:,:} W{h}^{V,\mathrm{t}}, \ \text{Attention}{h}^{\mathrm{t},c} &= \operatorname{Softmax}!\Bigl(\tfrac{Q{h}^{\mathrm{t},c} (K_{h}^{\mathrm{t},c})^T}{\sqrt{d_k}}\Bigr)V_{h}^{\mathrm{t},c}\in\mathbb{R}^{N\times d_k}. \end{align*} Concatenation and projection yield $$\mathbf{Z}_{\mathrm{t}}^{c,:,:} \in \mathbb{R}^{N\times d}$$. Stacked Channel-Time encoder blocks produce the final tensor $\mathbf{H} \in \mathbb{R}^{M \times N \times d}$.

3. Output Head and Loss Formulation

The output head aggregates encoded representations for each channel, typically by flattening:

$$\mathbf{h}^{(c)} = \mathrm{Flatten}(\mathbf{H}^{c,:,:}) \in \mathbb{R}^{N d}$$

and then linearly projecting to $T$-step forecasts:

$$\hat{\mathbf{y}}^{(c)} = W_{\mathrm{pred}} \mathbf{h}^{(c)} + \mathbf{b}_{\mathrm{pred}} \in \mathbb{R}^T.$$

Stacking the predictions across channels gives $\hat{\mathbf{Y}} \in \mathbb{R}^{M \times T}$.

Model training minimizes mean squared error (MSE):

$$\mathcal{L}_{\mathrm{MSE}} = \frac{1}{M T} \sum_{c=1}^M \sum_{t=1}^T \left( \hat{y}^{(c)}_t - y^{(c)}_t \right)^2$$

Mean absolute error (MAE) is also used for evaluation:

$$\mathcal{L}_{\mathrm{MAE}} = \frac{1}{M T} \sum_{c=1}^M \sum_{t=1}^T \left| \hat{y}^{(c)}_t - y^{(c)}_t \right|$$

4. Empirical Evaluation Settings

The model is evaluated on Danish renewable energy datasets (2014–2019, approximately 500,000 hourly samples), spanning three channels: OffshoreWindPower, OnshoreWindPower, and SolarPowerProd.

Key experimental parameters are as follows:

Parameter	Value(s)
Look-back window $L$	336 (and 512 in variant)
Forecast horizon $T$	96, 192, 336, 720
Patch length $P$	16
Stride $S$	8
Encoder depth	4
Channel-attention heads	1
Time-attention heads	16
Model dimension $d$	256
Feed-forward dimension	512
Batch size	128
Learning rate	$1 \times 10^{-3}$
Optimization	Adam, 50 epochs

Performance is benchmarked against PatchTST and a GRU-based Seq2Seq model. For forecasting window $T=96$ and $L=336$:

Model	MSE	MAE
Seq2Seq (GRU)	0.0187	0.0561
PatchTST	0.0066	0.0540
CT-PatchTST	0.0061	0.0504
CT-PatchTST-512	0.0060	0.0515

Relative to PatchTST, CT-PatchTST reduces MSE by approximately 7.6% at $T=96$ and consistently outperforms all baselines across the examined forecast horizons.

5. Ablation Studies and Model Robustness

Ablation studies investigate the impact of patch length $P$, forecast horizon $T$, and other hyperparameters. For all tested $P\in\{4,8,12,16,24,32,40\}$, CT-PatchTST consistently outperforms PatchTST, demonstrating stability across patching granularities. With $T$ varying from 96 up to 720, CT-PatchTST maintains superior forecasting accuracy, indicating robustness for long-range predictions.

Visualization of learned channel-attention weights reveals that CT-PatchTST discovers and employs cross-channel dependencies (such as correlations between offshore and onshore wind time series) prior to temporal processing, supporting its observed gains over channel-independent approaches.

6. Significance and Implications for Renewable Energy Forecasting

CT-PatchTST addresses the primary limitation of standard PatchTST—the oversight of inter-channel dependencies in multivariate settings—by integrating a lightweight channel-wise self-attention block before temporal modeling. This dual attention approach enables the architecture to leverage both cross-variable context and long-term temporal structure, thereby improving the predictability of renewable energy systems. The demonstrated gains in MSE and MAE suggest benefits for operational decision-making and integration of renewable sources into energy grids. A plausible implication is that similar channel-time dual attention strategies could be effective for other multivariate time series applications exhibiting nontrivial channel dependencies.

7. Context Within the Literature

By extending patch-based time-series Transformers to incorporate explicit inter-channel modeling, CT-PatchTST builds upon both attention mechanisms and multi-horizon forecasting research. It maintains channel-independent advantages, such as increased data granularity, while resolving a key weakness through a principled dual-attention design. The model’s success in large-scale real-world renewable generation scenarios highlights its practical relevance and points towards further applications in environmental and resource forecasting tasks (Huo et al., 15 Jan 2025).