---
title: 'CT-PatchTST: Channel-Time Transformer'
url: https://www.emergentmind.com/topics/channel-time-patchtst-ct-patchtst
type: topic
---

# CT-PatchTST: Channel-Time Transformer

The Channel-Time Patch Time-Series Transformer (CT-PatchTST) is a deep learning architecture specifically designed for long-term forecasting of multivariate renewable energy generation time series. It extends the Patch Time-Series Transformer (PatchTST) by introducing an inter-channel self-attention mechanism prior to the standard patch-wise temporal Transformer, enabling effective modeling of both channel interdependencies and temporal patterns. The method is evaluated on large-scale Danish offshore wind, onshore wind, and solar generation datasets, exhibiting superior predictive performance and robustness across a range of settings and ablation studies [2501.08620].

## 1. Multivariate Time Series Representation and Patch Extraction

CT-PatchTST processes a multivariate time series 
$$
\mathbf{X} = [\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_L] \in \mathbb{R}^{L \times M}
$$
where $L$ is the look-back window and $M$ is the number of channels (here: offshore wind, onshore wind, solar power). The forecasting target is the next $T$-length sequence $\{\mathbf{x}_{L+1}, \dots, \mathbf{x}_{L+T}\}$.

Each input channel $c=1,\dots, M$ is segmented into $N$ overlapping patches, each of length $P$ and stride $S=\frac{P}{2}$:
$$
x_{\mathrm{patch}}^{(c, i)} = \bigl[x_{c,\,t + (i{-}1)S}, \dots, x_{c,\,t + (i{-}1)S + P - 1}\bigr] \in \mathbb{R}^P, \quad i=1,\dots,N, \quad N = \Bigl\lfloor\frac{L - P}{S}\Bigr\rfloor + 1.
$$
After reversible instance normalization (RevIN), this yields a three-dimensional tensor $\tilde{\mathbf{X}} \in \mathbb{R}^{M \times N \times P}$.

Each $x_{\mathrm{patch}}^{(c, i)}$ is linearly embedded:
$$
Z^{(c,i)} = W_{\mathrm{emb}} x_{\mathrm{patch}}^{(c,i)} + E_{\mathrm{pos}}^{(i)} \in \mathbb{R}^d,
$$
stacked as $\mathbf{Z} \in \mathbb{R}^{M \times N \times d}$.

## 2. Channel-Time Dual Self-Attention Transformer Encoder

The CT-PatchTST encoder introduces two consecutive Transformer-style self-attention mechanisms: channel attention and time attention, each followed by a pointwise feed-forward network (FFN) with residual connections and layer normalization.

**a) Channel Attention (Inter-Channel)**

For a fixed patch index $i$, the model attends over channels:
$$
\mathbf{Z}^{:, i, :} \in \mathbb{R}^{M \times d}.
$$
Each attention head $h=1,\dots,H_{\mathrm{ch}}$ computes:
\begin{align*}
Q_{h}^{\mathrm{ch},i} &= \mathbf{Z}^{:,i,:} W_{h}^{Q,\mathrm{ch}}, \\
K_{h}^{\mathrm{ch},i} &= \mathbf{Z}^{:,i,:} W_{h}^{K,\mathrm{ch}}, \\
V_{h}^{\mathrm{ch},i} &= \mathbf{Z}^{:,i,:} W_{h}^{V,\mathrm{ch}}, \\
\text{Attention}_h^{\mathrm{ch},i} &= \operatorname{Softmax}\!\Bigl(\tfrac{Q_{h}^{\mathrm{ch},i}(K_{h}^{\mathrm{ch},i})^{T}}{\sqrt{d_k}}\Bigr) V_{h}^{\mathrm{ch},i} \in \mathbb{R}^{M \times d_k}.
\end{align*}
Multi-head channel attention is concatenated and projected, yielding $\mathbf{Z}_{\mathrm{ch}}^{:,i,:} \in \mathbb{R}^{M \times d}$ for each $i$.

**b) Time Attention (Intra-Channel)**

For each channel $c$, time attention attends over the patch dimension:
$$
\mathbf{Z}_{\mathrm{ch}}^{c,:,:} \in \mathbb{R}^{N \times d}
$$
with each head $h=1,\dots,H_t$ computing
\begin{align*}
Q_{h}^{\mathrm{t},c} &= \mathbf{Z}_{\mathrm{ch}}^{c,:,:} W_{h}^{Q,\mathrm{t}}, \\
K_{h}^{\mathrm{t},c} &= \mathbf{Z}_{\mathrm{ch}}^{c,:,:} W_{h}^{K,\mathrm{t}}, \\
V_{h}^{\mathrm{t},c} &= \mathbf{Z}_{\mathrm{ch}}^{c,:,:} W_{h}^{V,\mathrm{t}}, \\
\text{Attention}_{h}^{\mathrm{t},c} &= \operatorname{Softmax}\!\Bigl(\tfrac{Q_{h}^{\mathrm{t},c} (K_{h}^{\mathrm{t},c})^T}{\sqrt{d_k}}\Bigr)V_{h}^{\mathrm{t},c}\in\mathbb{R}^{N\times d_k}.
\end{align*}
Concatenation and projection yield $\mathbf{Z}_{\mathrm{t}}^{c,:,:} \in \mathbb{R}^{N\times d}$. Stacked Channel-Time encoder blocks produce the final tensor $\mathbf{H} \in \mathbb{R}^{M \times N \times d}$.

## 3. Output Head and Loss Formulation

The output head aggregates encoded representations for each channel, typically by flattening:
$$
\mathbf{h}^{(c)} = \mathrm{Flatten}(\mathbf{H}^{c,:,:}) \in \mathbb{R}^{N d}
$$
and then linearly projecting to $T$-step forecasts:
$$
\hat{\mathbf{y}}^{(c)} = W_{\mathrm{pred}} \mathbf{h}^{(c)} + \mathbf{b}_{\mathrm{pred}} \in \mathbb{R}^T.
$$
Stacking the predictions across channels gives $\hat{\mathbf{Y}} \in \mathbb{R}^{M \times T}$.

Model training minimizes mean squared error (MSE):
$$
\mathcal{L}_{\mathrm{MSE}} = \frac{1}{M T} \sum_{c=1}^M \sum_{t=1}^T \left( \hat{y}^{(c)}_t - y^{(c)}_t \right)^2
$$
Mean absolute error (MAE) is also used for evaluation:
$$
\mathcal{L}_{\mathrm{MAE}} = \frac{1}{M T} \sum_{c=1}^M \sum_{t=1}^T \left| \hat{y}^{(c)}_t - y^{(c)}_t \right|
$$

## 4. Empirical Evaluation Settings

The model is evaluated on Danish renewable energy datasets (2014–2019, approximately 500,000 hourly samples), spanning three channels: OffshoreWindPower, OnshoreWindPower, and SolarPowerProd.

Key experimental parameters are as follows:

| Parameter                   | Value(s)                  |
|-----------------------------|---------------------------|
| Look-back window $L$        | 336 (and 512 in variant)  |
| Forecast horizon $T$        | 96, 192, 336, 720         |
| Patch length $P$            | 16                        |
| Stride $S$                  | 8                         |
| Encoder depth               | 4                         |
| Channel-attention heads     | 1                         |
| Time-attention heads        | 16                        |
| Model dimension $d$         | 256                       |
| Feed-forward dimension      | 512                       |
| Batch size                  | 128                       |
| Learning rate               | $1 \times 10^{-3}$        |
| Optimization                | Adam, 50 epochs           |

Performance is benchmarked against PatchTST and a GRU-based Seq2Seq model. For forecasting window $T=96$ and $L=336$:

| Model            | MSE      | MAE     |
|------------------|----------|---------|
| Seq2Seq (GRU)    | 0.0187   | 0.0561  |
| PatchTST         | 0.0066   | 0.0540  |
| CT-PatchTST      | 0.0061   | 0.0504  |
| CT-PatchTST-512  | 0.0060   | 0.0515  |

Relative to PatchTST, CT-PatchTST reduces MSE by approximately 7.6% at $T=96$ and consistently outperforms all baselines across the examined forecast horizons.

## 5. Ablation Studies and Model Robustness

Ablation studies investigate the impact of patch length $P$, forecast horizon $T$, and other hyperparameters. For all tested $P\in\{4,8,12,16,24,32,40\}$, CT-PatchTST consistently outperforms PatchTST, demonstrating stability across patching granularities. With $T$ varying from 96 up to 720, CT-PatchTST maintains superior forecasting accuracy, indicating robustness for long-range predictions.

Visualization of learned channel-attention weights reveals that CT-PatchTST discovers and employs cross-channel dependencies (such as correlations between offshore and onshore wind time series) prior to temporal processing, supporting its observed gains over channel-independent approaches.

## 6. Significance and Implications for Renewable Energy Forecasting

CT-PatchTST addresses the primary limitation of standard PatchTST—the oversight of inter-channel dependencies in multivariate settings—by integrating a lightweight channel-wise self-attention block before temporal modeling. This dual attention approach enables the architecture to leverage both cross-variable context and long-term temporal structure, thereby improving the predictability of renewable energy systems. The demonstrated gains in MSE and MAE suggest benefits for operational decision-making and integration of renewable sources into energy grids. A plausible implication is that similar channel-time dual attention strategies could be effective for other multivariate time series applications exhibiting nontrivial channel dependencies.

## 7. Context Within the Literature

By extending patch-based time-series Transformers to incorporate explicit inter-channel modeling, CT-PatchTST builds upon both attention mechanisms and multi-horizon forecasting research. It maintains channel-independent advantages, such as increased data granularity, while resolving a key weakness through a principled dual-attention design. The model’s success in large-scale real-world renewable generation scenarios highlights its practical relevance and points towards further applications in environmental and resource forecasting tasks [2501.08620].

Source: https://www.emergentmind.com/topics/channel-time-patchtst-ct-patchtst