---
title: Temporal Cubic PatchGAN
url: https://www.emergentmind.com/topics/temporal-cubic-patchgan-tcup-gan
type: topic
---

# Temporal Cubic PatchGAN

Temporal Cubic PatchGAN (TCuP-GAN) is a generative adversarial network (GAN) architecture for automated three-dimensional (3D) biomedical image segmentation. It combines a generative feature learning framework with convolutional long short-term memory (ConvLSTM) modules for capturing cross-slice temporal dependencies, and it utilizes a 3D PatchGAN discriminator to enforce local spatial coherence by adversarial training. TCuP-GAN is designed for robust multi-class volume-to-volume segmentation, as demonstrated on four benchmarks from the 2023 Brain Tumor Segmentation (BraTS) Challenge, including Adult Glioma, Meningioma, Pediatric Tumors, and the Sub-Saharan Africa cohort [2311.14148].

## 1. Model Architecture

### 1.1 Generator: Temporal Cubic U-Net

The generator, termed "Temporal Cubic U-Net," employs a U-Net style encoder–decoder structure instantiated as follows:
- **Input**: Image cube of dimensions $(155 \times 4 \times 256 \times 256)$ (Depth × Channels × Height × Width), where the four channels comprise the T1, T2, T1ce, and FLAIR MRI sequences.
- **Encoder**: Five down-sampling stages. At each level $\ell \in \{1,2,3,4,5\}$:
  - **ConvLSTM2D** (kernel $3\times3$) with $F_\ell$ filters ($F_1=16,\ F_2=32,\ F_3=48,\ F_4=64,\ F_5=128$). Each ConvLSTM2D layer accumulates spatial and inter-slice "temporal" context along the depth dimension.
  - 2D convolution ($3\times3$, stride 1), instance normalization, leaky ReLU activation.
  - 2D max pooling ($2\times2$) reduces spatial resolution.
  - **Output after encoder**: $(155 \times 128 \times 8 \times 8)$.
- **Decoder**: Five up-sampling blocks, each consisting of:
  - 2D transposed convolution to double spatial resolution,
  - ConvLSTM2D with $F_\ell$ filters (matching encoder depth), skip-connection from corresponding encoder activations,
  - Instance normalization, leaky ReLU activation.
- **Final layer**: 2D convolution ($3\times3$, stride 1) with sigmoid activation.
- **Total generator parameters**: Approximately 2.3 million.

### 1.2 Discriminator: 3D PatchGAN

The discriminator is a 3D PatchGAN:
- **Input**: Concatenated MRI image cube and segmentation mask; size $(155 \times (4+K) \times 256 \times 256)$ with $K=3$ classes for multi-class segmentation.
- **Four convolutional blocks**:
  - Each: 3D convolution (kernel $1\times3\times3$, stride 1), instance normalization, tanh, and dropout (0.2).
  - Filter sizes: 16, 32, 64, 128; height/width are down-sampled by stride, depth is preserved.
- **Final layer**: Conv3D ($1\times3\times3$), sigmoid activation, outputs $(155 \times 3 \times 3)$ patch authenticity map.

### 1.3 Role of ConvLSTM2D

ConvLSTM2D layers in both encoder and decoder enable learning of joint 2D spatial and inter-slice ("temporal") representations, providing each slice context from neighboring slices via hidden state propagation. This design is crucial for modeling the volumetric continuity intrinsic to medical image data [2311.14148].

## 2. Mathematical Objectives

### 2.1 Adversarial Loss

The adversarial loss implements a PatchGAN objective:
$$
L_{\text{adv}}(G,D) = \mathbb{E}_{x,y}[\log D(x,y)] + \mathbb{E}_x[\log(1 - D(x, G(x)))]
$$

### 2.2 Supervised Segmentation Loss

A scaled, class-weighted binary cross-entropy is used:
$$
\text{BCE}'(G) = \gamma \cdot W \cdot \text{BCE}
$$
with $W = 1 - \sum \text{GT} / \sum(\text{all pixels in GT})$; $\gamma = 200$.

Additionally, a multi-class Dice loss can be used:
$$
L_{\text{Dice}} = 1 - \frac{2 \sum_i p_i g_i}{\sum_i p_i + \sum_i g_i}
$$
where $p_i$, $g_i$ are predicted and ground-truth one-hot values.

### 2.3 Combined Generator Objective

The generator is trained with:
$$
L_G = \text{BCE}'(G) + L_{\text{fake}}
$$
where $L_{\text{fake}} = \mathbb{E}_x[-\log D(x, G(x))]$.

The discriminator minimizes:
$$
L_D = \frac{1}{2}\Bigl(\mathbb{E}_{x,y}[-\log D(x,y)] + \mathbb{E}_x[ -\log(1 - D(x, G(x)))]\Bigr)
$$

## 3. Training and Implementation Details

### 3.1 Data Preprocessing and Augmentation

- Intensity normalization of each MRI sequence by 99th-percentile.
- Concatenation of four modality channels, resizing $x$, $y$ to $256\times256$.
- On-the-fly addition of Gaussian noise ($\mu=0$, $\sigma=0.1$) to non-background voxels.
- Segmentation masks are one-hot encoded into 3 classes and resized.

### 3.2 Training Schedule

- Optimizer: Adam ($\beta_1=0.9$, $\beta_2=0.999$) for both generator and discriminator.
- Learning rates: $r_G$ initial $5 \times 10^{-4}$, decayed by $0.95$ every 5 epochs; $r_D=1 \times 10^{-4}$, identical decay.
- Batch size: 2 volumes.
- Trained from scratch for 30 epochs on Adult Glioma. Transfer learning to Meningioma, Pediatric, SSA cohorts for 30 epochs at constant $r=1 \times 10^{-4}$.

## 4. Quantitative Results and Comparative Performance

Validation performance on BraTS 2023 test sets is summarized by LesionWise Dice similarity ($D_{lw}$) and $95\%$ Hausdorff distance ($H_{95,lw}$):

| Dataset                | Mean D_{lw} (WT/TC/ET)   | Mean H_{95,lw} (WT [mm]/TC/ET) |
|------------------------|--------------------------|-------------------------------|
| Adult Glioma (GLI)     | 0.83 / 0.76 / 0.76       | 29.3 / 43.5 / 35.6             |
| Meningioma (MEN)       | 0.67 / 0.68 / 0.68       | 86.8 / 82.2 / 81.8             |
| Pediatric (PED)        | 0.74 / 0.66 / 0.45       | 48.8 / 58.7 / 152.9            |
| Sub-Saharan Africa     | 0.67 / 0.64 / 0.63       | 76.0 / 74.7 / 72.0             |

$D_{lw}$: LesionWise Dice score (WT: Whole Tumor, TC: Tumor Core, ET: Enhancing Tumor).

Strong Dice ($>0.8$) is attained on the GLI cohort. A reduction in ET segmentation performance is observed on smaller PED and SSA sets, reflecting data scarcity and small lesion sizes [2311.14148].

## 5. Ablation Analysis, Limitations, and Failure Modes

### 5.1 ConvLSTM Modules

Ablation replacing ConvLSTM2D with standard 2D conv layers produces a $\sim5\%$ reduction in WT Dice on GLI (from $0.83$ to $\sim0.78$). This demonstrates the criticality of temporal context modeling for 3D segmentation performance.

### 5.2 Limitations

- Cases lacking predicted TC or ET completely yield $D_{lw}=0$ for those classes.
- Small, disconnected false-positive lesions reduce mean Dice.
- Elevated $H_{95}$ in PED ET class is linked to both tiny ground-truth ET volumes and large false positives—distinguishing architectural limitations from data scarcity is challenging.
- Median Dice remains $>0.8$ on most cohorts, indicating the robustness of the primary method.

## 6. Extensions and Future Directions

Generalization to other 3D biomedical segmentation tasks, such as multi-organelle electron microscopy, is achievable by modifying input/output channels and thresholding. Incorporating more complex objective functions—adversarial plus multi-class Dice and focal loss—facilitates stabilization in class-imbalanced scenarios. Extending the framework with attention gates or 3D ConvLSTM modules could allow learning of fully volumetric temporal features. Integration with citizen-science platforms for human-in-the-loop annotation enables rapid expert correction and downstream model fine-tuning [2311.14148].

Source: https://www.emergentmind.com/topics/temporal-cubic-patchgan-tcup-gan