Temporal PatchGAN: Enforcing Spatio-Temporal Coherence

• Temporal PatchGAN is a 3D adversarial framework that ensures both spatial and temporal consistency in video and volumetric structured prediction tasks.
• It utilizes patch-wise discrimination with hinge loss or binary cross-entropy to reduce artifacts and enhance local coherence across frames or slices.
• Empirical results demonstrate significant improvements in metrics like FID, LPIPS, and Dice scores in applications such as video inpainting and medical segmentation.

Temporal PatchGANs constitute a class of adversarial discriminators and loss formulations specifically designed to enforce both spatial and temporal (or volumetric) consistency in structured prediction tasks involving videos or 3D volumes. Their core innovation is the extension of the "PatchGAN" discriminator paradigm to operate along both spatial and temporal dimensions using 3D convolutions, allowing the adversarial framework to judge local coherence across multiple adjacent frames or slices, rather than per-frame or per-slice only. This approach has demonstrated substantial effectiveness in domains such as free-form video inpainting and volumetric segmentation where maintaining temporally smooth, realistic outputs is crucial (Chang et al., 2019, Mantha et al., 2023).

1. Mathematical Formulation and Loss Functions

Let $I = \{I_t\}_{t=1}^T$ denote the input masked sequence (video or volume), $V = \{V_t\}_{t=1}^T$ the ground-truth, and $G(I) \equiv O = \{O_t\}_{t=1}^T$ the generator output. The Temporal PatchGAN (T-PatchGAN) adversarial loss employs a discriminator $D$ that returns a 3D patch-wise map of real/fake logits over the spatio-temporal grid.

For the video inpainting setup (Chang et al., 2019), hinge loss is used:

• Discriminator loss:

$$\mathcal{L}_D = \mathbb{E}_{V \sim p_{\text{data}}} [\max(0, 1 + D(V))] + \mathbb{E}_{I \sim p_I} [\max(0, 1 - D(G(I)))]$$

• Generator T-PatchGAN loss:

$$\mathcal{L}_{\text{TPatchGAN}} = -\mathbb{E}_{I \sim p_I} \left[ D(G(I)) \right]$$

For segmentation with a Temporal Cubic PatchGAN (TCuP-GAN) (Mantha et al., 2023), the binary cross-entropy PatchGAN loss is used patch-wise in 3D:

• Discriminator losses:

$$L_{\text{real}} = -\mathbb{E}_{x, y} [ \log D(x, y) ], \qquad L_{\text{fake}} = -\mathbb{E}_{x} [ \log (1 - D(x, G(x))) ]$$

$$L_{\text{disc}} = \frac{1}{2}(L_{\text{real}} + L_{\text{fake}})$$

• Generator adversarial loss:

$$L_{\text{GAN}} = -\mathbb{E}_{x} [ \log D(x, G(x)) ]$$

The generator is also optimized for a reconstruction or segmentation objective, such as class-weighted binary cross-entropy.

The central feature in both settings is that the discriminator explicitly judges realism not of individual frames or pixels, but of local boxes in space-time or space-depth, enhancing continuity.

2. 3D PatchGAN Discriminator Design

The Temporal PatchGAN discriminator processes short video clips or 3D volumes, yielding a spatio-temporal map of patch-wise logits:

• Video Input Example: $(T, H, W, C)$, where $T=5$–$7$ frames.
• Layer Structure: Six 3D convolutional layers with kernels $(3_t \times 5_h \times 5_w)$ and stride $(1_t \times 2_h \times 2_w)$, progressively increasing receptive field and feature width (e.g., $64\to128\to256\ldots$).
• Activations: LeakyReLU after each convolution.
• Stabilization: Spectral normalization applied to all layers.
• Patch-wise Output: Final feature grid encodes real/fake scores for each spatio-temporal patch, with output eventually aggregated (averaged or summed) for inference.

For volumetric segmentation, as in TCuP-GAN (Mantha et al., 2023), the discriminator input consists of the concatenated raw MRI volume and (ground-truth or generated) mask, and the architecture uses 3D convolutions of kernel $1 \times 3 \times 3$ in depth $\times$ height $\times$ width, with four downsampling blocks followed by a final per-slice $3\times3$ sigmoid map.

By enforcing realism over local spatio-temporal or cubic patches rather than global slices, this framework ensures both intra-frame and cross-frame structural fidelity.

3. Integration with Full Model Objectives

In free-form video inpainting (Chang et al., 2019), the Temporal PatchGAN adversarial loss is one component of a composite objective:

$$\mathcal{L}_{\text{total}} = \lambda_{l_1} \mathcal{L}_{l_1} + \lambda_{l_1}^{\text{mask}} \mathcal{L}_{l_1}^{\text{mask}} + \lambda_{\text{perc}} \mathcal{L}_{\text{perc}} + \lambda_{\text{style}} \mathcal{L}_{\text{style}} + \lambda_G \mathcal{L}_{\text{TPatchGAN}}$$

Default weights follow the image inpainting literature: $\lambda_{l_1}=1.0$, $\lambda_{l_1}^{\text{mask}}=1.0$, $\lambda_{\text{perc}} \approx 0.1$, $\lambda_{\text{style}} \approx 10^{-3}$, $\lambda_{G} \approx 0.01$.

In TCuP-GAN, the generator is trained to minimize:

$L_{\text{gen}} = L_{\text{recon}} + L_{\text{GAN}}$

with $L_{\text{recon}}$ being a class-weighted cross-entropy and $L_{\text{GAN}}$ the adversarial cubic patch loss.

No explicit Dice, $L_1$, or temporal coherence terms are needed; in both cases, temporal smoothness is implicitly enforced by the 3D convolutional operations and stride-1 temporal filtering.

4. Implementation Details and Stabilization

Key implementation and stabilization techniques used in both settings include:

• Temporal Window: 5–7 consecutive frames or depth slices are jointly processed by generator and discriminator in video and volumetric applications, respectively.
• Spatial Patch Size: $5\times5$ (video, (Chang et al., 2019)) or $3\times3$ (volumetric, (Mantha et al., 2023)) realized via kernel size.
• Optimizer: Adam with $(\beta_1=0.5, \beta_2=0.999)$; learning rates $\sim2\cdot10^{-4}$ (Chang et al., 2019), $5 \times 10^{-4}$ (TCuP-GAN generator), $1 \times 10^{-4}$ (TCuP-GAN discriminator), with learning rate decay in segmentation.
• Spectral Normalization: Applied to all convolutional weights for stability during adversarial training.
• Hinge-GAN Loss: Employed in video inpainting to avoid mode collapse and to improve gradient flow.
• Data Augmentation: For medical segmentation, procedures include per-channel normalization, spatial resizing, and Gaussian noise injection.

Notably, no explicit optical-flow guidance or additional temporal regularization is required; the 3D convolutional design alone suffices to impose continuity.

5. Quantitative Effects and Ablation Evidence

Empirical evaluations corroborate the impact of Temporal PatchGAN on output coherence and realism:

• Video Inpainting (Chang et al., 2019): Adding Temporal PatchGAN with 3D gating lowers LPIPS from 0.1769 (2D, no GAN) to 0.1209 and FID from 1.243 to 1.034 on FVI data. Ablations confirm that both 3D convolutions and the adversarial loss term independently reduce flicker and perceptual artifacts.
• FaceForensics (curve-like masks): FID drops from $\approx 0.704$ (no T-PatchGAN) to $\approx 0.472$ when using T-PatchGAN.
• User Study (Chang et al., 2019): Model outputs were judged as "indistinguishable from real" in 23% of trials on curve-like masks.
• Segmentation (TCuP-GAN, (Mantha et al., 2023)): Achieves median LesionWise Dice of $0.90$ (whole tumor), $0.90$ (tumor core), $0.84$ (enhancing tumor) for adult glioma, and similarly competitive metrics for other datasets. Median $95\%$ Hausdorff distances are also low.

This pattern of results demonstrates that the Temporal PatchGAN discriminator directly reduces temporal artifacts and increases spatio-temporal coherence, serving as an effective, general-purpose mechanism without requiring explicit temporal regularization.

6. Generalization and Applicability

Beyond the domains of video inpainting and neuroimaging segmentation, the Temporal PatchGAN and its variants (including TCuP-GAN) apply to any task involving volumetric, sequential, or time-series data where local structural and temporal fidelity are critical. This includes cardiac 4D CT, fMRI, ultrasound B-mode, multi-organ CT, and even non-medical data such as point clouds represented as voxel grids (Mantha et al., 2023).

A plausible implication is that any application benefiting from local adversarial discrimination over spatio-temporal neighborhoods can employ Temporal PatchGAN architectures to enforce robust transition modeling, suppress flicker/artifacts, and enhance perceptual continuity in both generation and structured prediction tasks. The strategy of combining spatial U-Net encoding, ConvLSTM temporal context, and 3D PatchGAN local realism offers a modular backbone for architectures targeting 3D and 4D structured generative tasks.