ctdGAN for Multimodal Data Synthesis

• ctdGAN is a family of conditional GAN models designed for medical image translation, table data synthesis, and enhanced image generation in low-data regimes.
• Each variant employs domain-specific conditioning, including multi-channel CT inputs, cluster-based latent encoding, and heavy-tailed mixture modeling to address unique challenges.
• Empirical findings demonstrate improved stroke lesion segmentation, statistically significant tabular data fidelity, and increased diversity in small-sample image synthesis.

ctdGAN refers to three distinct conditional generative adversarial network (cGAN) models developed for different data modalities and use cases: (1) CT-to-DWI image translation for ischemic stroke lesion segmentation (Rubin et al., 2019), (2) tabular data synthesis with subspace-aware sampling for class imbalance (Akritidis et al., 1 Aug 2025), and (3) Student’s t-mixture conditional GAN for image generation under limited data regimes (Sun et al., 2018). Each variant incorporates domain-specific conditioning strategies, architectural features, and loss formulations to address the particular challenges of its respective domain.

1. ctdGAN for CT-to-DWI Image Translation in Stroke Lesion Segmentation

The ctdGAN model (Rubin et al., 2019) is a conditional GAN for cross-modality medical image translation. Its primary objective is to map 5-channel CT perfusion inputs to synthetic diffusion-weighted MRI (DWI) images, thereby supporting automated ischemic core segmentation in stroke triage.

Model Architecture

• Generator: A ResNet-based image-to-image translation network, inspired by pix2pix, receives five 2D channels (non-contrast CT, CBF, CBV, MTT, Tmax) of 256×256. The architecture entails:
  • Three front-end convolutions for initial downsampling.
  • Nine ResNet blocks with reflection padding, instance normalization, and dropout (p=0.5) for implicit stochasticity.
  • Two fractionally-strided convolutions for upsampling.
  • Final 7×7 convolution with Tanh activation for single-channel DWI synthesis.
• Discriminator: A PatchGAN module operating over 70×70 spatial patches, inputting both the CT-perfusion stack and either real or generated DWI. It consists of five convolutional layers with progressively increasing channels and instance normalization, yielding a 30×30 map averaged for the adversarial score.

Objective Functions

• Adversarial Loss:

$$\mathcal{L}_{\mathrm{GAN}}(G, D) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y)} \left[ \log D(x, y) \right] + \mathbb{E}_{x \sim p_{\mathrm{data}}(x), z \sim p_z(z)} \left[ \log \left( 1 - D \left( x, G(x, z) \right) \right) \right]$$

• L1 Reconstruction Loss:

$$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$

• Joint Training Objective:

$$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$

Training Setup

• Dataset: 94 paired CT/DWI volumes (63 subjects, ISLES 2018).
• Augmentation: random affine transforms.
• Cross-validation: 5-fold by subject.
• Optimization: Adam (lr=2×10⁻⁴, β₁=0.5, 200 epochs, batch size=1).

Downstream Segmentation

A fully convolutional network (FCN) with residual blocks, dilated convolutions, a pyramid pooling module, and focal loss is trained with either (i) CTP-only input or (ii) CTP plus ctdGAN-generated DWI. The latter configuration consistently outperforms the former in Dice coefficient, precision, recall, and volumetric measures.

Quantitatively, the addition of ctdGAN-synthesized DWI achieves Dice 0.54±0.23 vs 0.53±0.25, and improves mean absolute volume difference to 10.2% from 11.5%. Qualitatively, synthetic DWI strongly recapitulates infarct hyperintensities, and segmentations exhibit reduced false positives and improved lesion contiguity (Rubin et al., 2019).

2. ctdGAN for Tabular Data Generation with Cluster-Conditional Subspaces

The tabular-domain ctdGAN (Akritidis et al., 1 Aug 2025) is a cGAN tailored for class-imbalanced tabular data, integrating explicit space partitioning and conditional latent sampling to generate synthetic samples in realistic data subspaces.

Space Partitioning and Latent Coding

• Clustering: k-Means++ partitions the dataset, selecting $k$ to minimize

$SI(k) = \frac{I(k)}{I(1)} + a\,k$

with user-tunable penalty $a$.

• Cluster and Class Codes: Both are encoded as one-hot vectors and concatenated into the latent vector.
• Latent Vector Construction:

$$z = z^{c} \oplus \left(\bigoplus_{j=1}^{n_{d}} h(z_{j}^{d})\right) \oplus h(u) \oplus h(y)$$

where $z^c \sim \mathcal{N}(0, I_e)$, $z^u$ is sampled cluster, and $z^y$ is requested class.

Conditional Probabilistic Sampling

• For a target class $$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$0, the cluster code $$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$1 is drawn from the empirical conditional distribution $$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$2 computed from training data:

$$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$3

• Full sampling:

$$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$4

Network Architectures

• Generator: Two residual blocks (FC(·,256)→BN→ReLU→skip) followed by output heads for (a) continuous features (tanh), (b) discrete features (Gumbel-Softmax), (c) cluster code and (d) class code (both via Gumbel-Softmax).
• Discriminator/Critic: PacGAN-10 input concatenation, two FC(256) layers (LeakyReLU, Dropout), and linear head.

Loss Functions

• Critic Loss (WGAN-GP):

$$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$5

• Generator Loss:

$$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$6

with cross-entropy penalties on cluster ($$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$7) and class ($$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$8) consistency, both scaled per misprediction rates.

Training and Evaluation

• Training: Adam (lr=2×10⁻⁴, weight_decay=10⁻⁶, batch_size=100, 300 epochs), with PacGAN-style noise robustness.
• Evaluation: On 14 imbalanced datasets, ctdGAN outperforms CGAN, CopulaGAN, CTGAN, CTAB-GAN⁺, SB-GAN, TVAE, and Gaussian Copula in mean-rank classification improvement and sample fidelity (Friedman test $$\mathcal{L}_{L_1}(G) = \mathbb{E}_{x, y \sim p_{\mathrm{data}}(x, y), z \sim p_z(z)} \left[ \| y - G(x, z) \|_1 \right]$$9). Ablations confirm the importance of cluster- and class-penalty terms and cluster-wise scaling (Akritidis et al., 1 Aug 2025).

3. Student’s t-ctdGAN: Mixture-Based Conditional Image GANs

The Student’s t-variant of ctdGAN (Sun et al., 2018) augments the latent space of a conditional GAN with an attention-weighted mixture of multivariate t-distributions, targeting increased diversity and robustness for image generation in limited-data regimes.

Latent Space and Generator Construction

• Latent Vector: For each sample, $$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$0 components $$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$1 with $$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$2 (common dof $$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$3) are sampled and aggregated by learnable attention weights $$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$4:

$$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$5

• Generator Network: Receives $$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$6, applying FC layers (for MNIST/Fashion-MNIST) or mixed FC/deconv (CIFAR-10), outputting images with tanh/sigmoid activation.
• Discriminator (with auxiliary classifier): Shared convolutional trunk, with heads for (a) adversarial real/fake prediction and (b) class prediction.

Objective Functions

• Adversarial Loss (minimax GAN):

$$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$7

• Auxiliary Classification Loss: Jointly for both real and generated data:

$$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$8

• Optimization: Both generator and discriminator updated using Adam/RMSProp (lr ≈ 2×10⁻⁴, β₁ = 0.5).

Proof and Theoretical Properties

It is mathematically shown that any multivariate t-distribution can be expressed as a linear transformation of a standard t-distribution, facilitating sampling flexibility in the latent space.

Empirical Results

• On datasets with limited examples (e.g., MNIST/Fashion-MNIST: 500 samples/class), Student's t-ctdGAN yields Inception Scores superior to GAN, DeLiGAN, and in many cases DCGAN, indicating both improved sample diversity and quality.
• On CIFAR-10 (5,000 samples), the Inception score of 2.44±0.23 also exceeds competing baseline models (Sun et al., 2018).

4. Comparative Table: ctdGAN Variants

Variant	Domain	Conditioning Strategy
ctdGAN (Rubin et al., 2019)	CT→DWI, ischemic stroke segmentation	Multi-channel image input (CTP), pix2pix loss
ctdGAN (Akritidis et al., 1 Aug 2025)	Tabular, class-imbalanced generation	k-Means clustering, cluster/class codes
Student’s t-ctdGAN (Sun et al., 2018)	Small-sample image generation	Mixture of t-distributions, attention, auxiliary classifier

Each variant adapts the cGAN principle to domain-specific structure and conditioning, enabling improved data synthesis or downstream task performance.

5. Significance and Distinctive Elements

ctdGAN models exemplify how tailoring GAN architectures and sampling to domain structure (multimodal medical images, clustered tabular subspaces, heavy-tailed latent distributions) yields advances in:

• Data realism and fidelity for synthetic images and tables.
• Downstream supervised learning, such as segmentation or minority class classification.
• Cross-modal data translation and augmentation, especially when ground truth is scarce or ambiguous.

A plausible implication is that explicit modeling of conditional substructures—spatial, categorical, or latent—can partially overcome known GAN limitations regarding mode collapse, sample quality, and semantic controllability in challenging data regimes.

6. Empirical Findings and Limitations

• CT-to-DWI ctdGAN provides small, consistent improvement across segmentation metrics, with qualitative enhancement of infarct visibility, but without large absolute gains (Dice improvement 0.01, volume difference −1.3%). Statistical tests of significance were not reported (Rubin et al., 2019).
• Tabular ctdGAN shows statistically significant fidelity and classification enhancement over a wide range of baselines (rankings over 14 datasets, $$G^{*} = \arg\min_G \max_D \left\{ \mathcal{L}_{\mathrm{GAN}}(G, D) + \lambda\, \mathcal{L}_{L_1}(G) \right\}, \quad \lambda=100$$9), and ablations highlight the necessity of subspace-aware penalties and sampling (Akritidis et al., 1 Aug 2025).
• Student’s t-ctdGAN demonstrates sample diversity and quality gains in limited data, attributable to the heavy-tailed latent noise and trainable mixture modeling (Sun et al., 2018).
• For all variants, limited dataset size, the lack of formal statistical tests in some studies, and absence of ablation on model depth or specific architectural choices remain open factors meriting further investigation.