X2CT-GAN: 3D CT from Two X-Rays

• The paper introduces X2CT-GAN, a adversarial framework that reconstructs 128³ CT volumes from two 128×128 X-ray images using dual encoder-decoder streams.
• It leverages specialized 2D-to-3D transitions, skip connections, and a fusion strategy with projection-consistency loss to improve PSNR and SSIM over baselines.
• X2CT-GAN offers a low-cost method for gross anatomical reconstruction in resource-constrained settings, though it may struggle with fine structural details.

X2CT-GAN is a generative adversarial network (GAN)–based framework designed to reconstruct full 3D computed tomography (CT) volumes from just two orthogonal 2D X-ray projections, specifically the posterior–anterior (PA) and lateral views. This approach targets the clinically significant challenge of deriving volumetric anatomical information using minimal, low-cost imaging data, bypassing the need for a complete CT scanner or numerous rotational X-ray projections. The underlying methodology leverages learned anatomical priors and a specialized 2D-to-3D network architecture to address the severely under-determined nature of the problem (Ying et al., 2019).

1. Problem Formulation

X2CT-GAN aims to generate volumetric CT data (of dimension $128 \times 128 \times 128$ voxels) from two perpendicular 2D X-ray images (each $128 \times 128$), a scenario where classical CT reconstruction is inapplicable due to insufficient projections. This formulation eschews physical tomographic model constraints in favor of a purely data-driven adversarial learning approach, exploiting paired data for supervised mapping between biplanar X-rays and their corresponding CT volumes. The method requires learning strong anatomical priors from training data to plausibly approximate 3D structure from minimal 2D input (Ying et al., 2019).

2. Generator and Discriminator Architectures

The generator comprises three key subnetworks:

• Twin encoder–decoder streams: Each X-ray view (PA and lateral) feeds into a dedicated 2D encoder constructed from four dense modules (each with a 2D convolution, DenseNet-like connections, and channel compression), producing multi-scale feature maps at decreasing spatial resolutions and increasing channel depth.
• 2D–3D transition mechanisms:
  • Connection-A at the bottleneck converts the deepest 2D feature map into a 3D tensor via full flattening, a fully connected layer, and reshaping.
  • Connection-B serves as skip connections: 2D features at shallower levels are channel-matched using $1 \times 1$ convolutions, duplicated along the depth axis to form pseudo-3D blocks, and fused by basic 3D convolutions.
• Fusion network (Connection-C): At each decoder stage, features from the two views are aligned spatially and fused by element-wise averaging, producing a unified feature map, which is processed by further 3D deconvolutional blocks to output the final CT volume.

The discriminator, denoted "3DPatchDiscriminator," implements a 3D PatchGAN structure comprising four 3D convolution–instance-norm–ReLU blocks (three strides = 2, one stride = 1) with $4^3$ kernels, followed by a terminal 3D convolution yielding a patch-wise real/fake map (Ying et al., 2019).

3. Feature Fusion Strategy

The distinctive feature fusion integrates information from both X-ray views at each decoding layer. After 2D–3D bridging, the two streams' 3D feature tensors are spatially aligned and combined by computing their element-wise mean—no attention or gating mechanisms are employed. This design empirically suffices to resolve inconsistencies and propagate error gradients, resulting in blended anatomical cues from both views in the final volume. The fusion mechanism is critical for leveraging complementary projection information and addressing the inherent ambiguity of single-view reconstruction (Ying et al., 2019).

4. Training and Loss Functions

The training objective is a weighted sum of adversarial, reconstruction, and projection-consistency losses:

• Adversarial loss: Least Squares GAN (LSGAN) variant. The discriminator $D$ and generator $G$ are optimized as follows:

$$\mathcal{L}_{LSGAN}(D) = \frac{1}{2}\mathbb{E}_{y\sim p_{CT}}(D(y|x)-1)^2 + \frac{1}{2}\mathbb{E}_{x\sim p_{Xray}}(D(G(x)|x))^2$$

$$\mathcal{L}_{LSGAN}(G) = \frac{1}{2}\mathbb{E}_{x\sim p_{Xray}}(D(G(x)\mid x)-1)^2$$

• Reconstruction loss: voxel-wise mean squared error, $\mathcal{L}_{re} = \mathbb{E}_{x,y}\|y-G(x)\|_2^2$.
• Projection loss: enforces similarity of orthogonal projections (axial, coronal, sagittal) between generated and reference CTs:

$$\mathcal{L}_{pl} = \frac{1}{3}\sum_{v \in \{ax,co,sa\}}\mathbb{E}_{x,y}\|P_v(y) - P_v(G(x))\|_1$$

• Total objective: $G^*$ and $D^*$ are optimized as

$$G^* = \arg\min_G[\lambda_1\mathcal{L}_{LSGAN}(G) + \lambda_2\mathcal{L}_{re} + \lambda_3\mathcal{L}_{pl}]$$

$D^* = \arg\min_D\lambda_1 \mathcal{L}_{LSGAN}(D)$

with weights $\lambda_1=0.1$, $\lambda_2=\lambda_3=10$.

Training employs the LIDC-IDRI chest CT dataset (1,018 scans), resampled and paired with simulated digitally reconstructed radiographs. CycleGAN-driven style transfer is applied to real X-rays for domain harmonization. Optimization uses Adam ($2\times 10^{-4}$, $\beta_1=0.5$, $\beta_2=0.99$), batch size 1, and instance normalization (Ying et al., 2019).

5. Quantitative and Qualitative Evaluation

Performance evaluation on 102 held-out test volumes uses PSNR (dB) and SSIM:

Model (Input/Method)	PSNR (dB)	SSIM
2DCNN (single-view, baseline)	23.10 ± 0.21	0.461 ± 0.005
X2CT-CNN+S (MSE only, single)	23.12 ± 0.02	0.587 ± 0.001
X2CT-CNN+B (MSE, biplanar)	27.29 ± 0.04	0.721 ± 0.001
X2CT-GAN+S (GAN, single-view)	22.30 ± 0.10	0.525 ± 0.004
X2CT-GAN+B (GAN, biplanar)	26.19 ± 0.13	0.656 ± 0.008

Ablation studies show that both skip connections (Connection-B) and biplanar input improve reconstruction fidelity (PSNR and SSIM). The biplanar GAN yields sharper vasculature and organ boundaries compared to MSE-only or single-view networks. Inclusion of the projection loss and conditional discriminator further enhances anatomical plausibility. Qualitative assessment indicates realistic preservation of major structures, with real X-rays (post-style transfer) producing plausible 3D lung and mediastinal geometry (Ying et al., 2019).

6. Limitations and Application Scenarios

X2CT-GAN presents specific limitations: small anatomical structures remain susceptible to artifacts and the method is not intended to supplant standard CT for detailed diagnosis, being suited primarily for gross anatomy. The reliance on synthetic DRRs introduces domain adaptation challenges, for which CycleGAN-driven style transfer is partially, but not wholly, sufficient.

Potential application domains include rapid estimation of thoracoabdominal organ volumes and positions, pre- or intra-operative planning in X-ray-only environments, low-dose radiation therapy contexts, and enhancement of standard X-ray in resource-constrained settings. Use cases are described as “niche,” focusing on augmenting existing capabilities rather than providing comprehensive diagnostic imaging (Ying et al., 2019).

7. Related Work and Context

The X2CT-GAN approach diverges from traditional CT reconstruction, which requires dense (>100) angular projections and explicit tomographic modeling. Prior GAN applications focused on image-to-image translation or 2D domain mappings; X2CT-GAN is differentiated by its 2D-to-3D generative architecture and feature fusion strategy, addressing the unique dimensionality gap and leveraging sparse multi-view input. The work situates within broader research linking generative modeling, multi-modal medical imaging, and learning-based inverse problems (Ying et al., 2019).