Parallel GAN (P-GAN): Architectures & Optimization

• Parallel GAN (P-GAN) is a family of architectures that integrates parallel computing elements—such as multiple generators, decentralized workers, cellular networks, asynchronous updates, or tempered distributions—into the adversarial framework to enhance learning performance.
• Parallel GANs employ diverse methodologies like decentralized optimization, aggressive parameter sharing, and system-level co-design to mitigate mode collapse and improve convergence speed.
• Empirical evaluations on datasets including CIFAR-10, ImageNet, and MNIST demonstrate that P-GAN approaches achieve superior Inception Scores, lower FID metrics, and enhanced training efficiency compared to traditional GANs.

Parallel GAN (P-GAN) denotes a family of GAN constructions in which parallelism is introduced at different levels of the adversarial pipeline: multiple generators in a probabilistic mixture, multiple workers in decentralized or distributed optimization, multiple GAN instances in a cellular coevolutionary population, asynchronous separation of generator and discriminator execution, or simultaneous learning of a family of tempered target distributions (Hoang et al., 2017, Liu et al., 2019, Perez et al., 2020, Ren et al., 2021, Shi et al., 2024, Sohn et al., 2024). This suggests that P-GAN is not a single canonical architecture, but rather an umbrella label whose precise meaning is determined by the training topology, optimization design, and statistical objective adopted in a given work.

1. Terminological scope and recurring meanings

The literature uses “Parallel GAN” and “P-GAN” in materially different ways. In "A Decentralized Parallel Algorithm for Training Generative Adversarial Nets" (Liu et al., 2019), the focus is decentralized parallel optimization of a single GAN objective across workers, and the paper explicitly states that it does not coin the term Parallel GAN or P-GAN. In "Multi-Generator Generative Adversarial Nets" (Hoang et al., 2017), MGAN is presented as a P-GAN instantiation in which parallelism arises from a mixture of generators. In "A New Distributed Method for Training Generative Adversarial Networks" (Ren et al., 2021), P-GAN refers to a distributed framework with local discriminators and a single global generator. In "Parallel/distributed implementation of cellular training for generative adversarial neural networks" (Perez et al., 2020), the term denotes a spatial cellular coevolutionary system of many GANs trained concurrently on a distributed-memory platform. In "ParaGAN: A Scalable Distributed Training Framework for Generative Adversarial Networks" (Shi et al., 2024), the emphasis is asynchronous training and an asymmetric optimization policy. In "Parallelly Tempered Generative Adversarial Networks" (Sohn et al., 2024), the “parallel” aspect comes from jointly learning tempered distributions indexed by a temperature variable.

Usage of parallelism	Core mechanism	Representative paper
Parallel generators	Mixture of $K$ generators with one discriminator and one classifier	(Hoang et al., 2017)
Decentralized workers	Mixing matrix $W$, multiple communication rounds, simultaneous updates	(Liu et al., 2019)
Server-device distribution	Local discriminators on devices, global generator at server	(Ren et al., 2021)
Cellular coevolution	Many GANs on a 2D toroidal grid with local neighborhoods	(Perez et al., 2020)
Asynchronous system split	Bounded-staleness buffers and asymmetric G/D optimization	(Shi et al., 2024)
Parallel tempering	Shared $G,D$ over interpolated distributions $Q^\alpha$	(Sohn et al., 2024)

A common misconception is to treat P-GAN as synonymous with Progressive GAN. The cellular-training paper explicitly distinguishes “P-GAN” in this context from Progressive GAN, stating that the former refers to a Parallel GAN approach based on multiple GAN instances trained concurrently in a spatial grid, whereas Progressive GAN progressively grows network resolution and is unrelated to parallel or distributed training (Perez et al., 2020). A related clarification appears in the distributed server-device work, which states that P-GAN is unrelated to Progressive GAN (ProGAN) or Perceptual GAN (Ren et al., 2021).

2. Mixture-based P-GAN: multiple generators with a shared discriminator and classifier

MGAN instantiates P-GAN by replacing a single generator with a parallel mixture of $K$ generators $\{G_k\}$, trained jointly against a single discriminator $D$ and a single auxiliary classifier $C$ (Hoang et al., 2017). Internal samples are created by all generators, and one of them is randomly selected as the model’s final output according to fixed mixture coefficients $\pi = [\pi_1,\dots,\pi_K]$. The sampling mechanism is

$$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$

so that the model distribution is

$W$0

Here $W$1 is a binary discriminator that distinguishes real versus fake mixture samples, while $W$2 is a multi-class classifier that predicts which generator produced a given synthetic sample.

The global minimax objective is

$W$3

with

$W$4

For fixed generators and mixture weights, the optimal discriminator and classifier are

$W$5

Substituting these into the objective yields

$W$6

Accordingly,

$W$7

The first term drives the mixture towards the data distribution, and the second term, weighted by $W$8, pushes generator components apart to reduce overlap. The paper’s equilibrium analysis states that the Jensen–Shannon divergence between the mixture of generators’ distributions and the empirical data distribution is minimal, whilst the JSD among generators’ distributions is maximal, hence effectively avoiding the mode collapse.

A central design feature is aggressive parameter sharing. All generators share parameters in all layers except their input layer, and $W$9 and $G,D$0 share parameters in all layers except their output layer. The generators thus differ only in the mapping from $G,D$1 to the first hidden layer, while the shared trunk performs image synthesis. This keeps parameter count close to a single-GAN baseline and adds minimal computational cost to the standard GAN. The training algorithm alternates between updating the shared $G,D$2 trunk plus their separate heads, and updating the shared generator parameters with the non-saturating loss

$G,D$3

Empirically, MGAN was evaluated on synthetic 2D data, CIFAR-10, STL-10, and ImageNet. On CIFAR-10, the reported Inception Scores are DCGAN $G,D$4, D2GAN $G,D$5, DFM $G,D$6, and MGAN $G,D$7; on STL-10 (48×48), MGAN reports $G,D$8, and at 96×96, $G,D$9; on ImageNet (32×32), MGAN reports $Q^\alpha$0 (Hoang et al., 2017). The paper further reports that generators specialize early and become consistent later, for example one generator focusing on ships, another on cars/trucks, and others on flying objects or animal portraits. This supports the paper’s interpretation that the classifier term induces both diversity and specialization rather than merely increasing parameter count.

3. Optimization- and communication-based P-GAN: decentralized and distributed training

One major meaning of P-GAN concerns the parallelization of a single GAN training problem across workers or devices. In the decentralized formulation of (Liu et al., 2019), the objective is a general nonconvex-nonconcave min–max stochastic optimization problem,

$Q^\alpha$1

The variables are concatenated as $Q^\alpha$2, with saddle-point gradient mapping

$Q^\alpha$3

Workers are connected by an undirected graph $Q^\alpha$4 with a symmetric doubly-stochastic mixing matrix $Q^\alpha$5, and spectral parameter

$Q^\alpha$6

The algorithmic design uses multiple communication rounds per iteration and simultaneous generator/discriminator updates. In matrix form,

$Q^\alpha$7

Per worker $Q^\alpha$8,

$Q^\alpha$9

The algorithm uses optimistic gradient updates, equivalently single-step extragradient or “one-call extragradient” in differentiable games, and does not use gradient tracking or variance reduction in the core theoretical algorithm.

The theoretical result is a non-asymptotic convergence guarantee to first-order stationarity under $K$0-Lipschitz continuity of $K$1, unbiased stochastic gradients with bounded variance, bounded gradients, and a coherence-type assumption. If the number of communication rounds $K$2 and stepsize $K$3 satisfy the stated conditions, then

$K$4

The paper states that this is the first gradient-based decentralized parallel algorithm that allows workers to have multiple rounds of communications in one iteration and to update the discriminator and generator simultaneously. In the centralized special case, if $K$5 has all entries $K$6 and $K$7, the algorithm reduces to CPOSG with computational complexity $K$8 and busiest-node communication complexity $K$9. With $\{G_k\}$0, $\{G_k\}$1, and $\{G_k\}$2, the total computation complexity is $\{G_k\}$3, while under well-connected graphs or random mixing on a complete graph the busiest-node total communication to reach $\{G_k\}$4-stationarity is $\{G_k\}$5 (Liu et al., 2019).

The practical variants used in experiments are CP-OAdam, DP-OAdam, and Rand-DP-OAdam. These were evaluated on WGAN-GP for CIFAR-10 and SAGAN for ImageNet, using 16 learners with total batch size fixed at 256 in both an HPC low-latency environment (approximately $\{G_k\}$6) and a cloud high-latency environment (approximately $\{G_k\}$7). The reported findings are that DP-OAdam matches CP-OAdam convergence in terms of epochs under low latency, Rand-DP-OAdam improves convergence further due to random mixing, and both decentralized methods significantly outperform centralized all-reduce in run-time and speedup on high-latency networks.

A different distributed instantiation appears in (Ren et al., 2021), where each device computes a local discriminator using local data while a single server aggregates their results and computes a global GAN. If $\{G_k\}$8 is the scheduled subset of devices and $\{G_k\}$9 are local batch sizes, then the global discriminator parameters are updated by the sample-size-weighted average

$D$0

The server then updates the global generator through

$D$1

Two update schedules are proposed. The parallel update schedule lets server and devices simultaneously update their models based on the same previous $D$2, reducing round time but introducing mild staleness. The serial update schedule first updates local discriminators, then aggregates and broadcasts $D$3, and finally updates the generator with the fresh discriminator, producing tighter coupling and typically fewer rounds to the stopping criterion. Using DCGAN, the paper reports $D$4 parameters for the generator and $D$5 for the discriminator, with 16-bit parameter exchange. Device upload per scheduled device is approximately $D$6 MB, server broadcast per device per round is approximately $D$7 MB, and P-GAN uploads only discriminators, reducing device-to-server uplink by approximately $D$8 versus FedGAN. On CelebA, CIFAR-10, and RSNA Pneumonia, both schedules converge in FID, the serial schedule converges faster than the parallel schedule, and scheduling the top 20–50% of devices significantly improves speed relative to scheduling 100% of devices under poor channels (Ren et al., 2021).

Taken together, these two strands show that optimization-based P-GAN can mean either decentralized peer-to-peer consensus with simultaneous updates or centralized aggregation with adversarial-role splitting across devices and server. The papers agree, however, that communication topology and synchronization schedule are first-order determinants of both convergence and wall-clock behavior.

4. Cellular coevolutionary P-GAN: spatial populations of GANs

A distinct P-GAN lineage replaces one adversarial game with a population of locally interacting games. The distributed cellular-training framework of (Perez et al., 2020) presents a parallel/distributed implementation of a cellular competitive coevolutionary method to train two populations of GANs. The organization is a 2D toroidal grid of size $D$9, each cell hosting a GAN composed of a generator and a discriminator. Neighborhoods are overlapping Moore neighborhoods of size $C$0, containing the center plus its West, North, East, and South neighbors. Because interactions are restricted to neighborhoods, the cost of naïvely evaluating all generator–discriminator pairings, $C$1, is reduced to $C$2 per neighborhood across $C$3 neighborhoods.

Training proceeds by local adversarial learning inside each cell against the neighborhood’s sub-populations, followed by exchange of center models after each training epoch. Selection within neighborhoods is tournament-based with tournament size $C$4. Mixture mutation and hyperparameter mutation are also used: mixture mutation scale $C$5, hyperparameter mutation probability $C$6, mutation rate $C$7, and Adam with initial learning rate $C$8. Population size per cell is $C$9 in the reported experiments, but the framework supports larger populations. The adversarial objective is stated in general form as

$\pi = [\pi_1,\dots,\pi_K]$0

where

$\pi = [\pi_1,\dots,\pi_K]$1

For completeness, the paper also gives the original Goodfellow objective

$\pi = [\pi_1,\dots,\pi_K]$2

The distributed-memory implementation uses a two-level parallel model: multithreading inside each process and MPI via mpi4py across processes. The orchestration is master–slave. WORLD is the global communicator used at startup and for status control, LOCAL includes only active slaves within a given grid and is used for collective operations between slaves, and GLOBAL includes all slaves and the master for final collective operations. Training is asynchronous within cells, with synchronization primarily at epoch boundaries when exchanging center models. The paper notes that communication time remained roughly constant while compute-heavy steps benefited significantly from distribution.

The experiments use MNIST with MLP generators and discriminators: latent dimension $\pi = [\pi_1,\dots,\pi_K]$3, two hidden layers with $\pi = [\pi_1,\dots,\pi_K]$4 neurons per hidden layer, output dimension $\pi = [\pi_1,\dots,\pi_K]$5, tanh activation, batch size $\pi = [\pi_1,\dots,\pi_K]$6, skip $\pi = [\pi_1,\dots,\pi_K]$7 discriminator steps $\pi = [\pi_1,\dots,\pi_K]$8, and $\pi = [\pi_1,\dots,\pi_K]$9 iterations. The platform is Cluster-UY with Xeon Gold 6138 CPUs, and the implementation is CPU-only in Python 3, PyTorch, and mpi4py. Reported timing results are: for a $$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$0 grid, single-core $$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$1 min versus distributed $$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$2 min, speedup $$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$3 on $$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$4 cores, efficiency approximately $$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$5; for $$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$6, single-core $$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$7 min versus distributed $$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$8 min, speedup $$z \sim P_z,\qquad u \sim \mathrm{Mult}(\pi_1,\dots,\pi_K),\qquad x = G_u(z),$$9 on $W$00 cores, efficiency approximately $W$01; for $W$02, single-core $W$03 min versus distributed $W$04 min, speedup $W$05 on $W$06 cores, efficiency approximately $W$07 (Perez et al., 2020).

This framework differs sharply from data-parallel, model-parallel, or asynchronous single-pair GAN training. The paper’s position is that local neighborhoods foster diversity and stabilize training through a population-based arm race, while distributed memory HPC execution reduces training times and scales properly when considering different grid sizes for training. At the same time, the paper does not report numerical quality metrics such as FID or Inception Score, and the experiments are limited to MNIST.

5. System-co-designed P-GAN: asynchronous training, asymmetric optimization, and accelerator utilization

ParaGAN formulates P-GAN as a co-designed system-and-optimization approach to scalable, stable GAN training (Shi et al., 2024). Its central claim is that two intertwined problems must be handled jointly: the computational intensity of high-resolution, large-batch GANs and their numerical instability, including mode collapse and oscillations. The framework therefore decouples and parallelizes the generator and discriminator via asynchronous training, while stabilizing the adversarial dynamics through an asymmetric optimization policy that treats the two networks differently.

The asynchronous update scheme introduces two bounded-staleness buffers: img_buff, which stores generated images produced by generator workers at prior steps, and pred_buff, which stores discriminator predictions or logits produced by discriminator workers at prior steps. The updates are written as

$W$08

where $W$09 and $W$10 are small staleness values, often one iteration. Within each sub-network, gradients are aggregated with all-reduce; cross-network communication is limited to enqueuing generated images and logits into buffers rather than forcing strict synchronous cross-parameter dependencies.

ParaGAN supports standard non-saturating GAN loss,

$W$11

and the hinge-loss formulation commonly used by BigGAN,

$W$12

For conditional BigGAN with a projection discriminator,

$W$13

The asymmetric optimization policy assigns a more adaptive optimizer such as AdaBelief to the generator and a robust optimizer such as Adam to the discriminator, with independent hyperparameters for optimizer choice, learning-rate schedules, warmup, gradient norm clipping, and even distinct batch sizes or update ratios.

The system side is equally prominent. A congestion-aware data pipeline monitors I/O latency with a sliding window and dynamically scales prefetch threads and buffer sizes to keep accelerators fed. A hardware-aware layout transformation opportunistically batches and reshapes tensors to accelerator-friendly layouts, reducing zero-padding waste and improving matrix-unit occupancy. Mixed precision uses bf16 for activations and fp32 for sensitive weights, gradients, and final layers of $W$14 and $W$15.

The reported throughput ablation on BigGAN 128×128, ImageNet, 128 TPUv3 accelerators, and global batch $W$16 is: baseline $W$17 images/sec; with congestion-aware pipeline $W$18 (+10.8%); with layout transformation $W$19 (+3.9% from previous); and with mixed precision $W$20 (+15.2% from previous; approximately 32% over baseline). Under strong scaling with fixed global batch $W$21, time-to-solution drops from more than $W$22 hours to approximately $W$23 hours as workers increase. Under weak scaling, near-flat steps/sec up to $W$24 workers is reported. The framework reaches $W$25 scaling efficiency at $W$26 TPU accelerators and reduces BigGAN training time from $W$27 days on $W$28 V100s to $W$29 hours on $W$30 TPUs, while enabling direct $W$31 image generation on ImageNet. For image quality, the paper reports Inception Score $W$32 and FID $W$33 for BigGAN trained directly at $W$34 resolution (Shi et al., 2024).

The limitations are also explicit. Asynchrony introduces staleness; while the discriminator empirically tolerates small staleness in generated inputs, very large staleness can degrade convergence, especially at very high resolutions. The paper therefore recommends a phase switch to synchronous updates in the later phase at high resolution for best convergence. This is a systems-level P-GAN in the strongest sense: parallelism is not merely a matter of replication, but is coupled to optimizer asymmetry, buffer semantics, data ingress, tensor layout, and precision policy.

6. Tempered-distribution P-GAN: simultaneous learning across a temperature ladder

Parallelly Tempered GANs define P-GAN through a different kind of parallelism: a single shared generator and discriminator are trained to simultaneously model a family of tempered distributions indexed by $W$35 (Sohn et al., 2024). Because the unknown density $W$36 cannot be directly tempered as $W$37, the paper constructs tempered distributions by convex interpolation of data points. For paired real samples $W$38,

$W$39

and for paired latent variables,

$W$40

The temperature variable is sampled from

$W$41

For images, the paper uses $W$42; for tabular utility experiments, $W$43; and for the fairness variant, $W$44.

Training is posed on the joint distributions of $W$45 and $W$46, with neural distance

$W$47

The empirical loss is

$W$48

A single $W$49 and $W$50 are shared across all temperatures; $W$51 is appended as an extra input dimension, and symmetry is enforced by passing

$W$52

instead of $W$53.

A novel coherency regularizer is added to prevent the model from degenerating into effectively separate GANs for different $W$54 values. The penalty is

$W$55

where $W$56 and $W$57 are two interpolations from the same pair, and $W$58. The practical discriminator and generator losses are

$W$59

The paper’s theory links GAN training instability to multimodality through gradient variance. Under the stated assumptions on bounded parameters, Lipschitz activations, bounded supports, and positive homogeneity, the paper gives gradient bounds of the form

$W$60

and under tempering,

$W$61

The stated conclusion is that, if the tempered neural distance is smaller than the vanilla $W$62 distance, the upper bound of gradient variance under tempering is strictly smaller than vanilla training. The estimator is also shown to achieve a nearly optimal $W$63 minimax rate.

Empirically, PTGAN is evaluated on CIFAR10, TinyImageNet, and tabular datasets. With PT + CP and $W$64, the reported best image-synthesis scores include CIFAR10 under JSD with IS $W$65 and FID $W$66, TinyImageNet under JSD with IS $W$67 and FID $W$68, and CIFAR10 under ND with IS $W$69 and FID $W$70. On Adult, Law School Admission, and Credit Card Default, PT+CP consistently improves the average discrepancy $W$71 relative to MP. For fairness, the paper defines statistical parity

$W$72

and introduces FairPTGAN through cross-group interpolation. Reported examples include Adult RF at AUC $W$73, where SP is $W$74 for PT versus $W$75 for FairWGANGP and $W$76 for GeoRepair, and Law School LR at AUC $W$77, where SP is $W$78 versus $W$79 and $W$80 (Sohn et al., 2024).

The broader conceptual implication is that P-GAN can refer not only to parallel hardware execution or multiple adversarial agents, but also to a joint adversarial problem defined over multiple related target distributions. This suggests a useful boundary for the term: “parallel” in GAN research has come to mean parallel generators, parallel workers, parallel games, parallel update streams, or parallel temperatures, rather than any single fixed recipe.