Scalable GANs with Transformers (2509.24935v1)

Abstract: Scalability has driven recent advances in generative modeling, yet its principles remain underexplored for adversarial learning. We investigate the scalability of Generative Adversarial Networks (GANs) through two design choices that have proven to be effective in other types of generative models: training in a compact Variational Autoencoder latent space and adopting purely transformer-based generators and discriminators. Training in latent space enables efficient computation while preserving perceptual fidelity, and this efficiency pairs naturally with plain transformers, whose performance scales with computational budget. Building on these choices, we analyze failure modes that emerge when naively scaling GANs. Specifically, we find issues as underutilization of early layers in the generator and optimization instability as the network scales. Accordingly, we provide simple and scale-friendly solutions as lightweight intermediate supervision and width-aware learning-rate adjustment. Our experiments show that GAT, a purely transformer-based and latent-space GANs, can be easily trained reliably across a wide range of capacities (S through XL). Moreover, GAT-XL/2 achieves state-of-the-art single-step, class-conditional generation performance (FID of 2.96) on ImageNet-256 in just 40 epochs, 6x fewer epochs than strong baselines.

• The paper proposes a novel transformer-based GAN (GAT) that leverages latent-space training to address early-layer underutilization and optimization instability.
• It implements Multi-level Noise-perturbed image Guidance (MNG) for coarse-to-fine synthesis and adaptive learning rate scaling to ensure stable training across model sizes.
• Experiments achieve a state-of-the-art FID of 2.96 on ImageNet-256 within 40 epochs, demonstrating the framework's efficiency and scalability.

Scalable GANs with Transformers: A Technical Analysis

Introduction

The paper "Scalable GANs with Transformers" (2509.24935) presents a systematic paper of Generative Adversarial Networks (GANs) scalability by leveraging transformer architectures and latent-space training. The authors introduce Generative Adversarial Transformers (GAT), a framework that combines pure transformer-based generators and discriminators operating in the latent space of a Variational Autoencoder (VAE). The work addresses two critical challenges in scaling GANs: underutilization of early generator layers and instability in optimization as model capacity increases. The proposed solutions—Multi-level Noise-perturbed image Guidance (MNG) and width-aware adaptive learning rate scaling—enable reliable training of GANs from small to extra-large scales, achieving state-of-the-art single-step generation performance on ImageNet-256.

Figure 1: Curated examples of GAT-XL/2 on ImageNet-256, demonstrating strong generation capability (FID 2.96) within 40 epochs and effective latent interpolation.

Architectural Design and Training Paradigm

Transformer-based GANs in Latent Space

GAT employs Vision Transformer (ViT) backbones for both generator and discriminator, with architectural modifications to facilitate adversarial training. The generator omits the patchify layer, instead using an unpatchify (linear decoder) to synthesize images from latent representations. Conditioning is achieved via a mapping network that produces style vectors from latent codes and class labels, modulating features through adaptive normalization and Layerscale. The discriminator similarly uses a ViT backbone with Layerscale and a dedicated $[\text{cls}]$ token for real/fake classification.

Training is performed in the latent space of a pre-trained VAE (SD-VAE), reducing computational cost and enabling efficient scaling. All models are trained at a spatial resolution of $32 \times 32$ in the latent space, corresponding to $256 \times 256$ in pixel space.

Multi-level Noise-perturbed image Guidance (MNG)

A key innovation is the MNG strategy, which activates early generator layers by providing intermediate supervision. The generator is divided into $K$ stages, each producing auxiliary outputs. These outputs are perturbed with Gaussian noise of decreasing strength along the depth, forming a coarse-to-fine synthesis trajectory. The discriminator receives all perturbed outputs, guiding each generator stage to learn the appropriate level of structure. This mechanism ensures uniform layer utilization and mitigates the tendency of transformers to concentrate synthesis in late layers.

Figure 2: Visualization of intermediate generator features and their effects; MNG activates early layers, yielding a coarse-to-fine synthesis process and improved perceptual contribution.

Adaptive Learning Rate Scaling

The authors identify that scaling model width and depth in GANs leads to instability if hyperparameters are not adjusted. Unlike diffusion models, GANs require learning rate adaptation proportional to channel dimension to maintain consistent update magnitudes. The proposed rule sets the learning rate $$\eta_{\text{adapt}} = \eta_{\text{base}} \cdot \frac{C_{\text{base}}}{C_{\text{model}}}$$, where $C$ is the channel size. This principled adjustment stabilizes training across scales without manual tuning.

Figure 3: Ablation paper showing that adaptive learning rate scaling ensures stable convergence across model sizes, while improper scaling leads to divergence or slow learning.

Empirical Results and Scalability Analysis

State-of-the-Art Performance

GAT-XL/2 achieves an FID of 2.96 on ImageNet-256 in only 40 epochs, outperforming strong baselines such as GigaGAN and MeanFlow, which require significantly more training epochs. The model maintains GAN advantages, including single-step inference and semantic latent space manipulation.

Scaling Behavior

Experiments across four model capacities (S, B, L, XL) demonstrate monotonic improvement in FID as model size increases. The scaling trend persists throughout training, not just at convergence, indicating robust scalability. Patch size experiments confirm that GAT is resilient to tokenization granularity, and GFLOPs analysis reveals a strong negative correlation ($-0.95$) between compute and FID, substantiating the claim that higher compute yields better generative performance.

Figure 4: Scalability of GAT; larger models and higher computational power systematically improve FID, confirming effective utilization of transformer scalability.

Ablation and Further Analysis

Ablation studies validate the necessity of MNG, adaptive learning rate, and VFM alignment objectives. MNG consistently enhances performance and distributes generative responsibility across layers. The REPA objective, aligning discriminator features with Vision Foundation Models (VFMs), further improves synthesis quality, indicating that representation learning advances from diffusion models transfer effectively to GANs.

Decoupled scaling of generator and discriminator shows that increasing discriminator capacity yields greater improvements in FID than scaling the generator, highlighting the centrality of discriminator representation learning in adversarial frameworks.

Figure 5: Decoupled scaling analysis; scaling the discriminator is more effective than scaling the generator, and feature alignment with VFMs is higher for fake data.

Implementation Details

The models are implemented using PyTorch with bfloat16 precision, gradient checkpointing, and Scaled Dot-Product Attention. The generator uses a latent code of dimension 64, with style modulation and Layerscale initialized to small values for stability. Four intermediate outputs are synthesized for MNG, spaced uniformly across transformer blocks. The discriminator employs a 3-layer MLP for VFM alignment and uses DINOv2-B as the reference model. Differentiable augmentation is applied in conjunction with noise perturbations.

For GAT-XL/2, training on ImageNet-256 for 40 epochs requires approximately 12 days on 8$\times$NVIDIA RTX A6000 GPUs. Hyperparameters are shared across model sizes except for the learning rate, which is adapted as described.

Theoretical and Practical Implications

The work demonstrates that GANs, when paired with transformer architectures and latent-space training, can be scaled reliably and efficiently, matching or surpassing the performance of diffusion and autoregressive models in single-step generation tasks. The findings challenge the prevailing notion that GANs are inherently less scalable than other generative paradigms. The proposed MNG and adaptive learning rate strategies are lightweight, easy to implement, and generalizable to other adversarial frameworks.

The strong empirical results suggest that further scaling and longer training could yield additional gains. The centrality of discriminator representation learning, as evidenced by decoupled scaling and VFM alignment, points to promising directions for future research, including more sophisticated alignment objectives and integration with foundation models.

Conclusion

"Scalable GANs with Transformers" provides a comprehensive framework for scaling GANs using transformer architectures and latent-space training. By addressing early-layer underuse and scale-coupled instability, the GAT framework achieves state-of-the-art single-step generation on ImageNet-256 with high data efficiency. The work establishes GANs as a competitive and scalable alternative to diffusion and autoregressive models, with practical implications for efficient high-fidelity image synthesis and theoretical insights into adversarial learning dynamics.