MicrobatchGAN Framework

• The paper introduces a microbatchGAN framework that uses coreset selection to mimic large minibatch diversity with minimal computational overhead.
• Its methodology involves oversampling candidates and applying greedy k-center selection with dimensionality reduction to ensure efficient, diverse microbatch formation.
• Empirical evaluations demonstrate improved mode coverage, lower FID scores, and enhanced anomaly detection across benchmarks like CIFAR-10 and ImageNet.

MicrobatchGAN, also referred to as Small-GAN in some literature, designates a family of generative adversarial network (GAN) training frameworks that address the limitations of conventional minibatch-based adversarial training via principled microbatch selection or microbatch-based discrimination. The general objective is to either match the gradient quality and diversity of large minibatches without prohibitive computational overhead or explicitly drive generator diversity through a multi-discriminator microbatch partitioning scheme. These methodologies leverage advances in coreset selection and multi-adversarial training to stabilize GAN optimization, enhance mode coverage, and enable scaling to hardware-limited or high-resolution training settings (Sinha et al., 2019, Mordido et al., 2020).

1. MicrobatchGAN via Coreset Selection: Problem Statement and Motivation

In classical GAN optimization, improvements in mode coverage and stability are disproportionately correlated with increasing the minibatch size. However, hardware constraints limit the feasible batch sizes, particularly for high-resolution image synthesis. MicrobatchGAN frameworks address this by constructing a synthetic minibatch of size $k$ that mimics the statistical diversity and gradient variance of an actual large batch $n \gg k$ at a fraction of the resource cost. The main concept is a two-step process at each draw of real or latent samples: (i) oversample a large pool of candidates; (ii) select an information-rich, diverse subset via a coreset-selection algorithm optimized for coverage with respect to an intrinsic metric, typically a Euclidean metric in an appropriate embedding space (Sinha et al., 2019). This enables GANs to operate as if training with a batch size $n = f\cdot k$, $f\gg 1$, while incurring only $O(k)$ memory and compute overhead per step.

2. Mathematical Foundations: Greedy k-Center Coreset Selection

Given an oversampled pool $P = \{p_1,\ldots,p_n\}$ (points in either latent space or data embeddings), microbatchGAN seeks to select a subset $Q \subset P, |Q| = k$, that solves the minimax facility-location objective:

$$\min_{Q \subset P,\, |Q|=k}\,\max_{x \in P} \min_{y \in Q} d(x, y)$$

where $d(\cdot,\cdot)$ is the relevant metric. The NP-hard problem is addressed via a classical greedy 2-approximation: initialize $Q$ by an arbitrary (or maximally distant) point, then, at each iteration, add the point in $n \gg k$0 farthest from the current coreset under $n \gg k$1. This ensures that the microbatch is maximally diverse, reflecting the original batch's statistical properties while controlling computational complexity to $n \gg k$2 per round (Sinha et al., 2019).

Table: Workflow of Coreset-Based Microbatch Selection

Step	Operation	Purpose
Oversampling	Draw $n \gg k$3 candidates from $n \gg k$4 or $n \gg k$5	Ensure broad coverage of modes/features
Embedding	For real data: Compute/cached high-D Inception features; project to low-D	Enable efficient selection in meaningful space
Coreset Selection	Run greedy k-center on embedded/latent points	Select diverse, information-rich microbatch

3. Embedding and Dimensionality Reduction for Real Data

Raw image data is not amenable to Euclidean distance-based selection due to the curse of dimensionality and semantic redundancy in pixel space. The microbatchGAN pipeline caches the Inception-v4 feature vector $n \gg k$6 ($n \gg k$7) for each training image. These features are further reduced by a random Gaussian projection $n \gg k$8, $n \gg k$9 to $n = f\cdot k$0 dimensions, typically $n = f\cdot k$1. The Johnson–Lindenstrauss lemma guarantees that pairwise distances in the reduced space are preserved up to $n = f\cdot k$2, enabling faithful approximation of the diversity structure while accelerating selection (Sinha et al., 2019). Selection directly in pixel space is expressly shown to degrade to random sampling.

4. MicrobatchGAN Training Algorithm

The microbatchGAN training loop replaces random minibatch sampling with the above "oversample + coreset-select" routine for both real and latent batches. The following steps are executed per iteration:

1. Sample $n = f\cdot k$3 latent codes, $n = f\cdot k$4 real images.
2. For real images: project Inception embeddings using $n = f\cdot k$5; apply greedy k-center to select $n = f\cdot k$6 samples.
3. For latent codes: apply greedy k-center directly.
4. Compute original GAN losses on these $n = f\cdot k$7 fake and $n = f\cdot k$8 real microbatches.
5. Update network parameters with the standard optimization rules.

This procedure operates as a fully generic wrapper: the only change required to the standard GAN codebase is the pre-processing and selection of microbatches via the coreset routine (Sinha et al., 2019).

5. Computational Complexity, Memory, and Effective Speed-Up

Let $n = f\cdot k$9 denote the microbatch size, $f\gg 1$0 denote the latent and data oversampling ratios. The coreset selection in latent space is $f\gg 1$1 per step; in projected embedding space it is $f\gg 1$2 but in $f\gg 1$3 where $f\gg 1$4. Empirically, with $f\gg 1$5, $f\gg 1$6, $f\gg 1$7, and $f\gg 1$8, the total overhead is $f\gg 1$9 per update on a Titan-XP GPU, which is negligible relative to neural forward/backward cost. The memory footprint is marginally increased by the storage of the $O(k)$0 cache, with the GPU still seeing only $O(k)$1 real plus $O(k)$2 fake images. The framework reliably emulates the variance and statistical coverage of much larger batches (e.g., 2–8$O(k)$3 increase), crucially reducing gradient variance and realizing lower empirical FID scores without further GPU burden (Sinha et al., 2019).

6. Empirical Evaluation and Key Results

Experiments were conducted on synthetic (2D mixture of Gaussians, 100 modes), anomaly detection (MNIST), and image synthesis benchmarks (CIFAR-10, LSUN, ImageNet):

• Mode Coverage (2D Gaussians):
  • Vanilla GAN: 90.7% recovered modes, 23.3% high-quality.
  • MicrobatchGAN: 97.3% recovered modes, 49.9% high-quality.
• Anomaly Detection (MNIST):
  • Integrating microbatchGAN into the Maximum Entropy Generator (MEG) pipeline improved AUPR by 10–25% across all tested anomalies.
• Image Synthesis (FID):
  • CIFAR-10 SN-GAN: FID for batch=128 improved from 18.75 to 16.73 with microbatch; for batch=512, FID decreased from 15.68 to 15.08.
  • LSUN (outdoor church) with SAGAN: Batch=64, FID improved from 14.82 to 13.08 (equivalent to baseline batch=128).
  • ImageNet SAGAN: FID improved from 19.40 to 17.33.
• Timing and Overhead:
  • On CIFAR-10, 50 steps: baseline batch=128 takes 13.31 s vs. microbatchGAN at 14.51 s ($O(k)$40.024 s overhead per step).
• Ablation:
  • Omitting random projection or coreset on both real/latent spaces raises FID; direct selection in pixel space is not competitive.

7. Extensions and Alternative Microbatch Schemes

MicrobatchGAN as described above is fundamentally orthogonal to several other microbatch paradigms:

• Moving-Window Microbatch Training: Maintains a buffer of "historical" latent codes, processing $O(k)$5 new samples per step. The batch-wise loss is evaluated over both historical and current codes, while the data-dependent gradient is back-propagated only through the new $O(k)$6 points. Empirical evidence shows that even with $O(k)$7, one can reach FID and sample quality comparable to full-batch training for WAE, MMD-GAN, and related generative frameworks, while scaling to ultra-high-resolution images on limited hardware (Spurek et al., 2019).
• Multi-Adversarial Microbatch Discrimination: In a distinct interpretation, microbatchGAN can refer to a setting where the minibatch is partitioned among $O(k)$8 discriminators, each assigned a "microbatch." A diversity parameter $O(k)$9 interpolates between standard real–vs–fake discrimination and an "inside vs outside" microbatch loss, guaranteeing generator diversity by making mode collapse suboptimal. Empirical results confirm that increasing $P = \{p_1,\ldots,p_n\}$0 and $P = \{p_1,\ldots,p_n\}$1 systematically improves Intra FID and sample diversity across MNIST, CIFAR-10, CelebA, and ImageNet, with up to 15% increase in the Inception Score (Mordido et al., 2020).

• Small-GAN: Speeding Up GAN Training Using Core-sets (Sinha et al., 2019)
• one-element Batch Training by Moving Window (Spurek et al., 2019)
• microbatchGAN: Stimulating Diversity with Multi-Adversarial Discrimination (Mordido et al., 2020)