Relativistic Critic in GANs

• Relativistic critic is an advanced discriminator that compares real and fake samples, yielding statistically sound divergences and enhanced gradient dynamics.
• It reformulates standard GAN discrimination by coupling outputs for real and fake batches, mitigating issues like vanishing gradients and overconfidence.
• This approach demonstrates improved sample quality, stability, and faster convergence in applications ranging from image synthesis to reinforcement learning in large language models.

A relativistic critic, also called a relativistic discriminator, is an architectural and objective reformulation of the discriminator module in adversarial frameworks—notably generative adversarial networks (GANs)—in which the discriminator does not estimate the solitary realism of a sample (“how real is $x$?”) but instead evaluates samples in a relative fashion (“how much more real is $x$ than $y$?”). This approach replaces the classical pointwise discrimination in standard GANs with a formulation that inherently couples the outputs for real and fake examples, producing well-characterized statistical divergences and leading to improved empirical stability, sample quality, and optimization characteristics in both generative modeling and modern LLM reinforcement learning paradigms (Jolicoeur-Martineau, 2018, Cai et al., 26 Nov 2025, Jolicoeur-Martineau, 2019, Nguyen et al., 2021).

1. Mathematical Definition and Formulation

The canonical relativistic discriminator does not estimate $P(x \;\text{is real})$ directly. Instead, it estimates the probability that a real sample $x_r\sim P$ is more realistic than a generated (fake) sample $x_f\sim Q$: $$D_{\mathrm{rel}}(x_r, x_f) = \sigma(C(x_r) - C(x_f))$$ where $C(\cdot) \in \mathbb{R}$ is the real-valued critic/logit and $\sigma(\cdot)$ is the sigmoid activation. This construction is extended to multiple variants:

• Relativistic Standard GAN (RSGAN):

$$L_D^{\mathrm{RSGAN}} = -\mathbb{E}_{x_r, x_f}\left[\log \sigma(C(x_r) - C(x_f))\right]$$

$x$0

• Relativistic Average GAN (RaGAN): Compares each sample’s score to the batch-mean of the opposite type.

$x$1

For a real $x$2:

$x$3

Losses become:

$x$4

• Extension to Arbitrary $x$5-Divergences: Any $x$6-divergence GAN objective

$x$7

is relativized by replacing arguments $x$8 (or means), generalizing to paired and average relativistic $x$9-divergence forms (Jolicoeur-Martineau, 2019).

This construction ensures the discriminator's output is directly influenced by both real and fake batches, producing a margin-based, pairwise interaction rather than independent scalar values.

2. Motivation and Theoretical Benefits

The relativistic formulation is motivated by several deficiencies in classical SGAN learning:

• Symmetry and Coupling: In a mini-batch containing equal numbers of real and fake samples, making fake samples more realistic (raising their score) should naturally reduce the discriminator confidence in real samples. SGAN's pointwise setup lacks this coupling, potentially allowing for degenerate solutions.
• Divergence Minimization: Standard SGAN approximates the Jensen-Shannon divergence, where optimal convergence should force discriminator outputs for both real and fake to $y$0. However, classical SGAN only forces $y$1, producing overconfident discriminators and vanishing gradients on the real side.
• Gradient Dynamics: In IPM-GANs, the critic's gradient always mixes $y$2 and $y$3, so both influence updates. In non-relativistic SGANs, perfect discrimination causes the real-sample gradient to vanish, leading to “critic stalling.” Relativistic discriminators keep both gradients active throughout training.
• Statistical Divergence Properties: For a broad class of concave $y$4 functions, relativistic GAN objectives define bona fide statistical divergences, which are (topologically) strictly stronger than their classical counterparts, yet yield better optimization characteristics (Jolicoeur-Martineau, 2019).

3. Variants and Generalizations

Multiple variants extend the core relativistic concept:

• Relativistic Paired GAN (RpGAN): Uses strictly pairwise comparisons:

$y$5

• Relativistic Average (RaGAN): Each sample is compared to the opposing batch mean.
• Further Generalizations:
  • RalfGAN: One-sided relativization (only real batch compared to fake mean).
  • RcGAN: Centered scores using the mean of both distributions.

The choice of relativization (paired, batch mean, mixed mean) impacts both divergence strength and estimator bias/noise characteristics.

$y$6

with topological strength $y$7 (Jolicoeur-Martineau, 2019).

4. Algorithms and Implementation

The relativistic critic is implemented with minimal changes to canonical GAN architectures. The algorithmic steps are as follows:

• For RSGAN and RpGAN:

1. Sample batches of real $y$8 and fake $y$9.
2. For $P(x \;\text{is real})$0 discriminator steps, update $P(x \;\text{is real})$1 by maximizing the paired loss over $P(x \;\text{is real})$2.
3. After $P(x \;\text{is real})$3 steps, update the generator (or policy) using the reversed loss.

• For RaGAN:

1. Compute batch means $P(x \;\text{is real})$4, $P(x \;\text{is real})$5 of critic scores.
2. For each $P(x \;\text{is real})$6, use relativistic score differences with mean.
3. Apply spectral normalization or gradient penalty for stability.
4. Empirically, a single discriminator update per generator update suffices—no need for frequent critic steps.

• In LLM Reasoning/RL (RARO):
  • Policy $P(x \;\text{is real})$7 and critic $P(x \;\text{is real})$8 are trained adversarially.
  • Critic compares each expert answer $P(x \;\text{is real})$9 to a policy (model) answer $x_r\sim P$0 via $x_r\sim P$1.
  • The policy is updated using PPO, with the critic's relativistic margin as reward.
  • Stabilization relies on two-time-scale updates, gradient penalties, and normalization (Cai et al., 26 Nov 2025).

5. Empirical Performance and Observed Effects

Extensive empirical studies demonstrate:

• Stability: Relativistic GANs (RSGAN, RaGAN, RaLSGAN, RMCosGAN) yield systematically lower FID variance and improved convergence as compared to non-relativistic GANs, across multiple datasets and architectures.
• Sample Quality: RaGAN with gradient penalty matches or surpasses WGAN-GP in FID with inexpensive, single-step D updates ($x_r\sim P$2400% faster to reach SOTA). High-resolution image synthesis (e.g., 256x256 with $x_r\sim P$3) becomes tractable where SGAN, LSGAN, and even WGAN-GP collapse.
• Comparative Performance:

| Loss | FID (CIFAR-10, $x_r\sim P$4=1) | FID (CAT, 64x64, min) | |--------------|------------------------|----------------------| | SGAN | 40.64 | 16.56 | | RSGAN | 36.61 | 19.03 | | RaSGAN | 31.98 | 15.38 | | WGAN-GP | 83.89 | >155 (at 256x256) | | RSGAN-GP | 25.60 | — | | RaLSGAN | — | 11.97 |

(Jolicoeur-Martineau, 2018, Nguyen et al., 2021)

• Loss Function Variants: Relativistic margin cosine loss (RMCosGAN) outperforms both CE and Ra-LS loss in FID and IS; e.g., CIFAR-10 FID = $x_r\sim P$5 for RMCosGAN vs $x_r\sim P$6 for CE (Nguyen et al., 2021).
• Estimator Bias / Variance: Minimum-variance unbiased estimators for judiciously relativistic divergences do not improve sample quality—additional estimation noise regularly acts as implicit regularization, improving the generator (Jolicoeur-Martineau, 2019).

6. Applications Beyond Image Synthesis

The relativistic critic is central not only in classic image-based GANs but in modern adversarial RL and imitation learning, especially in LLM training:

• RARO (Relativistic Adversarial Reasoning Optimization): Develops expert-level reasoning in LLMs where verifiers are absent, using a relativistic critic to deliver policy improvement signals (Cai et al., 26 Nov 2025).
• RL Policy Updates: The reward signal is the pairwise margin between model and expert response; this “relativizes" the imitation objective and prevents reward collapse or saturation.

7. Architectural and Hyperparameter Recommendations

• Standard Practice: DCGAN backbone, spectral normalization in $x_r\sim P$7, batch norm in $x_r\sim P$8, Adam optimizer ($x_r\sim P$9, $x_f\sim Q$0, $x_f\sim Q$1), batch size 64, $x_f\sim Q$2 sufficient for stability (Jolicoeur-Martineau, 2018, Nguyen et al., 2021).
• RARO Critic: Small MLP head (hidden size 512) atop a frozen LLM encoder; spectral norm or gradient penalty regularization; critic LR $x_f\sim Q$3 vs policy LR $x_f\sim Q$4; 2–5 critic steps per generator update; reward normalization/clipping (Cai et al., 26 Nov 2025).
• Critical Hyperparameters: For RMCosGAN, angular margin $x_f\sim Q$5 must be carefully tuned ($x_f\sim Q$6 recommended), too large or too small leads to instability or feature collapse (Nguyen et al., 2021).

---

In sum, the relativistic critic fundamentally shifts adversarial learning from isolated sample scoring to paired, margin-based discrimination. This approach yields statistically principled divergences, improves optimization dynamics, and consistently enhances stability and generative performance across adversarial machine learning domains (Jolicoeur-Martineau, 2018, Jolicoeur-Martineau, 2019, Nguyen et al., 2021, Cai et al., 26 Nov 2025).