Banach Wasserstein GAN

• Banach Wasserstein GAN is a generalization of Wasserstein GANs that replaces the Euclidean norm with arbitrary Banach space norms to capture nuanced image features.
• It enforces Banach–Lipschitz constraints using techniques like gradient penalties and spectral normalization to maintain training stability and optimal transport efficiency.
• Empirical evaluations on datasets like CIFAR-10 and CelebA demonstrate improved inception scores and FID, underscoring its tailored control over image synthesis quality.

The Banach Wasserstein Generative Adversarial Network (BWGAN) is a generalization of the Wasserstein GAN framework in which the underlying metric structure is extended from the Euclidean space with the $\ell^2$ norm to arbitrary Banach spaces equipped with a general norm $\|\cdot\|_B$. This extension enables practitioners to target nuanced distributional distances between probability measures, emphasizing specific image features such as edges, outliers, or global structure, by appropriate norm choice in the underlying Banach space. The BWGAN formalism encompasses both the classical WGAN with gradient penalty and alternative optimal transport-based training objectives, as demonstrated in multiple independent works (Adler et al., 2018, Laschos et al., 2019).

1. Banach Spaces, Duals, and Wasserstein Distances

A Banach space $B$ is a real normed vector space $(B,\|\cdot\|_B)$ that is complete with respect to the norm-induced metric. The topological dual $B^*$ consists of all bounded linear functionals $x^*:B\to\mathbb R$, equipped with the dual norm $\|x^*\|_{B^*} = \sup_{x\neq 0} |x^*(x)|/\|x\|_B$. The classical Wasserstein-1 distance between two probability measures $P_r$ and $P_g$ on $B$ is defined via the Kantorovich–Rubinstein duality:

$$W_1(P_r, P_g) = \sup_{f:B\to\mathbb R,\ \mathrm{Lip}_B(f)\leq 1} \mathbb E_{x\sim P_r} f(x) - \mathbb E_{x\sim P_g} f(x)$$

where $\mathrm{Lip}_B(f)$ denotes the minimal constant $\gamma$ such that $|f(x)-f(y)|\leq \gamma \|x-y\|_B$ for all $x,y\in B$ (Adler et al., 2018).

For general cost functions $c(x,y)$, the Wasserstein-$c$ distance is given by the Monge–Kantorovich optimal transport problem

$$W_c(P_r, P_\theta) = \inf_{\pi\in\Pi(P_r, P_\theta)} \int c(x,y)\, d\pi(x,y)$$

with dual formulations involving potential functions subject to $c$-Lipschitz constraints (Laschos et al., 2019).

2. Enforcing the Banach–Lipschitz Constraint

The Lipschitz constraint $|f(x)-f(y)|\leq \|x-y\|_B$ is characterized for Banach spaces via the norm of the Fréchet derivative $\partial f(x)\in B^*$: $f$ is $\gamma$-Lipschitz if and only if $\|\partial f(x)\|_{B^*} \leq \gamma$ for all $x$ (Adler et al., 2018). In the BWGAN critic (discriminator), this translates to enforcing $\|\partial D(x)\|_{B^*}\leq 1$. In practice, if $B\cong\mathbb R^n$, the dual norm is computed based on the usual gradient $\nabla g(u)\in\mathbb R^n$ via identification with the dual coordinates.

To impose this constraint during optimization, two principal approaches are employed:

• Gradient penalty: Add $$\lambda\, \mathbb E_{\hat x} [(\|\partial D(\hat x)\|_{B^*} - 1)^2]$$ to the critic loss, where $\hat x$ are interpolated between real and generated samples ($\hat x = t x + (1-t)x'$ for $t\sim\mathrm{Uniform}[0,1]$) (Adler et al., 2018).
• Weight or spectral normalization: Generalize traditional spectral normalization or weight clipping to bound the operator norm associated with the dual Banach norm, applicable to the Jacobian of the neural network layers (Laschos et al., 2019).

3. Specialization: $L^p$ and Sobolev Norms

The BWGAN framework accommodates a wide class of Banach norms. Prominent choices include:

• $L^p$ norms: For $p\in [1,\infty]$, $\|\cdot\|_p$ on $\mathbb R^n$ yields dual exponent $q$ with $1/p+1/q=1$, and the dual norm $\|\cdot\|_q$ is calculated on the gradient vector.
• Sobolev norms $W^{s,p}$: For domains $\Omega\subset\mathbb R^d$, the Sobolev norm is defined via the Fourier transform

$$\|x\|_{W^{s,p}} = \left(\int_\Omega | F^{-1}[(1+|\xi|^2)^{s/2} F x](t)|^p dt\right)^{1/p}$$

and the dual is $[W^{s,p}]^* = W^{-s,q}$. For integer $s$, this includes $L^p$ norms of $x$ and its weak derivatives up to order $s$. The implementation for Sobolev spaces involves mapping the gradient to the frequency domain, applying the appropriate weight, and evaluating the $\ell^q$ norm (Adler et al., 2018).

Qualitative effects of norm choice: negative $s$ in Sobolev norms accentuates low-frequency features (global structure), positive $s$ emphasizes high-frequency content (edges), while large $p$ in $L^p$-spaces increases sensitivity to outliers and localized discrepancies, often improving sharpness and sample detail.

4. BWGAN Training Algorithm and Implementation

The BWGAN objective generalizes the WGAN-GP adversarial training dynamics. The generator $G_\theta$ and the critic $D$ (potential $f$ or $\psi$) are parameterized by neural networks. Training proceeds with alternating updates:

• Critic step: Maximize

$$\mathbb E_{x\sim P_r}D(x) - \mathbb E_{x\sim P_g}D(x) + \lambda\, \mathbb E_{\hat x}[(\|\partial D(\hat x)\|_{B^*} - 1)^2]$$

(standard WGAN-GP when $\|\cdot\|_B = \ell^2$).

• Generator step: Minimize $\mathbb E_{x\sim P_g}D(x)$.

For general transport cost $c(x,y)=\|x-y\|_p$, especially in assignment-based BWGAN variants (Laschos et al., 2019), the generator update evaluates

$$L_G(\theta) = \frac{1}{N} \sum_{i=1}^N c(G_\theta(z_i), y_i)$$

where $$y_i = \arg\min_{y \in \mathrm{supp}(P_r)} [c(x_i, y) + \psi_w(y)]$$, and updates $\theta$ via backpropagation. The gradient penalty term adapts to the chosen dual norm.

Typical hyperparameters are inherited from WGAN-GP: Adam with learning rate $2\times 10^{-4}$, $\beta_1=0$, $\beta_2=0.9$, five critic steps per generator step, batch size 64. The penalty weight $\lambda$ is heuristically set to $\mathbb E_{x\sim P_r}\|x\|_B$; the scaling for critic outputs may be set to $\gamma \simeq \mathbb E_{x\sim P_r} \|x\|_{B^*}$.

5. Experimental Evaluation and Empirical Implications

BWGAN was empirically tested on CIFAR-10 and CelebA ($64\times 64$ resolution) with various $L^p$ and Sobolev $W^{s,2}$ norms. Evaluation utilized Inception Score (higher is better) and FID (lower is better):

Model / Norm	CIFAR-10 Inception Score	CIFAR-10 FID	CelebA FID
WGAN-GP ($\ell^2$)	$\approx 7.86 \pm 0.07$	—	—
BWGAN $W^{-3/2,2}$	$\approx 8.26 \pm 0.07$	—	Best for $s\in[-1,0]$
BWGAN $L^{10}$	$\approx 8.31 \pm 0.07$	—	Unstable at $p=10$
BWGAN $L^{4}$	—	$\approx 16.43$	Best for $p\in[2,5]$

Qualitative assessment confirmed that choice of norm controls the nature of synthesized images: negative Sobolev exponents bias toward global coherence, positive to edge sharpness, high $p$ accentuates local features and outlier intensity. On both datasets, BWGAN with suitable norm choice achieved improved Inception and FID scores relative to baseline WGAN-GP (Adler et al., 2018).

A plausible implication is that BWGAN confers finer control over learned distributional distances, supporting tailored image synthesis objectives through norm selection.

6. Extensions and Related Optimal Transport Frameworks

BWGAN encompasses a broader class of generative adversarial frameworks using general optimal transport cost functions $c(x,y)$, as formalized via the Monge–Kantorovich primal and dual problems (Laschos et al., 2019). The assignment-based dual approach yields objectives of the form

$$\sup_{\psi_w\in\mathrm{Lip}_c} \mathbb E_{x\sim P_r}[\psi_w^c(x)] - \mathbb E_{y\sim P_g}[\psi_w(y)]$$

where

$\psi_w^c(x) = \inf_y [c(x,y) + \psi_w(y)]$

and the generator update is implemented by minimizing

$$L_G(\theta) = \frac{1}{N}\sum_{i=1}^N c(G_\theta(z_i), y_i)$$

with $y_i$ obtained by assignment in the real data batch. This framework is stable and avoids mode collapse, with empirical evidence of consistent OT distance convergence and no observed failure cases under adequate batch coverage.

Concrete specializations include $c(x,y) = \|x-y\|_p$ with dual Lipschitz constraints implemented in terms of $\|\nabla \psi(x)\|_q \leq 1$, matching the Banach dual structure. For $p=2$ (standard Wasserstein-2), the update rules revert to classic WGAN-GP; for $p\ne2$, distinct gradient norms and penalties are introduced. Large real batch sizes are advantageous for high $p$ cost functions to capture support adequately (Laschos et al., 2019).

7. Significance and Summary

BWGAN decouples the Wasserstein GAN machinery from reliance on the $\ell^2$ metric, enabling distributional comparisons and training dynamics attuned to the statistical geometry most relevant to the application. By substituting the gradient-norm penalty in the critic loss with an arbitrary dual Banach norm, BWGAN enables practitioners to emphasize features such as low- or high-frequency content, edge structure, or outlier sensitivity in synthesized samples with minimal architectural changes. This generalization is mathematically rigorous and empirically validated, with competitive or superior results on canonical image synthesis benchmarks, and a straightforward implementation path for both $L^p$ and Sobolev norms (Adler et al., 2018, Laschos et al., 2019).