Fast Spectral Normalization (FSN)

• Fast Spectral Normalization (FSN) is a family of efficient normalization methods that reduce the computational cost of estimating layer spectral norms while enforcing Lipschitz continuity.
• FSN leverages operator-level, structure-aware, and FFT-based approximations to control spectral norms in convolutions, enabling improved robustness against adversarial attacks.
• Empirical studies indicate that FSN improves test accuracy and robustness with minimal overhead compared to traditional spectral normalization methods in various neural network architectures.

Searching arXiv for the core FSN papers and closely related spectral normalization work. Fast spectral normalization (FSN) denotes a family of efficiency-oriented spectral-normalization procedures that preserve Lipschitz control while reducing the cost of estimating or enforcing layer spectral norms. In the cited literature, the term is used in at least three closely related senses: an efficient operator-level normalization scheme for adversarial training of networks with arbitrary convolution stride and padding (Farnia et al., 2018), a structure-aware method for depthwise separable convolutions (Runkel et al., 2021), and an FFT- and layer-separation-based approximate regularizer for robust convolutional networks (Pan et al., 2021). These variants are best understood relative to the original spectral normalization formulation for GAN discriminators, which normalizes each linear or convolutional layer by an estimate of its largest singular value and thereby constrains the network Lipschitz constant (Miyato et al., 2018).

1. Mathematical basis

The common mathematical object across FSN variants is the spectral norm of a linear operator. For a matrix $$W \in \mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}$$, the spectral norm is

$$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$

equivalently the largest singular value. In the adversarial-robustness literature, this is also written as

$$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$

emphasizing that $\sigma(W)$ is the tight worst-case amplification factor for input perturbations (Miyato et al., 2018, Pan et al., 2021).

For a feedforward network

$$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$

if each activation $a_l$ is $1$-Lipschitz, then the network Lipschitz constant is upper-bounded by the product of layer spectral norms,

$$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$

This product also appears in adversarial margin-based generalization bounds, together with related quantities such as $\lip(\nabla \ell \circ f_{\mathbf{w}})$, so constraining every $\|W_i\|_2$ directly controls both sensitivity to perturbations and the capacity terms entering the bounds (Farnia et al., 2018).

The original layer-wise normalization rule is

$$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$0

which enforces $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$1. Later supervised and adversarial-training variants instead use a capped form,

$$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$2

which enforces $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$3 without forcing equality (Farnia et al., 2018). In all cases, FSN retains the same conceptual target: cheap, repeated estimation or control of $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$4 during training.

2. Baseline spectral normalization and the original computational profile

The immediate precursor to FSN is the discriminator normalization method introduced for GANs by Miyato et al. Spectral normalization was proposed to stabilize GAN training by directly controlling discriminator capacity. The motivating failure mode was an unstable discriminator that becomes too sharp, perfectly separates real and fake samples in regions where supports are disjoint, and therefore produces near-zero gradients for the generator, leading to collapse or severe mode dropping (Miyato et al., 2018).

In that formulation, every linear or convolutional layer of the discriminator is replaced at each update by a normalized operator $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$5. Convolution kernels $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$6 are reshaped into a matrix of shape $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$7 when computing $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$8. The generator is generally not spectrally normalized in the original experiments (Miyato et al., 2018).

The key approximation is one-step power iteration per training step per layer. Starting from a unit vector, the method alternates

$$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$9

and estimates the spectral norm by a Rayleigh quotient such as

$$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$0

The previous singular-vector estimate is reused at the next step, exploiting the fact that SGD changes $$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$1 only slightly between updates (Miyato et al., 2018).

This baseline is already relatively cheap. The reported cost is about $$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$2–$$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$3 that of a vanilla GAN on CIFAR-10, and much lower than WGAN-GP, whose gradient penalty requires an extra backward pass. In practice, the resulting layer spectral norms hover around $$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$4–$$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$5 during training (Miyato et al., 2018). The bottleneck that later FSN work addresses is therefore not the existence of spectral normalization itself, but the residual cost of repeating power-iteration-based norm tracking for many layers, especially convolutional ones.

3. Operator-level FSN for adversarial training

In “Generalizable Adversarial Training via Spectral Normalization,” spectral normalization is tied explicitly to adversarial margin-based generalization bounds for ERM, FGM, PGM, and WRM. The central theoretical observation is that the complexity terms $$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$6, $$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$7, $$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$8, and $$\sigma(W) \triangleq \max_{\xi \neq 0} \frac{\lVert W\xi \rVert_2}{\lVert \xi \rVert_2},$$9 all depend strongly on products of spectral norms $\sigma(W)$0. This motivates applying spectral normalization to all linear layers so that $\sigma(W)$1, shrinking the adversarial complexity terms and tightening the PAC-Bayes-style bounds (Farnia et al., 2018).

The fast component in this paper is its treatment of convolutional layers with arbitrary stride and padding. Rather than flattening a kernel tensor and normalizing the kernel matrix, the method treats the convolution as a linear operator $\sigma(W)$2 acting on feature maps and runs power iteration directly through existing conv and conv_transpose primitives: $\sigma(W)$3

$\sigma(W)$4

The norm estimate is then

$\sigma(W)$5

and the layer is normalized as

$\sigma(W)$6

Only one power iteration per batch is used, and the singular-vector state is reused across SGD steps (Farnia et al., 2018).

This operator viewpoint is significant because it constrains the spectral norm of the full convolution operator, including stride and padding. The paper contrasts this with the kernel-based GAN formulation of Miyato et al., arguing that kernel normalization does not control the operator norm once stride and padding are taken into account. The authors report that, under the same nominal $\sigma(W)$7, Miyato-style kernel normalization can yield effective convolution-operator spectral norms around $\sigma(W)$8–$\sigma(W)$9 instead of $$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$0, with corresponding overfitting and worse test performance (Farnia et al., 2018).

Empirically, the method improves adversarial test accuracy while keeping overhead modest. On AlexNet for CIFAR-10, the reported adversarial test accuracy increases are $$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$1 for FGM $$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$2, $$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$3 for PGM $$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$4, and $$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$5 for WRM. Training-time overhead is typically $$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$6–$$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$7; relative to Miyato’s kernel-based SN, the cost ratio is approximately $$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$8–$$f(x,\theta) = W^{L+1} a_L\left(W^L a_{L-1}\big(\dots a_1(W^1 x)\dots\big)\right),$$9 on CIFAR10 experiments (Farnia et al., 2018). The theory assumes $a_l$0-Lipschitz, $a_l$1-smooth activations, whereas the empirical study reports similar behavior for ReLU and ELU.

4. Structure-aware FSN for depthwise separable convolutions

A different FSN line exploits the factorized structure of depthwise separable convolutions. In this setting, a standard multi-channel convolution is decomposed into a depthwise part, which performs spatial convolution independently in each channel, and a pointwise $a_l$2 part, which mixes channels without spatial coupling. The paper “Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization” uses this decomposition to obtain spectral norm estimates with negligible computational and memory overhead (Runkel et al., 2021).

For the depthwise component, each channel is a single-channel convolution. Under circulant boundary conditions, the spectral norm equals the maximum absolute value of the DFT of the padded filter,

$a_l$3

For zero-padded multi-channel depthwise convolution, the paper gives an upper bound,

$a_l$4

where $a_l$5. Thus the depthwise norm can be bounded by padding each per-channel filter, applying FFT, and taking the maximum absolute coefficient across channels and frequencies (Runkel et al., 2021).

For the pointwise component, the operator reduces exactly to a small connectivity matrix $a_l$6, and the spectral norm of the pointwise convolution is

$a_l$7

This norm is estimated with a warm-start power method on the small channel-mixing matrix rather than on a full feature-map operator. The stored singular vector has length $a_l$8 or $a_l$9, not $1$0, so memory overhead is negligible (Runkel et al., 2021).

The composite depthwise-separable block is then controlled by normalizing each linear piece individually and optionally applying either hard scaling or soft scaling. With $1$1-Lipschitz activations,

$1$2

The appeal of this FSN variant is that its cost does not scale with spatial resolution. The reported training times are $1$3 s/epoch without SN versus $1$4 s/epoch with this method for a ResNet-34-like architecture with depthwise separable convolutions on CIFAR-10, and $1$5 s/epoch without SN versus $1$6 s/epoch for MobileNetV2 on ImageNet, i.e. roughly $1$7 and $1$8, respectively (Runkel et al., 2021).

The quality of the depthwise bound depends on stride and feature-map size. For pretrained MobileNetV2 on ImageNet, the $1$9 unit-stride depthwise layers show average overestimation $$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$0 with standard deviation $$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$1, whereas the $$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$2 stride-$$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$3 depthwise layers show average overestimation about $$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$4 with standard deviation $$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$5 (Runkel et al., 2021). This indicates that the FFT-based bound is tight for unit-stride depthwise convolutions on realistic resolutions but materially looser for strided ones. The experiments omit batch normalization because BN changes the effective Lipschitz constant.

5. FFT- and layer-separation-based approximate FSN

The paper “Fast Approximate Spectral Normalization for Robust Deep Neural Networks” uses the name FSN explicitly for a training regularizer that combines kernel-level low-rank separation with FFT-based spectral norm computation (Pan et al., 2021). The objective is

$$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$6

where $$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$7 is estimated for every layer at every SGD step. The paper positions this as an alternative to power-iteration-based SN, which it regards as costly and coarse when only a few iterations are used (Pan et al., 2021).

The acceleration comes from two devices. First, a separable $$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$8D kernel $$\|f\|_{\mathrm{Lip}} \le \prod_{l=1}^{L+1} \sigma(W^l).$$9 is written as an outer product of a row vector and a column vector, so a $\lip(\nabla \ell \circ f_{\mathbf{w}})$0 convolution can be replaced by two $\lip(\nabla \ell \circ f_{\mathbf{w}})$1D convolutions of sizes $\lip(\nabla \ell \circ f_{\mathbf{w}})$2 and $\lip(\nabla \ell \circ f_{\mathbf{w}})$3. Second, for inseparable kernels, the method performs SVD on the small kernel matrix itself and uses a low-rank approximation, typically rank-$\lip(\nabla \ell \circ f_{\mathbf{w}})$4, before spectral-norm evaluation. The paper emphasizes that this kernel-level SVD is cheap because modern CNN kernels are usually $\lip(\nabla \ell \circ f_{\mathbf{w}})$5 or $\lip(\nabla \ell \circ f_{\mathbf{w}})$6 (Pan et al., 2021).

For the resulting convolution matrix, modeled as doubly block circulant, the paper states that the eigenvalues are exactly the entries of the $\lip(\nabla \ell \circ f_{\mathbf{w}})$7D Fourier transform of the kernel, and the singular values are their magnitudes. Hence

$\lip(\nabla \ell \circ f_{\mathbf{w}})$8

The proposed Algorithm 2 computes a $\lip(\nabla \ell \circ f_{\mathbf{w}})$9D FFT of the kernel, isolates the coefficient of maximum magnitude, performs an inverse FFT, and extracts the spectral norm; conceptually, the key quantity is the maximum magnitude of the Fourier coefficients (Pan et al., 2021).

The reported computational complexity for a convolution layer with $\|W_i\|_2$0 input channels, $\|W_i\|_2$1 output channels, and kernel size $\|W_i\|_2$2 is

$\|W_i\|_2$3

contrasted with SVD-based computation on the effective convolution matrix and with iterative power methods (Pan et al., 2021).

The empirical study covers GTSB with VGG16 and USD with LeNet-5. On GTSB, the tabled per-epoch times and best clean accuracies are: Normal $\|W_i\|_2$4 s/epoch and $\|W_i\|_2$5, SN $\|W_i\|_2$6 s/epoch and $\|W_i\|_2$7, FSN $\|W_i\|_2$8 s/epoch and $\|W_i\|_2$9. Under attacks, FSN yields $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$00 for FGSM with $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$01, $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$02 for CW-$$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$03 with $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$04, $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$05 for JSMA with $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$06, and $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$07 for DeepFool with $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$08. In the unbounded FGSM test, the unregularized model drops below $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$09 accuracy at $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$10, whereas FSN does so at $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$11 (Pan et al., 2021).

On USD, the reported per-epoch times and best clean accuracies are: Normal $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$12 s/epoch and $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$13, SN $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$14 s/epoch and $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$15, FSN $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$16 s/epoch and $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$17. Under attacks, FSN reaches $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$18 for FGSM, $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$19 for CW-$$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$20, $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$21 for JSMA, and $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$22 for DeepFool. The paper states that FSN “refined it to $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$23 which gives a $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$24 average improvement on accuracy” versus the unregularized model for bounded attacks (Pan et al., 2021).

The exactness claim in this paper is local to the FFT-based norm evaluation of the convolution matrix. The approximation enters through layer separation and low-rank kernel approximation. The authors argue qualitatively that the error is small when one singular value dominates, but do not provide a formal bound on $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$25 or on the resulting training dynamics (Pan et al., 2021).

6. Comparative perspective, terminology, and limitations

Across the cited works, FSN is not a single fixed algorithm so much as a common design objective: preserve spectral-norm-based Lipschitz control while reducing the cost of repeated norm computation or enforcement. This suggests that the phrase is best read as a class label for efficient spectral normalization strategies rather than as one canonical operator (Miyato et al., 2018, Farnia et al., 2018, Runkel et al., 2021, Pan et al., 2021).

Paper	Setting	Fast mechanism
“Spectral Normalization for Generative Adversarial Networks” (Miyato et al., 2018)	GAN discriminator stabilization	One-step warm-start power iteration on layer weights
“Generalizable Adversarial Training via Spectral Normalization” (Farnia et al., 2018)	Adversarial training for CNNs with arbitrary stride and padding	Operator-level power iteration via conv and conv_transpose
“Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization” (Runkel et al., 2021)	Depthwise separable architectures such as MobileNetV2	FFT bound for depthwise filters and small-matrix power method for pointwise convs
“Fast Approximate Spectral Normalization for Robust Deep Neural Networks” (Pan et al., 2021)	Robust CNN training with spectral-norm penalty	Layer separation and FFT-based norm computation

Several distinctions are technically important. First, kernel normalization and operator normalization are not equivalent once stride and padding are introduced. The adversarial-training paper treats this as a central defect of kernel-based convolution SN and reports effective operator norms around $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$26–$$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$27 instead of $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$28 under the same nominal constraint (Farnia et al., 2018). Second, “fast” can refer to different savings: fewer and better-conditioned power iterations, elimination of explicit convolution-matrix construction, exploiting separable channel structure, or replacing power iteration entirely with FFT-based formulas. Third, exactness claims are method-specific. In the depthwise-separable case, the pointwise norm is exact but the zero-padded depthwise norm is an upper bound; in the FFT/layer-separation method, the Fourier-domain norm is exact for the modeled convolution matrix, but the low-rank kernel approximation is not (Runkel et al., 2021, Pan et al., 2021).

The limitations also differ by formulation. The adversarial generalization theory of (Farnia et al., 2018) assumes $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$29-Lipschitz, $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$30-smooth activations and requires tuning $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$31; in some easy setups, $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$32 remains optimal. The depthwise-separable method requires known feature-map sizes and yields looser bounds for strided depthwise layers, while the choice of scaling constant $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$33 and learning rate must be tuned jointly (Runkel et al., 2021). The FFT/layer-separation method does not provide a formal worst-case approximation bound for the rank-reduced kernels, even though it reports substantial speed and robustness gains in practice (Pan et al., 2021).

Taken together, these works place FSN within a broader evolution of spectral normalization. The original GAN formulation established that per-layer control of $$\sigma(W) \;=\; \max_{\|\mathbf{h}\|_2 = 1} \|W \mathbf{h}\|_2,$$34 can stabilize training at modest cost (Miyato et al., 2018). Later work reinterpreted the same principle through adversarial generalization bounds, operator-level convolution norms, architecture-specific factorization, and FFT-based exact or approximate norm evaluation (Farnia et al., 2018, Runkel et al., 2021, Pan et al., 2021). The unifying theme is unchanged: spectral normalization is valuable because it controls the Lipschitz behavior of deep networks, and FSN comprises the algorithmic strategies that make that control practical at scale.