Relative Flatness in Machine Learning

• Relative flatness is a framework that measures loss-surface curvature relative to a selected feature layer, thus controlling for reparameterization sensitivity.
• It links local curvature metrics to improved generalization and adversarial robustness by accounting for parameter scaling and feature-space stability.
• The concept is applied as both an optimization objective and a diagnostic tool in deep learning, with parallels in geometric and algebraic contexts.

Searching arXiv for papers on "relative flatness" in machine learning and related contexts. Relative flatness is a family of notions in which “flatness” is measured relative to an ambient structure rather than as an absolute curvature quantity. In contemporary machine learning, the term usually denotes a reparameterization-aware layerwise measure of loss-surface curvature for models of the form $f(x)=g(w\phi(x))$, where curvature is assessed with respect to one selected layer and weighted by the scale of that layer’s parameters (Petzka et al., 2020). In this usage, the central motivation is that classical Hessian-based flatness is sensitive to function-preserving reparameterizations, whereas relative flatness is designed to track feature-space robustness and generalization in a way that is invariant to neuron-wise rescaling and orthogonal transformations (Han et al., 22 Sep 2025). More broadly, the phrase also appears in geometry, representation learning, and visual place recognition, where it again denotes a context-dependent flatness notion: flatness relative to a base scheme, to anchor-induced manifold segments, or to boundary data rather than an absolute global geometry (Avramov et al., 2010, Hisada et al., 1 Jun 2026).

1. Conceptual scope and terminology

In the machine-learning literature, relative flatness arose as a response to the reparameterization problem for classical sharpness measures. Traditional flatness measures based directly on the Hessian in parameter space can change under layer rescalings that leave the realized function unchanged, which weakens their explanatory power for generalization (Petzka et al., 2020). Relative flatness addresses this by evaluating curvature in relation to the scale and geometry of a chosen layer’s weights, rather than treating parameter-space curvature as an absolute object (Walter et al., 2024).

A standard decomposition writes the network as

$f(x)=g(w\phi(x)),$

with $\phi$ a feature extractor and $w$ the weights of a selected layer (Walter et al., 2024). The term “relative” refers to the fact that curvature is measured with respect to that layer and to the scale of its parameters, rather than over the whole network in a reparameterization-sensitive way (Walter et al., 2024). This suggests that the notion is fundamentally local in both architectural and geometric senses: it is attached to a representation layer and to perturbations meaningful for that layer.

Outside machine learning, the same modifier plays an analogous role. In algebraic geometry and commutative algebra, flatness is explicitly relative to a base $Y$ or to an ideal $\mathfrak a$, so that the question is not whether an object is flat in isolation, but whether it is flat over the specified base or along the $\mathfrak a$-adic neighborhood (Avramov et al., 2010, Yekutieli, 2016). In visual place recognition, FlatVPR uses “relative flatness” to mean flatness of a feature manifold with respect to the anchor segments used for map reconstruction, not absolute Euclidean flatness of the ambient embedding space (Hisada et al., 1 Jun 2026).

2. Formal definitions in deep learning

The foundational machine-learning formulation is due to the relative-flatness framework adopted by Petzka et al. and used in later work (Petzka et al., 2020). For a model $f(x,w)=g(w\phi(x))$, with $w\in\mathbb{R}^{d\times m}$, relative flatness is defined by

$$\kappa^\phi_{Tr}(w) := \sum_{s,s'=1}^d \langle w_s,w_{s'}\rangle \cdot \operatorname{Tr}\!\left(H_{s,s'}(w,\phi(S))\right),$$

where $f(x)=g(w\phi(x)),$0 and $f(x)=g(w\phi(x)),$1 are rows of the selected weight matrix and $f(x)=g(w\phi(x)),$2 is the corresponding Hessian block of the empirical loss (Petzka et al., 2020). The weight inner products make the quantity parameterization-aware rather than a raw Hessian statistic.

A closely related trace proxy appears in adversarial-robustness work. For a chosen layer $f(x)=g(w\phi(x)),$3, the relative sharpness proxy is

$f(x)=g(w\phi(x)),$4

with $f(x)=g(w\phi(x)),$5 the Hessian of the loss with respect to $f(x)=g(w\phi(x)),$6, averaged over the dataset $f(x)=g(w\phi(x)),$7 (Walter et al., 2024). Because small Hessian trace corresponds to a flatter landscape, this quantity is smaller for flatter minima and larger for sharper ones, so some papers use “relative sharpness” in practice to avoid sign confusion (Walter et al., 2024). A closely parallel formulation is used for the penultimate layer in adversarial analysis: $f(x)=g(w\phi(x)),$8 for $f(x)=g(w\phi(x)),$9 (Walter et al., 16 Oct 2025).

For cross-entropy and a single example, the Hessian admits a closed form: $\phi$0 and hence

$\phi$1

This makes the measure computationally cheap for the selected layer and explains its dependence on prediction confidence, feature scale, and weight scale (Walter et al., 2024). A related single-example penultimate-layer formula is

$\phi$2

with

$\phi$3

which makes explicit that the measure can collapse in highly confident regions, including confidently wrong ones (Walter et al., 16 Oct 2025).

Later work generalizes the same idea to specialized architectures. For convolutional neural networks with a final convolutional layer, global average pooling, and softmax cross-entropy, the exact trace of the Hessian with respect to the convolutional kernels is

$\phi$4

and this is then inserted into a filter-aware relative-flatness functional specialized to convolutional layers (Taleghani et al., 9 Mar 2026).

3. Reparameterization invariance and relation to generalization

A central claim of the relative-flatness literature is that the quantity is intended to resolve the reparameterization pathology of classical sharpness measures. The 2020 formulation emphasizes that raw Hessian flatness can vary under layer-wise and neuron-wise rescalings that preserve the represented function, whereas the relative measure is invariant under layer-wise reparameterizations and neuron-wise reparameterizations, with a mild normalization step for the feature layer (Petzka et al., 2020). A later grokking study states the same point more tersely, emphasizing invariance to neuron-wise rescaling and orthogonal transformations (Han et al., 22 Sep 2025).

The conceptual bridge to generalization is feature-space robustness. The identity

$\phi$5

shows that multiplicative perturbations of the features correspond to structured perturbations of the selected layer’s parameters (Petzka et al., 2020). This allows loss variation under feature perturbations to be expressed in terms of curvature with respect to $\phi$6. Under locally constant labels in feature space and at a local minimum, the expected loss increase under orthogonally averaged feature perturbations is approximated by relative flatness: $\phi$7 This provides the paper’s formal mechanism linking flatness and generalization (Petzka et al., 2020).

The same work situates flatness inside a broader decomposition of the generalization gap into a representativeness term and a feature-robustness term. If the training set is representative in feature space and labels are locally stable there, then flatness becomes the dominant local geometric contributor to generalization (Petzka et al., 2020). This suggests that relative flatness is not presented as a universal explanation, but as a conditional one.

Grokking offers a temporally separated test of that claim. In modular-arithmetic experiments with a 2-layer transformer, neural collapse and relative flatness both emerge near the onset of generalization, but only relative flatness aligns tightly with the actual transition: NCC starts decreasing already during memorization, while $\phi$8 stays high until the point where validation accuracy begins to rise, at which moment it drops sharply (Han et al., 22 Sep 2025). In CIFAR-10 experiments, regularizing the network away from flat solutions using

$\phi$9

delays generalization, whereas suppressing neural collapse by

$w$0

does not prevent it (Han et al., 22 Sep 2025). Within the studied regimes, the conclusion is that relative flatness is argued to be necessary, while neural collapse is neither necessary nor sufficient for generalization (Han et al., 22 Sep 2025).

4. Relative flatness as an optimization objective

Once relative flatness was formulated as a computable curvature quantity, it was used directly as a regularizer. FAM, or Relative Flatness Aware Minimization, optimizes

$w$1

where $w$2 is the relative flatness of a selected layer, usually a penultimate or bottleneck layer (Adilova et al., 2023). The method is explicitly motivated by the claim that relative flatness corresponds to robustness of the representation produced by that layer and therefore need not be computed for the entire network (Adilova et al., 2023).

The exact gradient involves both Hessian-trace terms and third derivatives. The paper gives a gradient decomposition of $w$3 into a term weighted by $w$4 and a term involving third-order partial derivatives of the loss (Adilova et al., 2023). Because the exact pairwise version can be expensive, the paper also uses the simplified approximation

$w$5

which retains the scale-aware character of the measure while being easier to evaluate (Adilova et al., 2023).

The empirical scope is broad. Reported results include CIFAR10, CIFAR100, SVHN, FashionMNIST, 3D skull-shape reconstruction, and BERT-base fine-tuning on RTE (Adilova et al., 2023). On CIFAR100, the reported values are Baseline $w$6, SAM $w$7, and FAM $w$8; on CIFAR10, Baseline $w$9, SAM $Y$0, and FAM $Y$1 (Adilova et al., 2023). On RTE, the reported validation accuracy improves from $Y$2 to $Y$3, while the standard deviation decreases from $Y$4 to $Y$5 (Adilova et al., 2023). The same paper also records caveats: FAM is not uniformly better than SAM, its usefulness depends on the locally constant-label assumption, and current implementations can still be $Y$6–$Y$7 slower than vanilla or SAM training (Adilova et al., 2023).

This suggests that relative flatness functions both as an explanatory quantity and as an operational regularizer, but only under assumptions about feature geometry and label stability that remain explicit in the theory.

5. Adversarial robustness and the “flat uncanny valley”

Relative flatness became especially prominent in work on adversarial examples. A widely observed empirical pattern is that, during an iterative white-box PGD attack, the flatness of the loss surface around the attacked example first becomes sharper until the label flips, but if the attack continues it enters a flat region in which the label remains flipped (Walter et al., 2024). The loss rises monotonically or saturates, while relative sharpness rises first and then falls. This is termed the “flat uncanny valley” (Walter et al., 2024).

The phenomenon was reported across CIFAR-10 and CIFAR-100 and across architectures including ResNet-18, WideResNet-28-4, VGG11, and DenseNet121, and it was still visible for adversarially trained models, although the valley was farther away and stronger attacks were needed to reach it (Walter et al., 2024). The same overall pattern was also observed for jailbreak trajectories in Vicuna-7B, Llama-2-7B-Chat, and Guanaco-7B-HF, though it was weaker and less often truly flat because the input space is discrete and the attacks are comparatively weaker (Walter et al., 2024).

A subsequent theoretical treatment sharpened the interpretation. Relative flatness is shown to imply only local adversarial robustness, not global robustness (Walter et al., 16 Oct 2025). Under an $Y$8-Lipschitz feature extractor with $Y$9, any perturbation $\mathfrak a$0 satisfying $\mathfrak a$1 induces a controlled feature perturbation

$\mathfrak a$2

for some orthogonal matrix $\mathfrak a$3 (Walter et al., 16 Oct 2025). A Taylor expansion then yields

$\mathfrak a$4

and after bounding the third-order remainder,

$\mathfrak a$5

Thus smaller relative sharpness implies slower local loss increase under small input perturbations, but the guarantee saturates and does not extend to the full attack trajectory (Walter et al., 16 Oct 2025).

The same paper stresses that robustness beyond a local neighborhood requires the loss to curve sharply away from the data manifold. If the loss stayed flat everywhere along an adversarial path, then the attack could move through large regions without penalty (Walter et al., 16 Oct 2025). This is consistent with the empirical finding that adversarial examples often lie in large, flat regions where the model is confidently wrong (Walter et al., 16 Oct 2025). A common misconception is therefore that flat minima imply global adversarial robustness; the cited results explicitly reject that implication (Walter et al., 2024, Walter et al., 16 Oct 2025).

6. Specialized and adjacent formulations

Relative flatness has been specialized beyond dense classifiers. In CNNs with a $\mathfrak a$6 convolution plus global average pooling classifier head, the exact symbolic Hessian trace makes it possible to compute a deterministic, architecture-aware flatness score rather than relying on stochastic trace estimation (Taleghani et al., 9 Mar 2026). In the reported correlation study on 84 modified ResNet-18 models trained on CIFAR-10 for 30 epochs, the proposed score yields $\mathfrak a$7, Pearson $\mathfrak a$8, and Spearman $\mathfrak a$9, with the authors emphasizing the stronger monotonic than linear trend (Taleghani et al., 9 Mar 2026). The same study reports that SGD with momentum tends to find flatter minima and smaller generalization gaps than AdamW, and that flatness stabilization can be used as an early-stopping signal (Taleghani et al., 9 Mar 2026).

In representation learning for visual place recognition, FlatVPR uses “flatness” in a different but structurally analogous sense. The goal is to rectify a foundation model’s latent manifold so that any descriptor between two adjacent anchors $\mathfrak a$0 and $\mathfrak a$1 can be reconstructed by

$\mathfrak a$2

The associated Pullback Flatness Loss is

$\mathfrak a$3

which penalizes deviation from the line segment connecting adjacent anchors (Hisada et al., 1 Jun 2026). The paper explicitly interprets this as relative flatness of the feature manifold with respect to anchor segments used for map reconstruction, not as a full Riemannian curvature estimate (Hisada et al., 1 Jun 2026).

This usage is technically distinct from Hessian-based relative flatness, but the structural analogy is clear: in both cases, flatness is defined relative to a representation geometry and to perturbations or interpolations considered operationally meaningful. A plausible implication is that “relative flatness” functions as a general design pattern for geometry-aware stability measures rather than a single universal formula.

7. Other mathematical meanings of flatness relative to a base

In algebraic geometry, the phrase “relative flatness” takes its classical meaning: flatness of a coherent sheaf or morphism over a specified base. For an essentially finite type morphism $\mathfrak a$4 with $\mathfrak a$5 smooth over a field and a coherent sheaf $\mathfrak a$6 on $\mathfrak a$7, flatness over $\mathfrak a$8 is characterized by the existence of $\mathfrak a$9 such that

$f(x,w)=g(w\phi(x))$0

Equivalently, in commutative algebra, if $f(x,w)=g(w\phi(x))$1 is an essentially smooth $f(x,w)=g(w\phi(x))$2-algebra, $f(x,w)=g(w\phi(x))$3 is essentially of finite type over $f(x,w)=g(w\phi(x))$4, and $f(x,w)=g(w\phi(x))$5 is a finite $f(x,w)=g(w\phi(x))$6-module, then

$f(x,w)=g(w\phi(x))$7

(Avramov et al., 2010). Here the relation is explicitly to the base ring or base scheme.

A different relative notion appears in $f(x,w)=g(w\phi(x))$8-adic flatness. An $f(x,w)=g(w\phi(x))$9-module $w\in\mathbb{R}^{d\times m}$0 is $w\in\mathbb{R}^{d\times m}$1-adically flat if

$w\in\mathbb{R}^{d\times m}$2

This is weaker than ordinary flatness because it tests only against $w\in\mathbb{R}^{d\times m}$3-torsion modules rather than all modules (Yekutieli, 2016). Under weak proregularity, completion preserves $w\in\mathbb{R}^{d\times m}$4-adic flatness, and in the noetherian complete setting the distinction collapses: $w\in\mathbb{R}^{d\times m}$5 (Yekutieli, 2016).

These mathematical uses are conceptually separate from neural-network loss geometry. Their commonality lies only in the modifier “relative”: flatness is always assessed with respect to a specified ambient object, such as a base scheme, an ideal, a feature layer, or anchor-induced representation segments.

8. Synthesis and open interpretive issues

Across domains, relative flatness is not a single invariant but a methodological principle: flatness should be measured relative to the geometry that matters for the task. In neural networks, that geometry is the penultimate or chosen feature layer and the perturbations that preserve the network’s functional semantics (Petzka et al., 2020). In adversarial analysis, the same measure is informative about local robustness basins but does not certify global correctness or robustness (Walter et al., 16 Oct 2025). In grokking and standard classification, it is presented as more fundamental than neural collapse for predicting the onset of generalization (Han et al., 22 Sep 2025). In CNN-specific work, exact formulas make the notion architecturally faithful for GAP-based convolutional heads (Taleghani et al., 9 Mar 2026). In VPR, the notion is reformulated as interpolation-consistency along anchor segments (Hisada et al., 1 Jun 2026).

Several controversies or misconceptions recur. One is the belief that any flatness measure can explain generalization; the relative-flatness literature insists that the connection holds only under assumptions such as representativeness and local label constancy in feature space (Petzka et al., 2020). Another is the belief that flatness implies adversarial robustness simpliciter; the uncanny-valley results show that adversarial examples can occupy broad, flat plateaus on the wrong side of the decision boundary (Walter et al., 2024). A third is the tendency to treat neural collapse as the geometric cause of generalization; the grokking results argue instead that neural collapse is best understood as one route toward relative flatness in some regimes (Han et al., 22 Sep 2025).

Taken together, these works support a restrained interpretation. Relative flatness is a useful geometric lens because it links curvature to perturbations that are operationally meaningful and because it avoids the worst reparameterization pathologies of raw Hessian measures. At the same time, the cited results consistently show that flatness alone is insufficient: for generalization it must be coupled with representativeness and local label stability, and for robustness it must be coupled with smoothness or Lipschitz control of the feature extractor and with the behavior of the function away from the data manifold (Petzka et al., 2020, Walter et al., 2024).