Step Normalization: Techniques and Applications

• Step normalization is a family of techniques that attach normalization to specific pipeline stages, update rules, or proof methods, adapting to diverse domain requirements.
• It improves optimization and continuous-time modeling by selectively normalizing update steps or derivative scales, yielding measurable gains in convergence and performance.
• The approach also refines preprocessing in imaging, streamlines database design, and supports logical proofs in programming languages to ensure consistency and effective signal scaling.

Searching arXiv for recent and relevant papers on “step normalization” across domains. Using the arXiv search tool now. Step normalization is a domain-dependent term rather than a single standardized technique. Across the cited literature, it can denote a post-orthogonalization update normalization in matrix-valued optimization, a selective normalization of stochastic-gradient steps, a time-scale normalization induced by scaling the state derivative in continuous-time neural models, a decomposition of graph normalization into centering and scaling steps, a front-end preprocessing stage in imaging and biosignal pipelines, a graded measure of progression between database normal forms, or a step-indexed proof method for establishing normalization of a logical fragment in the presence of general recursion (Zhang et al., 25 Feb 2026, Curtis et al., 2017, Weigand et al., 2024, Scholkemper et al., 2024, Simkó et al., 2019, Truong et al., 15 Jun 2025, Qureshi et al., 2012, Casinghino et al., 2012).

1. Terminological scope and major meanings

A useful way to read the literature is to treat “step normalization” as a family of constructions in which normalization is attached to a specific stage of a pipeline, update rule, or proof. This suggests that the term is best understood operationally: what is normalized, when, and for what purpose all vary by field.

Domain	Meaning of “step normalization”	Representative paper
Optimizer design	Normalize an update step or only selected stochastic steps	(Zhang et al., 25 Feb 2026, Curtis et al., 2017)
Continuous-time models	Normalize derivative scale, and thereby effective time scale	(Weigand et al., 2024)
GNNs	Separate centering and scaling steps inside normalization	(Scholkemper et al., 2024)
Medical/EEG pipelines	Normalization as an explicit preprocessing stage	(Simkó et al., 2019, Truong et al., 15 Jun 2025, Delisle et al., 2019)
Database design	Stepwise closeness to the next normal form	(Qureshi et al., 2012)
Programming languages	Step-indexed logical relations for normalization proofs	(Casinghino et al., 2012)

A terminological clarification appears in preprocessing work that proposes “Integer Scaling Normalization”: it explicitly does not present a step normalization method in the usual sense of a piecewise, threshold-based, staircase, or interval-wise transformation. Its method is digit-based and arithmetic instead (Patro et al., 2015).

2. Update-step normalization in optimization

In large-scale optimization, step normalization appears most explicitly in two distinct forms. The first is Muon+, which modifies the Muon optimizer by inserting one additional normalization step after orthogonalization of the momentum matrix. For a weight matrix $W_t \in \mathbb{R}^{m \times n}$, Muon uses

$$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$

Muon+ replaces the update by

$$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$

where $\mathrm{Norm}_{(d)}$ is row-wise, column-wise, or two-pass row/column $\ell_2$ normalization of the orthogonalized update matrix, not of the raw gradient, not of the momentum before orthogonalization, and not of the parameters (Zhang et al., 25 Feb 2026).

The paper’s empirical claim is that this one extra normalization materially improves Muon. On GPT-style pretraining, validation perplexity improves from 29.66 to 27.64 for GPT-Small, from 21.70 to 19.98 for GPT-Base, and from 17.82 to 16.91 for GPT-Large. On LLaMA-style models, the gain is also consistent, including 14.02 to 13.41 at 350M and 10.68 to 10.31 at 1B. The same study reports that any normalization helps relative to the Muon baseline, that row-wise normalization outperforms column-wise normalization in those experiments, and that two-pass normalization often performs best (Zhang et al., 25 Feb 2026).

A second meaning appears in TRish, a stochastic trust-region-ish method based on “careful step normalization.” TRish does not normalize every stochastic gradient. Instead, with parameters $\alpha_k>0$ and $\gamma_{1,k}>\gamma_{2,k}>0$, it uses

$x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$

The normalized trust-region step is therefore used only in a middle regime. The motivation is that always normalizing can be harmful even for unbiased stochastic gradients; the paper gives a scalar counterexample in which $g_k$ equals $6$ with probability $$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$0 and $$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$1 with probability $$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$2, so $$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$3, but the normalized direction is twice as likely to be an ascent direction as a descent direction (Curtis et al., 2017).

The convergence theory for TRish is designed to remain SG-like under conditions on the gradient estimator and on the ratio $$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$4. Empirically, the method often outperforms standard SG on logistic regression and CNN training, while its central conceptual lesson is that normalization is beneficial when used carefully rather than indiscriminately (Curtis et al., 2017).

3. Time-scale normalization in continuous-time neural models

In continuous-time state-space identification, “step normalization” is most naturally interpreted as normalization of the effective integration scale. The central proposal is state derivative normalization (SDN), which inserts a constant $$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$5 into the model

$$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$6

The paper argues that hidden state normalization, hidden state derivative normalization, and time-interval or effective integration-step normalization are inherently coupled, and that a single normalization constant at the derivative level addresses all three (Weigand et al., 2024).

The coupling is explicit. The same equation can be read as state scaling by defining $$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$7, or as time scaling by defining $$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$8. In integral form,

$$M_t = \mu M_{t-1} + (1 - \mu) G_t,\qquad O_t = \mathrm{Ortho}(M_t),\qquad W_t = W_{t-1} - \eta \cdot \sqrt{m/n} \cdot O_t.$$9

is equivalent to integration over $$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$0. In that sense, derivative normalization is also normalization of the effective solver step or effective integration length (Weigand et al., 2024).

The paper proposes three ways to choose $$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$1: treat it as a trainable parameter, select it by cross-validation, or estimate it heuristically from an approximate model. Its existence-based heuristic sets

$$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$2

and the practical BLA-based estimate is

$$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$3

On the cascaded tanks benchmark, the unnormalized setting $$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$4 corresponds to $$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$5 and is clearly suboptimal, with median RMSE $$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$6 V and mean RMSE $$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$7 V. The reported best SDN result is $$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$8 V for cross-validated normalization, while the BLA heuristic gives $$O_t = \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(M_t)\right),$$9, which lies in the empirically good region (Weigand et al., 2024).

4. Normalization steps inside neural architectures

In graph neural networks, the literature analyzes normalization itself as a two-step operator. For a feature column $\mathrm{Norm}_{(d)}$0, BatchNorm is written as

$\mathrm{Norm}_{(d)}$1

so the layer is explicitly decomposed into centering

$\mathrm{Norm}_{(d)}$2

followed by scaling to unit norm. The paper’s main theoretical result is that the anti-oversmoothing effect comes from the scaling step rather than the centering step. In linearized GNNs, BatchNorm prevents complete collapse of node representations to a one-dimensional subspace and yields convergence to the top-$\mathrm{Norm}_{(d)}$3 eigenspace of the effective propagation operator, whereas the centering step changes the operator from $\mathrm{Norm}_{(d)}$4 to $\mathrm{Norm}_{(d)}$5 and can distort structural graph signals (Scholkemper et al., 2024).

This distinction is formalized in several propositions. If $\mathrm{Norm}_{(d)}$6, then

$\mathrm{Norm}_{(d)}$7

with probability $\mathrm{Norm}_{(d)}$8, so complete collapse is prevented. Under stronger conditions, the normalized dynamics converge exponentially to the column space of the top-$\mathrm{Norm}_{(d)}$9 eigenvectors of $\ell_2$0. The paper also states that if centering were removed and only scaling retained, the same proof would go through with $\ell_2$1 replacing $\ell_2$2, which is the basis for the claim that scaling is the essential anti-oversmoothing ingredient (Scholkemper et al., 2024).

This analysis motivates GraphNormv2, in which the centering step is replaced by a learned graph-aware projection: $\ell_2$3 The intended effect is to preserve the useful scaling behavior while avoiding undesirable distortion of the original graph signal (Scholkemper et al., 2024).

5. Normalization as an explicit preprocessing step

A different usage treats normalization as a front-end pipeline stage rather than a hidden-layer operation. In MRI, the problem is low-frequency multiplicative bias or gain field correction, modeled as

$\ell_2$4

The network GetNet is trained without medical images, using 108,000 ImageNet crops converted to grayscale and normalized to $\ell_2$5, and predicts a $\ell_2$6 gain field from a $\ell_2$7 input. The model is intended as a learned replacement for N4ITK, with correction achieved by dividing the observed image by the predicted field. On simulated BrainWeb-derived $\ell_2$8-weighted brain MRI, GetNet improves relative MAE over N4ITK for the entire image and for CSF, gray matter, and white matter, and reduces total correction time from $\ell_2$9 s to $\alpha_k>0$0 s, a speedup factor of almost $\alpha_k>0$1 (Simkó et al., 2019).

In EEG deep learning, the cited study concludes that there is no universally best normalization step. It varies normalization by granularity—recording level versus window level—and by scope—cross-channel (“All”) versus within-channel (“Channel”)—using robust scaling with median and interquartile range rather than mean and standard deviation. For supervised age and gender prediction on HBN-EEG, the strongest results come from window-level within-channel normalization, with Channel-Channel giving $\alpha_k>0$2 MAE for age and $\alpha_k>0$3 balanced accuracy for gender. For CPC, especially with same-recording distractors, minimal normalization or window-level cross-channel normalization is more effective, with None-All reaching $\alpha_k>0$4 loss. The combination All-Channel is a consistent failure mode, producing NaN collapse in multiple settings (Truong et al., 15 Jun 2025).

In multi-domain brain MRI segmentation, normalization is elevated to an explicit trainable first step. A generator $\alpha_k>0$5 maps a raw image patch $\alpha_k>0$6 to a normalized patch $\alpha_k>0$7, which is then segmented by $\alpha_k>0$8, while a discriminator $\alpha_k>0$9 distinguishes real domains from generated images. The joint objective is

$\gamma_{1,k}>\gamma_{2,k}>0$0

with a weighted Dice segmentation loss plus a $\gamma_{1,k}>\gamma_{2,k}>0$1-class domain-classification loss. On iSEG and MRBrainS, Gaussian standardization yields Dice scores $\gamma_{1,k}>\gamma_{2,k}>0$2 for CSF/GM/WM, learned normalization without realism constraint yields $\gamma_{1,k}>\gamma_{2,k}>0$3, and the adversarial normalizer yields $\gamma_{1,k}>\gamma_{2,k}>0$4, with the paper reporting Dice improvements of up to $\gamma_{1,k}>\gamma_{2,k}>0$5 over the baseline (Delisle et al., 2019).

6. Stepwise normalization in relational database design

In database theory, step normalization refers to a graded assessment of how far a relation has progressed toward the next normal form rather than to a transformation of numeric values. The method proceeds in two steps. First, determine the current normal form $\gamma_{1,k}>\gamma_{2,k}>0$6 up to BCNF by ordinary analysis of composite attributes, partial dependencies, transitive dependencies, and determinants. Second, compute a fuzzy membership value in $\gamma_{1,k}>\gamma_{2,k}>0$7 that measures closeness to the next normal form (Qureshi et al., 2012).

The resulting normalization completeness is

$\gamma_{1,k}>\gamma_{2,k}>0$8

operationalized as

$\gamma_{1,k}>\gamma_{2,k}>0$9

Here, completeness attributes come from non-preventing functional dependencies, preventing attributes come from dependencies that block movement to the next normal form, and total attributes is the size of the relation (Qureshi et al., 2012).

The paper’s case study on StaffPropertyInspection illustrates the meaning of intermediate values. The initial schema is in 1NF with $x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$0, the first transformation yields a 2NF relation with $x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$1, and the second yields a 3NF relation with $x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$2. The intended interpretation is that a schema can be definitely in one normal form while being only partially complete toward the next (Qureshi et al., 2012).

7. Step-indexed normalization in programming-language metatheory

In type theory and programming-language semantics, step-indexed normalization is a proof technique rather than a data or parameter transformation. The language studied has two fragments, a logical fragment $x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$3 and a programmatic fragment $x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$4, with general recursion and recursive types only in $x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$5. It also includes an internalized logicality type $x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$6, together with explicit boxing and unboxing, so that the two fragments can exchange information (Casinghino et al., 2012).

The normalization goal is restricted to the logical fragment: if

$x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$7

then there exists a value $x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$8 such that

$x_{k+1}\gets x_k- \begin{cases} \gamma_{1,k}\alpha_k g_k &\text{if } \|g_k\|\in \left[0,\frac{1}{\gamma_{1,k}}\right),\[4pt] \alpha_k \dfrac{g_k}{\|g_k\|} &\text{if } \|g_k\|\in \left[\frac{1}{\gamma_{1,k}},\frac{1}{\gamma_{2,k}}\right],\[8pt] \gamma_{2,k}\alpha_k g_k &\text{if } \|g_k\|\in \left(\frac{1}{\gamma_{2,k}},\infty\right). \end{cases}$9

A standard Girard–Tait reducibility proof is not sufficient because logical terms may mention programmatic values, while recursive functions and recursive types in $g_k$0 prevent a straightforward total semantic interpretation. The solution is to combine reducibility for $g_k$1 with step-indexed logical relations for $g_k$2 (Casinghino et al., 2012).

The value and computation interpretations are indexed by $g_k$3. For programmatic computations,

$g_k$4

requires only partial correctness up to $g_k$5 steps: if a term reaches a value in $g_k$6 steps, that value must be semantically valid at index $g_k$7. For logical computations,

$g_k$8

requires actual normalization to a value in the logical value interpretation. Recursive functions and recursive types are handled by strictly decreasing the index. The central soundness theorem states that if

$g_k$9

and $6$0, then

$6$1

from which normalization of closed logical terms follows (Casinghino et al., 2012).

Taken together, these literatures show that “step normalization” is best treated as a structurally situated notion. It may normalize an optimizer update, an effective integration scale, a centering or scaling sub-operation, a preprocessing stage, a database-design transition, or a semantic proof obligation. The common pattern is not a shared formula but a shared emphasis on normalization at a specific step of computation, inference, or formal reasoning.