Dynamic Feature Normalization (DyFN)

• Dynamic Feature Normalization (DyFN) is a family of methods that adaptively adjusts feature scaling using statistics computed from the data.
• It addresses issues such as small-batch instability and distribution shifts by recalculating normalization parameters per sample, batch, or time step.
• Applications span robust image classification, domain adaptation, continual learning, and dynamic graph embedding, yielding significant performance gains.

Dynamic Feature Normalization (DyFN) denotes a family of normalization procedures in which the quantities used to standardize or re-scale features are allowed to vary with the observed data, the sample, the batch, the time step, or the deployment stream. In one prominent formulation, DyFN generalizes Batch Normalization (BN) by making the affine parameters $\gamma$ and $\beta$ functions of input statistics rather than static learned constants (Liu et al., 2021). Across adjacent literatures, the same label or closely related formulations also refer to inference-time re-estimation of normalization statistics under distribution shift, unit-sphere or radially scaled feature normalization for stability and geometry control, EMA-based adaptive scaling in continual learning, and recurrent modulation of latent feature statistics in streaming video geometry (Kaku et al., 2020, Yang et al., 2021, Marasco et al., 18 Mar 2026, Lyu et al., 25 May 2026). This breadth suggests that DyFN is best understood as an umbrella concept centered on dynamic, data-dependent normalization rather than as a single standardized operator.

1. Conceptual scope and defining mechanisms

A recurring point of departure is standard BN. For activations $X\in\mathbb{R}^{N\times C\times H\times W}$, BN normalizes each channel by subtracting the batch mean $\mu_c$, dividing by the batch standard deviation $\sigma_c$, and then applying learned, static $\gamma_c$ and $\beta_c$ shared across all batches and samples. The limitations emphasized in the DyFN literature are twofold: small-batch instability, because estimates of $\mu_c$ and $\sigma_c$ become noisy when $N$ is small, and lack of per-sample adaptivity, because the same $\beta$0 are used for every sample and batch after training (Liu et al., 2021).

A second lineage starts from a different limitation: the mismatch between training-time normalization statistics and test-time feature distributions. In that setting, the dynamic component is not necessarily $\beta$1; instead, the normalization statistics themselves are recomputed on test data, either per batch or per instance, while the learned affine parameters remain fixed (Kaku et al., 2020). A third lineage is geometric rather than batch-statistical: normalization is used to constrain features to a hypersphere or to transform their radial geometry so that oversmoothing, collapse, or poor OOD separation are mitigated (Yang et al., 2021, Bao, 2023, Janiak et al., 17 Oct 2025).

Setting	Dynamic quantity	Representative formulation
CNN normalization layers	$\beta$2 generated from $\beta$3 or $\beta$4	DN-C and DN-B (Liu et al., 2021)
Test-time robustness	$\beta$5 recomputed during inference	Adaptive feature normalization and DUA (Kaku et al., 2020, Mirza et al., 2021)
Dynamic graphs and SSL	Feature norm constrained or normalized before similarity	FeatureNorm and cosine-loss analysis (Yang et al., 2021, Bao, 2023)
OOD detection	Radial geometry controlled by $\beta$6	Mahalanobis normalization study (Janiak et al., 17 Oct 2025)
Continual and streaming settings	EMA-updated scales or recurrently predicted modulation	CLeAN and streaming video DyFN (Marasco et al., 18 Mar 2026, Lyu et al., 25 May 2026)

The common principle is that normalization is no longer treated as a fixed preprocessing or fixed affine post-standardization step. Instead, it becomes an adaptive mechanism coupled to the instantaneous statistics or geometry of features.

2. Dynamic affine normalization as a generalization of Batch Normalization

The most direct formulation of DyFN appears in "Dynamic Normalization" (Liu et al., 2021). Let $\beta$7 be a mini-batch of activations. For each channel $\beta$8,

$\beta$9

$X\in\mathbb{R}^{N\times C\times H\times W}$0

with

$X\in\mathbb{R}^{N\times C\times H\times W}$1

Two variants are defined. In DN-C, a batch-shared, channel-wise $X\in\mathbb{R}^{N\times C\times H\times W}$2 and $X\in\mathbb{R}^{N\times C\times H\times W}$3 are generated from the batch statistics: $X\in\mathbb{R}^{N\times C\times H\times W}$4 and the output is

$X\in\mathbb{R}^{N\times C\times H\times W}$5

In DN-B, for each sample $X\in\mathbb{R}^{N\times C\times H\times W}$6 and channel $X\in\mathbb{R}^{N\times C\times H\times W}$7,

$X\in\mathbb{R}^{N\times C\times H\times W}$8

and the per-sample affine parameters are

$X\in\mathbb{R}^{N\times C\times H\times W}$9

yielding

$\mu_c$0

The functions $\mu_c$1 and $\mu_c$2 are implemented by lightweight “bottleneck” MLPs or their $\mu_c$3-convolution analogs. The input dimension is $\mu_c$4 from concatenated $\mu_c$5 and $\mu_c$6, the hidden dimension is $\mu_c$7, the expansion returns to $\mu_c$8 outputs, and the module uses ReLU after the first FC with no activation on the final layer; the second FC can be implemented as grouped $\mu_c$9 convolution with group width $\sigma_c$0 channels per group (Liu et al., 2021).

Empirically, the two variants behave very differently. DN-C “can't train normally,” whereas DN-B “has very good robustness.” On MobileNetV2 for ImageNet-100, baseline BN top-1 test accuracy is 82.5%, while DN-B with $\sigma_c$1 reaches 84.1% with +0.6% Mult-Adds; the summary also reports that the peak improvement is more than 2% on some settings. On SSDLite for MS-COCO, baseline mAP is approximately 15.3%, and SSDLite with a DN-B backbone reaches approximately 19.2%, a +3.9% mAP gain under the same training schedule. Robustness claims are similarly explicit: BN with lr=0.5 collapses with –8% test accuracy, whereas DN-B at lr=0.5 drops only approximately 0.8% and still outperforms BN at lr=0.2; BN at batch size = 8 often fails to train, while DN-B trains stably and recovers more than 92% test accuracy on ImageNet-100 (Liu et al., 2021).

These results position DyFN, in this narrow sense, as a data-conditioned extension of BN in which the affine transformation becomes sample-aware and channel-aware. The failure of DN-C and the success of DN-B indicate that simply making the affine parameters batch-dependent is not sufficient; the per-sample pathway appears to be the practically effective regime in the reported experiments.

3. Inference-time adaptive normalization under distribution shift

A distinct DyFN line treats normalization as a deployment-time adaptation mechanism rather than a training-time replacement for BN. In "Be Like Water: Robustness to Extraneous Variables Via Adaptive Feature Normalization" (Kaku et al., 2020), the motivating problem is that extraneous variables can heavily affect the distribution of the available data, and models trained with BN learn features highly dependent on these variables. At inference, BN uses fixed running estimates $\sigma_c$2 and $\sigma_c$3, producing a mismatch when extraneous variables vary. The proposed remedy is to estimate feature statistics adaptively during inference.

For a layer $\sigma_c$4 and channel $\sigma_c$5, the dynamic statistics are

$\sigma_c$6

computed either over a test mini-batch $\sigma_c$7 whose examples share the same extraneous label $\sigma_c$8, or over a single test example in the instance-based case, with analogous formulas for $\sigma_c$9. The normalized feature is then

$\gamma_c$0

Crucially, $\gamma_c$1 are the same learned affine parameters used in batch-norm; they are not changed at test time (Kaku et al., 2020).

The reported gains are tied to datasets with strong extraneous-variable effects. On JIGSAWS, non-adaptive BN is approximately 67%, while instance-adaptive normalization reaches approximately 72.7%. On StrokeRehab, the comparison is approximately 48.3% versus approximately 63.9%. On CIFAR-10C, it is approximately 72.6% versus approximately 81.6%. At the same time, the paper states that on homogeneous data such as CIFAR-10 and Fashion-MNIST, traditional batch-norm wins, confirming that adaptive normalization helps only when extraneous shifts occur (Kaku et al., 2020). This is an important corrective to any universal interpretation of DyFN: its benefits are distribution-conditional.

"The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization" (Mirza et al., 2021) develops a related idea for streaming target-domain adaptation. Dynamic Unsupervised Adaptation (DUA) updates only BN running mean and variance at test time, with no gradient computations and no weight updates other than the BN statistics. If $\gamma_c$2 are the batch statistics of the $\gamma_c$3-th small test batch, DUA applies

$\gamma_c$4

$\gamma_c$5

with

$\gamma_c$6

and typical defaults $\gamma_c$7, $\gamma_c$8 (Mirza et al., 2021).

The paper states that DUA uses 0.2–0.9% of the unlabeled target samples in sequential order. On CIFAR-10C (15 corruptions, Level 5), source-only error is 49.2%, TTT is 31.4%, NORM is 32.9%, and DUA is 26.8%. On ImageNet-C (Level 5, ResNet-18), source is 86.2%, NORM is 77.3%, TTT is 83.6%, and DUA is 78.2%. The same document reports substantial improvements for digit recognition, Office-31, VIS-DA, and object detection under weather and cross-dataset domain shifts (Mirza et al., 2021). In this branch of the literature, DyFN is not about learning new affine parameters; it is about reopening and dynamically updating normalization statistics during deployment.

4. Geometric interpretations: hyperspherical, cosine-normalized, and radially scaled DyFN

Outside CNN BN layers, DyFN appears as a geometric constraint on representations. In "FeatureNorm: L2 Feature Normalization for Dynamic Graph Embedding" (Yang et al., 2021), the motivation is oversmoothing in dynamic GCNs. After a GCN block at time $\gamma_c$9, raw node embeddings $\beta_c$0 are first centered,

$\beta_c$1

and then rescaled to the unit sphere,

$\beta_c$2

The paper states that centering ensures the total pairwise distance

$\beta_c$3

remains exactly constant, equal to $\beta_c$4, and that the normalization prevents further shrink. The reported empirical effect is that attaching FeatureNorm yields +3–10% AUC/AP for DynGCN, +2–8% for GRUGCN, and approximately 3–7% for EvolveGCN variants, while the negative-to-positive smoothness ratio $\beta_c$5 drops by an order of magnitude (Yang et al., 2021).

In non-contrastive self-supervised learning, "Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamics" (Bao, 2023) analyzes cosine-normalized similarity instead of an $\beta_c$6 loss. The cosine loss is

$\beta_c$7

Under the stated high-dimensional and Gaussian assumptions, the induced eigenmode dynamics become a sixth-order polynomial ODE rather than the third-order ODE of the $\beta_c$8-loss case. The paper’s theorem states that, under the same model and for any $\beta_c$9, the induced sixth-order ODE always transitions into a regime admitting a non-zero stable equilibrium, so feature normalization “dynamically” prevents collapse (Bao, 2023). Here DyFN is less an architectural module than a dynamical consequence of normalizing features before measuring similarity.

"Dissecting Mahalanobis: How Feature Geometry and Normalization Shape OOD Detection" (Janiak et al., 17 Oct 2025) extends the geometric perspective further through radially scaled $\mu_c$0 normalization: $\mu_c$1 Special cases are explicit: $\mu_c$2 gives no normalization, $\mu_c$3 gives unit-sphere projection, $\mu_c$4 gives stronger contraction of high-norm points, and $\mu_c$5 gives radial expansion. Inserted before Mahalanobis distance computation, this normalization changes OOD performance materially. Averaged over all models and datasets, MD* achieves 0.358 FPR@95, \hat{\mathrm{MD}} achieves 0.375, unnormalized MD gives 0.415, and MD++ with $\mu_c$6 gives 0.392; for the Relative Mahalanobis variant, RMD* is 0.353, \hat{\mathrm{RMD}} is 0.361, RMD is 0.375, and RMD++ is 0.368 (Janiak et al., 17 Oct 2025). This formulation is not a BN replacement, but it belongs to the DyFN family insofar as normalization dynamically reshapes representation geometry through a tunable radial exponent.

Taken together, these works show that DyFN can target not only first- and second-order channel statistics but also the global geometry of embedding space. The shared technical aim is to prevent degeneration of representations—oversmoothing, collapse, or poor class-separation geometry—by enforcing or modulating norm structure.

5. Sequential and streaming formulations

Sequential settings make the temporal dimension of DyFN explicit. In "CLeAN: Continual Learning Adaptive Normalization in Dynamic Environments" (Marasco et al., 18 Mar 2026), the problem is continual learning for tabular data. For streaming experiences $\mu_c$7, CLeAN maintains trainable estimators $\mu_c$8 of global feature-wise max and min, updated by EMA: $\mu_c$9 where $\sigma_c$0 is typically $\sigma_c$1–$\sigma_c$2. Each sample is normalized and then passed through a learned diagonal affine scaling layer: $\sigma_c$3 The reported datasets are UNSW-NB15 (2.54 M samples, 53 numeric features after encoding) and CICIDS-2017 (2.83 M samples, 68 numeric features), streamed in chunks of 500 K, yielding 6 experiences for UNSW-NB15 and 5 for CICIDS-2017. On UNSW-NB15 under reservoir replay, final average accuracy is 98.6% for global normalization, 97.8% for CLeAN, 94.2% for local min-max, and 91.5% for continual normalization; final average forgetting is 0.05, 0.08, 0.15, and 0.22, respectively (Marasco et al., 18 Mar 2026). In this setting, DyFN is an online feature-scale estimator coupled to a learned rescaling layer.

"Stabilizing Streaming Video Geometry via Dynamic Feature Normalization" (Lyu et al., 25 May 2026) gives a recurrent vision-specific DyFN. A frozen monocular geometry model produces a latent feature map

$\sigma_c$4

which is normalized per channel,

$\sigma_c$5

A ConvGRU updates the hidden state,

$\sigma_c$6

and two $\sigma_c$7 convolutions predict dynamic scale and shift: $\sigma_c$8 The consistent feature fed to the frozen decoder is

$\sigma_c$9

The training strategy is parameter-efficient: the encoder and decoder are frozen, while DyFN comprises approximately 2% of total parameters. The total loss is

$N$0

with sequence length 12 frames, random stride in $N$1, approximately 1 M frames from mixed synthetic video datasets, batch size 4–8 clips, and learning rate $N$2 using Adam (Lyu et al., 25 May 2026).

The reported video-consistency gains are explicit. On ScanNet, AbsRel drops from 0.101 to 0.073 versus FlashDepth, while $N$3 rises from 90.3 to 96.6. On KITTI, AbsRel improves from 0.103 to 0.062, and $N$4 from 89.5 to 97.3. On Sintel, $N$5 improves from 64.2 to 73.0; on Bonn, it rises from 98.0 to 98.4. The paper also states that, because the encoder and decoder remain frozen, DyFN inherits exactly the same single-image accuracy as MoGe v1 on all four datasets (Lyu et al., 25 May 2026). This is a clear example of DyFN as a causal recurrent module for stabilizing latent feature statistics over time.

6. Empirical patterns, misconceptions, and limitations

Several empirical regularities recur across the literature. First, dynamic normalization is often justified by a mismatch argument: feature means, variances, ranges, or norms drift across batches, domains, time steps, or streams, and static normalization parameters fail to track that drift. Second, the dynamic component can be placed at different points in the pipeline: before normalization by recomputing $N$6; after normalization by generating $N$7; or at the representation level by constraining feature geometry. Third, improvements are often strongest precisely in the regimes where static normalization is least reliable: small batch sizes, aggressive learning rates, domain shift, extraneous covariates, oversmoothing-prone graph propagation, or long streaming horizons (Liu et al., 2021, Kaku et al., 2020, Yang et al., 2021, Lyu et al., 25 May 2026).

A common misconception is that all DyFN methods do the same thing. They do not. In (Liu et al., 2021), the defining move is to generate affine parameters dynamically; in (Kaku et al., 2020) and (Mirza et al., 2021), $N$8 remain fixed and only the normalization statistics are updated; in (Yang et al., 2021) and (Bao, 2023), the operative mechanism is hyperspherical or cosine-based feature normalization; in (Janiak et al., 17 Oct 2025), the dynamic object is the radial geometry of feature space; in (Marasco et al., 18 Mar 2026) and (Lyu et al., 25 May 2026), normalization is embedded into explicitly sequential models. This suggests that DyFN is a methodological family unified by adaptive normalization, not by one invariant implementation.

The limitations are equally heterogeneous and are explicitly documented. DN-C is “often unstable and fails to converge,” and DyFN in the DN-B sense still introduces slightly more parameters and FLOPs than BN (Liu et al., 2021). Adaptive inference-time normalization helps under extraneous shifts, but on homogeneous data traditional batch-norm wins (Kaku et al., 2020). DUA adapts only first- and second-order moments, relies on BN layers, and assumes a somewhat stationary or gradually shifting test stream; abrupt, multi-modal shifts could require reinitialization or hybrid strategies (Mirza et al., 2021). CLeAN has an initial warm-up period, requires tuning of $N$9 and the architecture of $\beta$00, and does not explicitly guard against adversarial outliers that could temporarily skew the EMA (Marasco et al., 18 Mar 2026). In OOD detection, empirically optimal $\beta$01 values vary widely from –1 to +2.5, so there is “no one-size-fits-all,” and mis-tuning may distort cluster angles (Janiak et al., 17 Oct 2025).

The prospective directions named in the literature are correspondingly broad. They include combining DyFN with Layer Norm, Instance Norm or Group Norm for tasks beyond vision such as RNNs and Transformers, applying DyFN in GANs or domain-adaptation networks, jointly optimizing reduction ratio $\beta$02 and group width $\beta$03 by architecture search, extending dynamic-momentum scheduling to other normalization schemes, and applying online regulation of activation statistics to streaming vision tasks such as semantic segmentation, optical-flow estimation, or generative video frame interpolation (Liu et al., 2021, Mirza et al., 2021, Lyu et al., 25 May 2026). A plausible implication is that future DyFN research will continue to move away from a narrow BN-centric view and toward a broader theory of adaptive feature statistics and feature geometry under non-stationarity.