D-Normal Regularizer Overview

• D-Normal Regularizer is a suite of techniques that exploit geometric, rescaling, and function-space structures to regularize models beyond standard weight decay.
• In 3D Gaussian surface reconstruction, it couples depth and normal information, improving surface quality and rendering performance as shown by significant F-score gains.
• Its applications extend to deep networks with diagonal rescaling and stochastic function norm penalties, yielding robust training and optimized rescaling schemes in varied data regimes.

Searching arXiv for the relevant papers and terminology. D-Normal Regularizer denotes several distinct but related constructions in recent literature. The most explicit named instance is the Depth-Normal regularizer for Gaussian surface reconstruction, where pseudo normals are compared with normals derived from rendered depth so that the regularization signal updates both Gaussian orientation and position (Chen et al., 2024). Closely related usages include geometry-aware total variation on sphere-valued normal representatives for surface segmentation (Baumgärtner et al., 2024), diagonal-rescaling objectives over positive matrices $D_k$ in ReLU-network equi-normalization (Stock et al., 2019), and direct function-norm penalties that regularize the mapping $f$ rather than only its parameters (Triki et al., 2016, Triki et al., 2017). This suggests that the term is best understood as a non-uniform label for regularizers that exploit geometric, rescaling, or function-space structure instead of relying exclusively on conventional weight decay.

1. Scope and principal meanings

In the cited literature, the label “D-Normal” is attached to different objects. In one line of work, it refers to a Depth-Normal coupling for 3D Gaussian Splatting. In another, it denotes a regularizer built from the normal-label geometry of a triangulated surface. In a third, it is naturally associated with a $D$-dependent normalization objective over positive diagonal rescalings in ReLU networks. A further usage interprets the term as direct function-norm regularization, where the controlled quantity is the norm of the realized mapping rather than the norm of the parameters.

Usage	Regularized quantity	Representative source
Depth-Normal regularizer	$$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$	(Chen et al., 2024)
Label-space total variation	$\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$	(Baumgärtner et al., 2024)
Equi-normalization over diagonal rescalings	$\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$	(Stock et al., 2019)
Weighted function norm	$\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$	(Triki et al., 2016, Triki et al., 2017)

A common thread is that each formulation regularizes an object with structural meaning: a depth-induced normal field, a sphere-valued label representative, a rescaling-equivalence class, or the function itself. A plausible implication is that the term “D-Normal” is less a single canonical method than a cluster of normalization- or normal-based regularizers whose precise meaning depends on the modeling context.

2. Depth-Normal regularizer in Gaussian surface reconstruction

In VCR-GauS, the D-Normal regularizer is introduced for surface reconstruction with 3D Gaussian Splatting. The motivating claim is that direct supervision of rendered normals is too weak for Gaussian geometry because the rendered normal is essentially tied to the Gaussian’s rotation. For a Gaussian with covariance

$$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$

the normal is computed from the Gaussian’s smallest scaling axis / rotation,

$$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$

so direct normal supervision mainly updates the rotation parameters and is less effective for position (Chen et al., 2024).

The method therefore couples depth and normal. A 3D Gaussian is written as

$$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$

with center $f$0, rotation $f$1, and diagonal scaling $f$2. To make Gaussian surfaces plane-like, the model applies the scale regularization

$f$3

Depth is not taken from the Gaussian center; instead it is the intersection between a ray and the flattened Gaussian plane. For a plane represented by position $f$4 and normal $f$5, any point $f$6 on the plane satisfies

$f$7

and for a ray point $f$8, the intersection depth along the z-axis is

$f$9

Rendered depth is then produced by weighted blending: $D$0

The critical step is to derive a normal from the rendered depth. The depth map is back-projected to 3D points, and a D-Normal is computed via finite differences and a cross product. Because this normal depends on the intersection depth, and the intersection depth depends on both $D$1 and $D$2, gradients propagate to both Gaussian normal/orientation and Gaussian position. The basic D-Normal loss compares depth-derived normals $D$3 with pseudo normals $D$4 from a monocular normal estimator: $D$5

The method further introduces a confidence term to mitigate cross-view inconsistency in pseudo normal maps: $D$6 and the final view-consistent D-Normal loss becomes

$D$7

The full objective is

$D$8

with reported weights $D$9, $$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$0, $$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$1, and $$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$2. The reported optimization pipeline initializes from a COLMAP sparse point cloud, uses RGB supervision, adds normal supervision from DSINE for outdoor scenes and GeoWizard for indoor scenes, applies scale regularization, computes intersection depth and D-Normals, uses confidence-weighted D-Normal regularization, performs densification and splitting until 15k iterations, continues optimization to 30k iterations, and extracts a mesh by fusing rendered depth maps with TSDF via Open3D.

The paper’s ablations assign a central role to the D-Normal construction. On TNT, removing D-Normal lowers F-score from 0.40 to 0.30; removing confidence lowers it to 0.36; removing intersection depth lowers it to 0.35; and removing densify-and-split lowers it to 0.33. The empirical interpretation in the paper is that normal rendering alone is insufficient, whereas depth-normal coupling plus confidence weighting and Gaussian refinement yields better geometry, smoother planar surfaces, fewer bumps and protrusions, and competitive appearance quality at about 128–145 FPS rendering.

3. Label-space total variation on normal labels

A second explicit geometric meaning of D-Normal appears in variational surface segmentation on triangular meshes. Here the task is to partition a surface represented by a triangular mesh whose piecewise constant outer normal field is

$$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$3

Each triangle $$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$4 receives a simplex-valued assignment

$$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$5

over label vectors $$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$6. The shared variational model is

$$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$7

with fidelity measured by geodesic distance on the sphere,

$$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$8

Within this framework, the D-Normal regularizer corresponds to label space total variation (L-TV) (Baumgärtner et al., 2024).

The defining idea is not to penalize jumps in the simplex coordinates directly. Instead, each assignment $$w\left(\|\bar{N}_d-N\|_1 + (1-\bar{N}_d\cdot N)\right)$$9 is mapped to a sphere-valued representative $\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$0 through the Karcher mean problem

$\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$1

with optimality condition

$\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$2

The resulting regularizer is

$\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$3

so neighboring triangles are penalized according to geodesic distance between their sphere-valued representatives. The full L-TV model is

$\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$4

The contrast class is assignment-space total variation (A-TV),

$\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$5

which penalizes simplex-coordinate jumps directly. The paper emphasizes that A-TV does not incorporate the metric structure of the labels, whereas L-TV does. This difference matters when multiple labels lie along a geodesic path on the sphere: A-TV may skip intermediate labels because it only measures simplex jumps, while L-TV charges transitions according to actual angular distance and can make intermediate labels cheaper and more natural. The reported consequence is that A-TV may prefer coarse segmentations and label skipping, whereas L-TV better preserves gradual transitions and uses available labels more faithfully, especially in regions of constant curvature.

Optimization is substantially more involved because $\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$6 depends nonlinearly on $\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$7. The paper uses an ADMM / split-Bregman-like scheme with auxiliary variables $\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$8, $\sum_{\edge \in \edges} |\edge|\, \metric{\mphi_{\eplus}}{\mphi_{\eminus}}$9, and $\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$0, an augmented Lagrangian, and four alternating primal subproblems: a $\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$1-subproblem solved by Euclidean gradient descent with Armijo backtracking; an $\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$2-subproblem with a closed-form vector soft-thresholding solution; an $\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$3-subproblem formulated as a convex QP on the simplex and solved by Riemannian gradient descent, active-set identification, and KKT solving; and a coupled nonlinear $\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$4-subproblem solved approximately by Riemannian gradient descent with Armijo backtracking. The empirical conclusion is that, although computationally more expensive, L-TV removes noise more reliably in regions of constant curvature, tends to use more of the available labels, is more robust to the choice of $\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$5, and often gives better segmentation quality.

4. Diagonal rescaling, equi-normalization, and $\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$6-dependent balancing

A distinct interpretation of D-Normal arises from positive diagonal rescaling in ReLU networks. In Equi-normalization of Neural Networks, the phrase “D-normal” does not appear explicitly, but the central object is a family of diagonal positive matrices $\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$7 that define a function-preserving rescaling equivalence (Stock et al., 2019). For a feedforward network

$\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$8

rescaled weights are written as

$\sum_{k=1}^q \|D_{k-1}^{-1}W_kD_k\|_p^p$9

This exploits positive homogeneity of ReLU,

$\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$0

so incoming weights of a hidden unit can be multiplied by a positive scalar and outgoing weights divided by the same scalar without changing the network’s input-output function.

The minimized objective is the global norm of the reparameterized network,

$\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$1

where $\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$2 encodes the diagonal scalings. For $\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$3,

$\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$4

which is exactly the standard weight decay regularizer up to the regularization coefficient. The method therefore chooses a canonical representative of the rescaling-equivalence class by minimizing the global $\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$5 norm, and for $\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$6 this is equivalent to minimizing weight decay within that class.

The algorithm is inspired by Sinkhorn-Knopp and performs coordinate descent over adjacent layers. At iteration $\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$7, one layer $\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$8 is updated by

$\mathbb{E}_{z\sim Q}\left[\|f(z)\|_2^2\right]$9

with closed-form coordinate update

$$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$0

In the pseudo-code, the method computes $$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$1, $$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$2, then sets

$$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$3

followed by

$$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$4

If biases are present, they are rescaled consistently as $$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$5. Functional equivalence is preserved because hidden activations transform as $$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$6, implying $$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$7.

The theoretical result states that, assuming each hidden neuron is connected to at least one input and one output neuron, the generated sequence converges to a unique $$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$8; the limiting rescaled weights minimize the global $$\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top \mathbf{R}^\top,$$9 norm among all rescaling-equivalent weights; and $$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$0 does not depend on initialization. The proof uses Tseng’s theorem on block coordinate descent, differentiability of $$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$1, compactness of the level set, and uniqueness of each block minimizer; strict convexity after the change of variables $$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$2 strengthens the uniqueness argument.

In training, the paper recommends interleaving one ENorm cycle with each SGD step and applying the same diagonal rescaling matrices to the momentum buffers. The method is batch-independent and requires no extra learned parameters. On ResNet-18, the reported accessed-element counts are approximately 12 million for ENorm, 636 million for BN, and 40 million for GN, with theoretical overheads of about 50× faster than BN and 3× faster than GN assuming similar parallelization. Empirically, on ImageNet with ResNet-18, ENorm gives top-1 accuracies of 69.70, 70.10, 71.03, and 71.14 at batch sizes 32, 64, 128, and 256; BN gives 68.01, 69.38, 70.83, and 71.37; and GN gives 68.94, 68.90, 70.69, and 70.64. The reported pattern is that ENorm is particularly effective when BN struggles due to small batch sizes.

5. Function-space interpretations and stochastic function norm penalties

Another important usage interprets D-Normal regularization as a direct function-norm penalty. The central claim is that weight-based regularizers do not measure function complexity in a principled way for deep networks because the same function can be represented by many different parameterizations through hidden-unit permutations, layer rescalings, and other symmetries. The proposed replacement is the weighted $$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$3 norm of the realized mapping,

$$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$4

where $$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$5 is a sampling distribution over inputs (Triki et al., 2016, Triki et al., 2017).

The training objective adds this penalty to empirical risk: $$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$6 Because exact computation of a DNN function norm is NP-hard for ReLU networks of depth at least 4, the regularizer is approximated by Monte Carlo sampling. If $$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$7, then

$$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$8

Practical training therefore samples an additional regularization minibatch, computes the average squared output norm on that batch, and backpropagates it together with the supervised loss. Two sampling strategies are emphasized: Gaussian sampling based on estimated data statistics, and VAE-generated sampling from a generative model trained on the data. The papers stress that a new regularization batch is generated at each iteration whenever possible.

The theoretical motivation has two components. First, exact function-norm computation is NP-hard, which justifies stochastic approximation. Second, the generalization bound decreases when the weighted function norm and the variance of $$\mathbf{n}=\mathbf{R}[k, :] \in \mathbb{R}^3,\quad k=\arg\min\big([s_1,s_2,s_3]\big),$$9 are controlled, and when the sampling distribution $$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$0 is close to the true marginal input distribution $$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$1. This places the choice of $$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$2 at the center of the method: good matching between $$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$3 and the data distribution improves both the approximation and the induced generalization control.

Empirically, the method is reported to be especially effective in low-sample regimes. On Oxford Flowers with kernel logistic regression, weighted function-norm regularization behaves similarly to RKHS norm regularization. On MNIST with LeNet trained on very small subsets, it improves over weight decay, dropout, and batch normalization; dropout can be complementary, whereas batch normalization can suppress or nullify the effect. The papers also report that poorly matched Gaussian samples can cause divergence or near-random performance, while VAE-based sampling works better. On Cityscapes semantic segmentation with ENet trained on only 500 labeled images, adding weighted function-norm regularization improves validation performance and output quality relative to the low-data baseline.

6. Related normalization-oriented regularizers and conceptual distinctions

Several closely related regularizers clarify what D-Normal regularization is not. Norm Loss is a soft unit-norm penalty that steers each filter toward the Oblique manifold by penalizing

$$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$4

so each output-channel filter is encouraged to satisfy $$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$5. Unlike weight decay, which always shrinks weights toward zero, Norm Loss can shrink or enlarge a filter depending on whether its norm is above or below 1; unlike orthogonal regularizers, it does not force mutual orthogonality; and unlike projection-based normalization, it does not abruptly renormalize the weights after the gradient step. The paper reports negligible overhead, greater stability with respect to batch size and regularization factor, and improvements over weight decay on CIFAR-100 and on ImageNet 2012 with ResNet50, where validation errors improve from 25.29 to 24.34 top-1 and from 7.86 to 7.44 top-5 within the reported setup (Georgiou et al., 2021).

A complementary distinction is developed for the weight normalization family. When weights are normalized or standardized, standard weight decay can lose its usual interpretation as a generalization-oriented regularizer. The normalized weights used by the network become invariant to overall scale, so the conventional $$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$6 penalty on the normalized weights collapses to a constant, and even explicit decay on the underlying parameter tensor mainly modulates the effective learning rate rather than the optimization target. The paper further argues that this can create two pathologies: the objective may have no global minimum, and the weight norm may collapse toward zero, causing effective learning-rate divergence and gradient float overflow. The proposed remedy is an $$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$7-shifted $$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$8 regularizer that penalizes deviation from a positive target norm. For weight normalization the modified penalty is

$$G(\mathbf{x}) = \exp\left\{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mathbf{p})\right\},$$9

with gradient

$f$00

This guarantees a finite optimum at positive norm, prevents collapse, improves stability, and yields modest but consistent gains on ImageNet, CIFAR-100, and COCO; for example, WS-ResNet-50 on ImageNet improves from 76.74% top-1 to 76.86% top-1, and Cascade R-CNN + FPN with ResNet-50 backbone improves AP from 41.1 baseline to 41.8 with WS + $f$01 (Xiang et al., 2019).

These related methods show that normalization-oriented regularization can target very different objects: unit filter norms, positive target norms under normalized parameterizations, diagonal rescaling equivalence classes, sphere-valued normal fields, or the function norm itself. A common misconception is to treat all such methods as variants of ordinary weight decay. The cited work argues otherwise. In the function-space setting, the penalty is defined on $f$02 rather than on parameters. In the weight-normalized setting, standard decay may have no influence on generalization and may act chiefly through effective learning-rate control. In geometry-driven settings such as VCR-GauS and label-space TV segmentation, the regularizer is designed to respect the geometry of normals or labels rather than merely suppress parameter magnitude.