Length-Normalized Entropy Overview

• Length-Normalized Entropy is a collection of techniques that normalize traditional entropy by calibrating it against a canonical measure of size, scale, or support.
• It is applied across diverse fields—from source coding and surface dynamics to spatial analysis and sparse regularization—using domain-specific normalization factors.
• The normalization methods ensure scale invariance, improved finite-sample estimation, and meaningful comparisons by referencing extensive quantities like sequence length, area, or maximal entropy.

Length-normalized entropy denotes a family of entropy constructions in which an extensive or scale-dependent entropy is divided by, or otherwise calibrated against, a natural measure of size. In the literature represented here, the phrase does not refer to a single invariant. Instead, it appears as per-symbol Shannon entropy for discrete sources, area-normalized entropy for pseudo-Anosov automorphisms, maximum-entropy normalization in spatial box counting, normalized-entropy regularization for effective sparsity, and a fixed-reservoir normalization mechanism for short time-series entropy estimation. A related but distinct direction treats entropy as a length-valued invariant on module categories rather than as a normalized Shannon quantity (Oliveira et al., 2018, Kojima et al., 2014, Chen, 2016, He et al., 14 Mar 2026, Heidari et al., 2022, Salce et al., 2017).

1. Taxonomy of normalization mechanisms

The available definitions show that normalization is domain-specific. In some settings the normalization factor is an explicit divisor, as in $H_L/L$ or $M_q/M_{\max}$. In others it is built into the state space or probability parametrization, as in $\pi_i(x)=|x_i|/\|x\|_1$ for effective sparsity or the fixed reservoir size $N=19{,}625$ in NNetEn. This suggests that “length-normalized entropy” is best understood as a structural principle: entropy is rendered comparable across objects of different size, scale, or support by referencing a canonical extensive quantity.

Setting	Quantity	Normalizing object
Discrete memoryless source	$H_L/L$	Sequence length $L$
Pseudo-Anosov automorphism	$$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$	Hyperbolic area $2\pi|\chi(\Sigma)|$
Spatial box counting	$$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$	Maximum entropy $M_{\max}(\varepsilon)$
Effective sparsity	$M_q/M_{\max}$0, $M_q/M_{\max}$1	Normalized magnitude distribution $M_q/M_{\max}$2
Short time series	$M_q/M_{\max}$3	Fixed reservoir size $M_q/M_{\max}$4

A plausible implication is that comparisons across these literatures require attention to what is being normalized: symbol count, area, scale, coefficient mass, or model architecture are not interchangeable.

2. Per-symbol entropy in source coding and short-sequence analysis

For a discrete memoryless source $M_q/M_{\max}$5 with alphabet $M_q/M_{\max}$6 and probabilities $M_q/M_{\max}$7, the information content of a single symbol is

$M_q/M_{\max}$8

For a length-$M_q/M_{\max}$9 block $\pi_i(x)=|x_i|/\|x\|_1$0, the Shannon entropy is

$\pi_i(x)=|x_i|/\|x\|_1$1

In the memoryless case,

$\pi_i(x)=|x_i|/\|x\|_1$2

so the normalized entropy is

$\pi_i(x)=|x_i|/\|x\|_1$3

This quantity is the expected number of bits of uncertainty in each source letter; in source coding it is the fundamental lower bound on the average code-length per symbol, and in ergodic processes it converges to the Shannon entropy rate. Dividing by $\pi_i(x)=|x_i|/\|x\|_1$4 also lets one compare different block-lengths on the same scale (Oliveira et al., 2018).

The short-sequence literature in the same source extends this normalization by introducing the second central moment of $\pi_i(x)=|x_i|/\|x\|_1$5, called the information fluctuation: $\pi_i(x)=|x_i|/\|x\|_1$6 For the plug-in estimator

$\pi_i(x)=|x_i|/\|x\|_1$7

the Central Limit Theorem yields

$\pi_i(x)=|x_i|/\|x\|_1$8

Hence the standard deviation of $\pi_i(x)=|x_i|/\|x\|_1$9 is $N=19{,}625$0. The same framework defines a confidence-adjusted practical coding rate

$N=19{,}625$1

and refines the asymptotic equipartition property through a statistical notion of typicality based on confidence intervals for empirical entropy. In this formulation, normalization by sequence length remains fundamental, but finite-$N=19{,}625$2 behavior is governed by an explicit $N=19{,}625$3 correction.

An operational counterpart appears in fixed-to-variable length resolution coding. The number of random bits required to approximate a target distribution in terms of un-normalized informational divergence is considered, and for a variable-to-variable length encoder this number is lower bounded by the entropy of the target distribution. A fixed-to-variable length encoder is constructed using M-type quantization and Tunstall coding, and in the limit it achieves an un-normalized informational divergence of zero with the number of random bits per generated symbol equal to the entropy of the target distribution. Numerical results show that the proposed encoder significantly outperforms the optimal block-to-block encoder in the finite length regime (Böcherer et al., 2013).

3. Area-normalized entropy in surface dynamics

For an oriented surface $N=19{,}625$4 of genus $N=19{,}625$5 with $N=19{,}625$6 punctures and $N=19{,}625$7, endowed with its unique complete hyperbolic metric of curvature $N=19{,}625$8, Gauss–Bonnet gives

$N=19{,}625$9

If $H_L/L$0 is pseudo-Anosov, the quantity

$H_L/L$1

is referred to as the length-normalized, or simply normalized, entropy. Equivalently, one sometimes considers $H_L/L$2 or the combination $H_L/L$3, but the fundamental normalization factor throughout is the hyperbolic area $H_L/L$4 (Kojima et al., 2014).

Kojima–McShane prove the linear entropy–volume bound

$H_L/L$5

or equivalently,

$H_L/L$6

Thus,

$H_L/L$7

The constant $H_L/L$8, equivalently $H_L/L$9, is independent of the type of $L$0. This is precisely the role of the normalization: it produces a quantity whose lower bound does not deteriorate with genus or puncture count.

Several corollaries follow. Using the fact that the smallest volume of a noncompact orientable hyperbolic $L$1-manifold is $L$2, one obtains

$L$3

for $L$4, improving Penner’s classical bound in that regime. Another corollary states that for each $L$5 there exist only finitely many cusped hyperbolic $L$6-manifolds $L$7 such that every pseudo-Anosov $L$8 with

$L$9

arises, up to isotopy, as the monodromy of a fibration on a Dehn filling of one of the $$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$0. The paper also proves a Weil–Petersson analogue: $$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$1 equivalently,

$$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$2

for closed $$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$3. In this setting, length-normalized entropy is therefore a geometric invariant calibrated by area and linked linearly to the hyperbolic volume of the mapping torus.

4. Maximum-entropy normalization in spatial systems

In functional box-counting analyses of spatial patterns, normalization is performed relative to the maximum entropy available at a given scale. Given a partition of space into $$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$4 nonempty boxes of linear size $$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$5, with probabilities

$$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$6

the order-$$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$7 Rényi entropy is

$$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$8

The special cases are

$$\|\phi\|_{\mathrm{norm}}=2\pi|\chi(\Sigma)|\cdot \mathrm{ent}\,\phi$$9

for macro-state entropy and

$2\pi|\chi(\Sigma)|$0

for Shannon information entropy. If the total number of boxes is

$2\pi|\chi(\Sigma)|$1

where $2\pi|\chi(\Sigma)|$2 is the embedding dimension, then the maximum entropy is

$2\pi|\chi(\Sigma)|$3

The generalized dimension is defined by

$2\pi|\chi(\Sigma)|$4

with $2\pi|\chi(\Sigma)|$5 (Chen, 2016).

The central equivalence relation is

$2\pi|\chi(\Sigma)|$6

Writing

$2\pi|\chi(\Sigma)|$7

one obtains a dimensionless measure in $2\pi|\chi(\Sigma)|$8. The paper interprets $2\pi|\chi(\Sigma)|$9 as the fraction of the maximum and therefore as a length-normalized entropy between $$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$0 and $$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$1: $$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$2 corresponds to complete concentration and $$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$3 to complete uniformity. In the limit $$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$4, the equality becomes exact; for finite grids,

$$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$5

This framework is used to analyze urban form. Chen applies a functional box-counting method to Beijing land-use maps for the years 1988–2009 at resolutions $$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$6, $$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$7, and for $$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$8. The theoretical inference is verified by observational data of urban form, and the paper concludes that normalized spatial entropy is equal to normalized fractal dimension. In practical terms, the computational workflow is explicit: rasterize or box-count the spatial dataset at scale $$J_q(\varepsilon)=M_q(\varepsilon)/M_{\max}(\varepsilon)$$9, compute $M_{\max}(\varepsilon)$0, choose $M_{\max}(\varepsilon)$1, evaluate $M_{\max}(\varepsilon)$2, compute $M_{\max}(\varepsilon)$3, and form

$M_{\max}(\varepsilon)$4

Here the normalization factor is not physical length alone, but the entropy of an entirely filled $M_{\max}(\varepsilon)$5-dimensional domain at the same resolution.

5. Normalized entropy and the effective number of nonzeros

In inverse problems and sparse regularization, normalized entropy is used to quantify concentration of coefficient magnitudes rather than sequence uncertainty. For $M_{\max}(\varepsilon)$6, define the normalized magnitude distribution

$M_{\max}(\varepsilon)$7

The length-normalized Shannon entropy is

$M_{\max}(\varepsilon)$8

and for $M_{\max}(\varepsilon)$9, $M_q/M_{\max}$00, the length-normalized Rényi entropy of order $M_q/M_{\max}$01 is

$M_q/M_{\max}$02

As $M_q/M_{\max}$03, $M_q/M_{\max}$04. The corresponding Effective Number of Nonzeros is

$M_q/M_{\max}$05

and the Rényi version is

$M_q/M_{\max}$06

Because $M_q/M_{\max}$07 is a strictly increasing transform of $M_q/M_{\max}$08, minimizing $M_q/M_{\max}$09 under a data-fit constraint $M_q/M_{\max}$10 is equivalent to the entropy-regularized formulation

$M_q/M_{\max}$11

(He et al., 14 Mar 2026).

A key structural result is the decomposition

$M_q/M_{\max}$12

where $M_q/M_{\max}$13, $M_q/M_{\max}$14 on $M_q/M_{\max}$15, and

$M_q/M_{\max}$16

Exponentiating gives

$M_q/M_{\max}$17

with

$M_q/M_{\max}$18

If all nonzeros $M_q/M_{\max}$19 on $M_q/M_{\max}$20 are equal, then $M_q/M_{\max}$21, $M_q/M_{\max}$22, $M_q/M_{\max}$23, and $M_q/M_{\max}$24. If the mass concentrates on a few coordinates, then $M_q/M_{\max}$25 and $M_q/M_{\max}$26. This is the sense in which normalized entropy interpolates between support size and amplitude concentration.

The theoretical guarantees are stated under the usual $M_q/M_{\max}$27-RIP assumption $M_q/M_{\max}$28. The paper proves that two vectors $M_q/M_{\max}$29 with at most $M_q/M_{\max}$30 “effective nonzeros” satisfy

$M_q/M_{\max}$31

where $M_q/M_{\max}$32 is the union of their top-$M_q/M_{\max}$33 supports, $M_q/M_{\max}$34 bound the tail energy outside top-$M_q/M_{\max}$35, and $M_q/M_{\max}$36 depend only on $M_q/M_{\max}$37. The corollaries are uniqueness in the noiseless case, stability under perturbations, and insensitivity to tiny noise because ENZ depends on the normalized distribution $M_q/M_{\max}$38, so adding arbitrarily small entries outside the dominant support does not change $M_q/M_{\max}$39 appreciably.

Algorithmically, the paper introduces the unnormalized surrogate

$M_q/M_{\max}$40

for a fixed scaling constant $M_q/M_{\max}$41. This function is separable and positively homogeneous of degree $M_q/M_{\max}$42, admits simple coordinate-wise derivatives, and preserves the ENZ-minimization property. In practice one fixes $M_q/M_{\max}$43 at the current $M_q/M_{\max}$44 or $M_q/M_{\max}$45 norm of the iterate, runs a smooth solver such as L-BFGS with an $M_q/M_{\max}$46-smoothed absolute value, then updates $M_q/M_{\max}$47 and repeats. Compressed sensing experiments with a correlated Gaussian sensing matrix show higher empirical success rates across a wide range of sparsities and noise levels than $M_q/M_{\max}$48 (IHT), $M_q/M_{\max}$49 (ISTA), and log-sum (IRL1); image denoising in the gradient domain yields superior or competitive PSNR and the best SSIM.

6. Length normalization for short time series and related non-equivalent notions

NNetEn addresses a different normalization problem: entropy estimation for time series of arbitrary length $M_q/M_{\max}$50, with $M_q/M_{\max}$51. The method uses the LogNNet neural network with $M_q/M_{\max}$52 hidden neurons and input dimension $M_q/M_{\max}$53, so the reservoir weight matrix $M_q/M_{\max}$54 has size

$M_q/M_{\max}$55

After training on the MNIST-10 dataset for $M_q/M_{\max}$56 epochs, the classification accuracy on a held-out test set of size $M_q/M_{\max}$57 is

$M_q/M_{\max}$58

and by definition

$M_q/M_{\max}$59

In practice one fixes $M_q/M_{\max}$60 and writes $M_q/M_{\max}$61 (Heidari et al., 2022).

The normalization over variable signal length is accomplished by mapping $M_q/M_{\max}$62 into the $M_q/M_{\max}$63 entries of $M_q/M_{\max}$64. Six reservoir-filling methods are studied. Two of them, Method 3 and Method 6, stretch the time series by linear interpolation: $M_q/M_{\max}$65

$M_q/M_{\max}$66

then fill the reservoir with $M_q/M_{\max}$67. Methods 3 and 6 are identical in the construction of $M_q/M_{\max}$68, namely by uniform linear interpolation between successive original samples. This stretching preserves the first-order dynamics of $M_q/M_{\max}$69 but ensures the reservoir is full. The most reliable methods for short time series are Method 3 and Method 5.

The paper contrasts this approach with conventional sample-Shannon entropy for a quantized series of $M_q/M_{\max}$70 bins,

$M_q/M_{\max}$71

for which

$M_q/M_{\max}$72

By contrast,

$M_q/M_{\max}$73

and

$M_q/M_{\max}$74

is uniformly small, $M_q/M_{\max}$75, even for $M_q/M_{\max}$76 as small as $M_q/M_{\max}$77. The numerical evaluation further states that the percentage deviation of NNetEn from its reference is under $M_q/M_{\max}$78 once $M_q/M_{\max}$79 or $M_q/M_{\max}$80, and entropy decreases with an increase in the bias component. Thus NNetEn is length-normalized in the architectural sense that outputs always lie in $M_q/M_{\max}$81 regardless of $M_q/M_{\max}$82, with robustness to short length and low-amplitude noise.

A common misconception is that every “length” in entropy theory indicates normalization by physical or sequence length. The module-theoretic notion of intrinsic valuation entropy shows otherwise. For an archimedean non-discrete valuation domain $M_q/M_{\max}$83, Northcott and Reufel’s unique non-discrete valuation length

$M_q/M_{\max}$84

is determined on cyclic modules by

$M_q/M_{\max}$85

and extended by upper continuity. For an $M_q/M_{\max}$86-module $M_q/M_{\max}$87, a submodule $M_q/M_{\max}$88 is $M_q/M_{\max}$89-inert if

$M_q/M_{\max}$90

With partial trajectories

$M_q/M_{\max}$91

one defines

$M_q/M_{\max}$92

The Intrinsic Algebraic Yuzvinski Formula states that if $M_q/M_{\max}$93 is $M_q/M_{\max}$94-linear and $M_q/M_{\max}$95 is primitive with leading coefficient $M_q/M_{\max}$96, then

$M_q/M_{\max}$97

The paper proves that $M_q/M_{\max}$98 is a length function for the category of $M_q/M_{\max}$99-modules and is essentially the unique invariant for $\pi_i(x)=|x_i|/\|x\|_1$00 with these properties (Salce et al., 2017).

Taken together, these constructions show that length-normalized entropy is not a single formula but a recurrent strategy for removing extraneous dependence on size, scale, or support. In one branch it yields per-symbol information rates; in another it calibrates entropy by hyperbolic area; in another it divides by a maximal entropy determined by resolution; in another it induces stable measures of effective sparsity; and in adjacent theories it motivates entropy built from non-discrete length functions rather than explicit normalization.