Discrete Entropy Spectrum

• Discrete Entropy Spectrum is defined as the range and distribution of entropy values from discrete probability measures, capturing inherent structural fluctuations and arithmetic properties.
• Analytical techniques, including orthogonal polynomials and zeta functions, reveal sharp peaks and bounds in the entropy landscape that inform stability and scaling behavior.
• This concept has broad applications in dynamical systems, coding theory, and quantum mechanics, guiding approaches in sampling, invariant criteria, and information limits.

A discrete entropy spectrum is the range, structure, and distribution of entropy values—typically Shannon entropy or related functionals—arising from discrete probability distributions, systems, or processes. The term “spectrum” emphasizes the variability, fluctuations, and possibly arithmetic or structural signatures that discrete distributions can display, as well as their implication in broader areas such as dynamical systems, operator theory, combinatorics, and quantum information. Recent research characterizes, bounds, and exploits the discrete entropy spectrum via analytical, probabilistic, algebraic, and algorithmic techniques across mathematical physics, probability, and information theory.

1. Theoretical Foundations and Definitions

For a discrete (possibly countably infinite) probability distribution $\pi$ over a finite or countable set $\mathcal{A}$, the Shannon entropy is

$$H(\pi) = -\sum_{x\in \mathcal{A}} \pi(x)\log \pi(x).$$

The spectrum arises as one considers entropy’s dependence on the underlying discrete structure, normalization, or induced marginal and conditional measures (as in processes), and investigates how these features vary across classes of distributions, sequences, or orbits.

In the context of orthogonal polynomials, the squared values $p_{i-1}^2(\lambda_j^{(n)})$ evaluated at the zeros $\lambda_j^{(n)}$ of the $n$th polynomial generate a discrete probability distribution, leading to the entropy functional

$$\mathcal S_{n,j} = -\sum_{i=1}^n \Psi_{ij}^2 \log(\Psi_{ij}^2),$$

where $$\Psi_{ij}^2 = \frac{p_{i-1}^2(\lambda_j^{(n)})}{\sum_{k=0}^{n-1} p_k^2(\lambda_j^{(n)})}$$ (0710.2134).

In measure-preserving transformations and ergodic theory, the concept of scaling entropy generalizes Kolmogorov entropy by associating to each transformation $T$ and admissible metric $p$ a scaling sequence $\{c_n\}$ that describes the growth rate of entropy-like invariants (e.g., E-entropy) under the action of $T$. Here, the spectrum captures the sequence of entropy values arising as the metric is averaged over the orbits of $T$ (1008.4946).

2. Analytical Tools, Invariants, and Explicit Spectra

Explicit expressions for the discrete entropy spectrum often exploit structural properties of the underlying system. For classical orthogonal polynomials, closed formulas such as

$$\mathcal S_{n,j} = \log n + \log 2 - 1 + \frac{\log 2}{n} + \mathcal R(d/(2n)),$$

with $d = \gcd(2j-1,n)$ and

$$\mathcal R(x) = -2 \sum_{k=1}^\infty \zeta(2k+1) x^{2k+1},$$

connect the entropy directly to number-theoretic functions (the Riemann zeta function) and the arithmetic structure of the polynomials’ zeros (0710.2134). This reveals sharp peaks and dips in the entropy as $j$ varies, governed by the greatest common divisor between index and degree.

In ergodic theory, the main result asserts that an automorphism has discrete spectrum if and only if the scaling sequence $\{c_n\}$ of the E-entropy (based on admissible metrics) remains bounded, providing a metric-invariant, non-spectral criterion for pure point spectrum (1008.4946).

Quantum and gravitational physics connects discrete entropy spectra to universal entropy bounds. The Bekenstein and Susskind bounds—when applied to black-body radiation—imply a minimal spatial and temporal cutoff, which means the entropy of any bounded region is quantized: the spectrum is discrete at the Planck scale, as each elementary volume supports a finite, countable number of bits (1106.1778).

3. Discrete Entropy Power Inequalities and Central Limit Regimes

The search for discrete analogues of continuous entropy power inequalities (EPI) has led to formulations where the geometric or Poisson distribution plays the extremal role:

• In discrete settings, the entropy power $V_g(X)$ is defined as the mean of a geometric random variable having entropy $H(X)$, with the geometric law maximizing entropy for a fixed mean (Guha et al., 2016).
• For independent discrete variables $X$ and $Y$, new forms of the EPI have been proposed and proved (in some cases) such as

$N_d[X] + N_d[Y] \leq 2 N_d[X+Y],$

where $N_d[X] = (1/2\pi e) e^{2H[X]}$ (Nekouei et al., 2019).

Under convolution or thinning operations, the entropy of sums or normalized combinations of discrete variables increases monotonically, with the spectrum converging towards that of the geometric (or Poisson, in certain restricted classes) distribution (Johnson, 2015, Guha et al., 2016, Gavalakis et al., 2021). In the context of the central limit theorem, the entropy of standardized sums of lattice random variables converges to the entropy of a discretized Gaussian after normalization, and the spectrum becomes sharply peaked around this limit (Gavalakis et al., 2021).

4. Dynamical Systems, Sequence Entropy, and Spectrum Rigidity

In dynamical systems and ergodic theory, spectral properties and entropy are tightly interwoven:

• Discrete (pure point) spectrum corresponds to systems isomorphic to rotations on compact abelian groups and is characterized by zero measure-theoretic sequence entropy; such systems, including all invariant measures of zero-entropy maps on quasi-graphs and certain dendrites, exhibit a rigid, arithmetic-like entropy spectrum (García-Ramos, 2014, Li et al., 2018, Foryś-Krawiec et al., 2021).
• Quasi-discrete spectrum extends the discrete case, yet systems with quasi-discrete spectrum also have zero entropy, hence exhibit the same rigidity—no complex or continuous entropy spectrum emerges (Haase et al., 2015).

The role of Gelfand’s theorem and factor maps onto odometers provides algebraic and functional-analytic tools to capture and classify the structure of invariant measures, showing that the discrete entropy spectrum arises as a decisive invariant in characterizing dynamics, especially in spaces with one-dimensional continuum structure.

5. Structural Spectrum, Flat Modes, and Multiscale Behavior

Recent advances in sampling discrete spaces highlight the importance of not only peak values but the “flatness” or volume of modes in the energy landscape. The Entropic Discrete Langevin Proposal (EDLP) for sampling discrete distributions couples the discrete variable $\theta$ with a continuous auxiliary variable $\theta_a$, defining a local entropy functional

$$\mathcal F(\theta_a;\eta) = \log \sum_{\theta \in \Theta} \exp \left( U(\theta) - \frac{1}{2\eta} \|\theta - \theta_a\|^2 \right).$$

This functional quantifies the volume ("flatness") of high-probability configurations near $\theta_a$, enriching the entropy spectrum by not only the heights of probability peaks but the “spread” of near-maximal configurations (Mohanty et al., 5 May 2025). The sampler then prefers flat modes—robust solutions with higher entropy “basins”—over sharp, isolated maxima.

This approach captures a richer entropy spectrum in combinatorial spaces, favoring configurations that are more stable under perturbations, with applications to Bernoulli models, restricted Boltzmann machines, binary neural networks, and combinatorial optimization. Non-asymptotic convergence guarantees and empirical results show superior diversity and stability when sampling distributions with flat entropy spectra.

6. Applications, Implications, and Broader Connections

The discrete entropy spectrum—defined and bounded through the above frameworks—has broad and fundamental implications:

• In quantum mechanics, information theory, and statistical physics, discrete entropy spectra reflect quantized informational content, uncertainty relations, and minimal bounds on physical measurement (1106.1778, Archer et al., 2013).
• In coding theory and analog source compression, the extension of entropy to integer-dimensional singular random variables informs the design of codes and the fundamental compressibility of singularly supported measures (Koliander et al., 2015).
• In random processes, bounds based on second-order statistics (e.g., auto-covariances, PSD) allow estimation of entropy rate spectra without explicit enumeration, crucial for large alphabets or hidden Markov systems (Tamir, 2022).
• Discrete entropy spectra underpin the analysis of entropy production in nonequilibrium Markov processes, with structure and inequalities (e.g., between marginal and total productions) sharp enough to detect hidden dynamics or irreversibility (Igarashi, 2022).
• The framework even extends to geometric and semantic analyses of time series, where local configuration spectra (via 13 geometric shapes) yield entropic measures correlated with dynamical regime changes (e.g., epileptic seizures) (Majumdar et al., 2018).

Because these diverse domains all manifest discrete entropy spectra, the underlying tools—be it scaling sequences, explicit polynomials, functional calculus for stochastic processes, or geometric local entropy—create an interconnected theory wherein discrete structure shapes the spectrum of unpredictability, randomness, and complexity.

7. Summary Table of Contexts and Key Features

Context	Governing Formula or Principle	Spectral Feature
Orthogonal polynomials	$\mathcal S_{n,j}$, closed forms with GCD, zeta terms	Peaks/dips from arithmetic
Dynamical systems (discrete spectrum)	Scaling entropy, sequence entropy, factor maps to odometers	Zero entropy, spectral rigidity
Poisson/Geometric/CLT regimes	Discrete EPI, entropy power $N_d[X]=e^{2H[X]}/2\pi e$	Max-entropy/information flow
Combinatorial optimization/sampling	Local entropy of flat modes, EDLP functional	Mode volume, robustness
Stochastic processes (time series)	Spectral decomposition, differential entropy rate	Frequency-dependent complexity
Source coding (singular variables)	Information dimension and $\varepsilon$-entropy	Non-integer-dimension scaling

The discrete entropy spectrum, thus, is a multi-faceted invariant capturing how entropy—in all its forms—varies, concentrates, and fluctuates in discrete structures, bridging the combinatorial, algebraic, probabilistic, and physical aspects of randomness and information.