--- title: Sharp Bounds on Rényi Entropy url: https://www.emergentmind.com/topics/sharp-bounds-on-renyi-entropy type: topic --- # Sharp Bounds on Rényi Entropy Rényi entropy, a one-parameter generalization of Shannon entropy, quantifies the diversity, uncertainty, or randomness of a probability distribution, with rich applications in information theory, probability, mathematical physics, and high-dimensional geometry. Sharp bounds on Rényi entropy play a critical role in establishing fundamental inequalities, continuity properties, and operational characterizations in classical, discrete, and quantum contexts. ## 1. Definition and Basic Properties For a probability density \( f \) on \( \mathbb{R}^n \) (or mass function \(p\) on a finite or countable set), the Rényi entropy of order \( \alpha > 0, \alpha \neq 1 \) is defined as \[ H_\alpha(f) = \frac{1}{1-\alpha} \log \int f(x)^\alpha dx, \] or, in discrete settings, \[ H_\alpha(p) = \frac{1}{1-\alpha} \log \sum_x p(x)^\alpha. \] The limits \(\alpha \to 1\), \(\alpha \to 0\), and \(\alpha \to \infty\) yield the Shannon entropy, Hartley entropy (support size), and min-entropy, respectively: - \( H_1(p) = -\sum_x p(x) \log p(x) \) - \( H_0(p) = \log |\{ x : p(x) > 0 \}| \) - \( H_\infty(p) = -\log \|p\|_\infty \) Rényi entropy is monotonic in its parameter: for any probability distribution, the mapping \(\alpha \mapsto H_\alpha\) is non-increasing. ## 2. Sharp Rényi Entropy Power Inequalities for \(0 < r < 1\) Let \( X \) and \( Y \) be independent random vectors in \( \mathbb{R}^n \) with log-concave densities. For \( r \in (0,1) \), the Rényi entropy power is \( N_r(X) = \exp\left(\frac{2}{n} H_r(X)\right) \). Using a sharp reverse Young's inequality and a comparison lemma due to Fradelizi, Madiman, and Wang, sharp entropy power inequalities (EPIs) for \( 0 < r < 1 \) are established, notably: \[ N_r(X+Y)^{\alpha(r)} \geq N_r(X)^{\alpha(r)} + N_r(Y)^{\alpha(r)}, \] with explicit exponent \(\alpha(r)\) and explicit constants: \[ \alpha(r) = \frac{(1-r)\ln 2}{(1+r)\ln(1+r) + r\ln(1/(4r))}. \] Equality holds asymptotically as \( r \to 1 \), recovering the Shannon EPI, while as \( r \to 0 \), \( \alpha(r) \to \infty \), corresponding to the Brunn–Minkowski regime. For the special case of uniform distributions or i.i.d. log-concave variables, further sharp constants and bounds on \(k\)-fold sums are provided, demonstrating tightness up to absolute constants. These results are sharp in the sense that matching lower bounds can be obtained by extremal exponential or Gaussian laws [1710.00800]. ## 3. Rényi Entropy Bounds for Log-Concave Measures Sharp lower bounds for Rényi entropy, particularly for symmetric log-concave random variables, play a central role in additive combinatorics, channel capacity estimation, and convex geometry. For one-dimensional, symmetric, log-concave \( X \) and any \( \alpha \in [0,1] \), the optimal bound is: \[ H_\alpha(X) \geq \log \| X \|_p + \log \left[2 (p+1)^{1/p}\right], \] where \( \| X \|_p = (\mathbb{E}|X|^p)^{1/p} \), with equality if and only if \( X \) is uniform on a symmetric interval. In the variance case (\( p = 2 \)), this becomes \[ H_\alpha(X) \geq \log \sigma + \log(2\sqrt{3}), \] with sharpness achieved only by the uniform distribution [1811.00345]. These bounds extend to consequences for the capacity of additive noise channels and lead to explicit reverse entropy power inequalities. For discrete log-concave integer-valued distributions, similar sharp reversals hold: for any \( X \) log-concave on \( \mathbb{Z} \): \[ H(X) - H_\infty(X) \leq \ln e = 1, \] with equality approached by geometric distributions as their parameter tends to zero. For all Rényi orders, a universal upper bound on the difference with the min-entropy is established [2005.10930]. ## 4. Sharp Continuity and Order-Comparison Bounds Uniform continuity and sharp comparison between Rényi entropies of different orders are fundamental for robustness, information stability, and operational interpretations. For any two finite-dimensional quantum states \( \rho \) and \( \sigma \), the sharp bound is provided in terms of their trace distance \( \delta \): \[ |H_\alpha(\rho) - H_\alpha(\sigma)| \leq f_\alpha(\delta, d), \] where for \( \alpha > 1 \), \[ f_\alpha(\delta, d) = \frac{d^{\alpha-1}}{\alpha-1} \left[1 - (1-\delta)^\alpha - (d-1)^{1-\alpha} \delta^\alpha \right] \] and similarly for \( 0 < \alpha < 1 \) [1701.02398]. Equality is attained for commuting states with prescribed eigenvalues. The relationship between Rényi entropies of distinct positive orders (\( \alpha \neq \beta \)) for finite probability vectors is characterized via extremal permutations of "one hot plus flat" (\(v_n\)) and "as many \(p\)'s as fit" (\(w_n\)) distributions. For \( p \in \Delta_n \), sharp inequalities of the form \[ B_w(\beta) \leq H_\beta(p) \leq B_v(\beta) \] hold in specific order regimes, with all bounds tight and explicit. Applications include sharp bounds for Arimoto's mutual information and Gallager's exponents [1605.00019]. For conditional Rényi entropy, sharp uniform continuity bounds for Arimoto's definition are established: \[ | H_\alpha(X|Y)_p - H_\alpha(X|Y)_q | \leq \frac{1}{1-\alpha} \log \big((1-ε)^\alpha + (|X|-1)^{1-\alpha} ε^\alpha \big) \] for total variation distance at most \( ε \), with extremizers constructed explicitly [2007.05049]. ## 5. Sharp Rényi Entropy Power Inequalities for \( \alpha > 1 \) The sharpest known (additive, multiplicative, or exponent-modified) entropy power inequalities for Rényi entropy are derived using sharpened Young's inequalities and convex optimization. For independent continuous random vectors \( X_k \) (\( k = 1, \dots, n \)), \( \alpha > 1 \): ### First Improvement (over Bobkov–Chistyakov): \[ N_\alpha(S_n) \geq c_\alpha^{(n)} \sum_{k=1}^n N_\alpha(X_k) \] with constant \[ c_\alpha^{(n)} = \alpha^{1/(\alpha-1)} \left(1 - \frac{1}{n\alpha'}\right)^{n\alpha'-1},\quad \alpha' = \frac{\alpha}{\alpha-1}. \] This sharp constant improves on all previous multiplicative REPIs [1601.06555]. ### Second (Tightest) Bound: Involves a further optimized multiplicative factor (given semi-explicitly via convex analysis or in closed-form for \( n=2 \)), always achieving strict improvement and sharpness. Limit behaviors recover the Shannon EPI (\( \alpha \to 1 \)) and give optimal constants as \( \alpha \to \infty \). Equality holds for Gaussians with proportional covariances [1601.06555]. ## 6. Sharp Bounds in Quantum Systems and Geometry For quantum many-body systems, rigorously sharp growth bounds for the instantaneous rate of Rényi entropies are established. For non-local Hamiltonians and a subsystem \(A\) of size \(|A|\) with Hilbert space dimension \(d_0^{|A|}\): \[ |S_\alpha'(t)| \leq \frac{2\alpha}{|\alpha-1|}\|H\| d_0^{2|A|}, \] whereas the von Neumann entropy (\(\alpha=1\)) scales only linearly in \(|A|\). In geometrically local models with sufficiently fast-decaying interactions, |S'_\alpha(t)| is controlled by the boundary size \(|\partial A|\) [2212.07444]. In mathematical physics, sharp upper bounds for Rényi entropy in spherically symmetric potentials, and on Lie groups with anisotropy, have been derived, with explicit optimal constants in terms of moments and group structural data [2402.10003, 1305.5334]. ## 7. Technical Methods and Extremality Structures The proofs of sharp bounds heavily utilize reverse Young's inequalities with optimal constants, comparison lemmas for log-concave measures, convex-analytic optimization, extremality and majorization arguments, information-theoretic representations, and normal (optimal) transport techniques. For many of the strongest results, the extremals (i.e., distributions attaining equality) are either uniform, exponential, or Gaussian, depending on the scenario and parameter regime. These techniques yield all previously known sharp constants and often underpin new, broader regimes (notably \(0 < r < 1\)) for generalized entropy inequalities [1710.00800, 1902.06120, 1811.00345]. --- ### Key Reference Papers | Paper Title | Area | arXiv ID | |-----------------------------------------------------------------------------------------|-----------------------|-----------------| | "On the entropy power inequality for the Rényi entropy of order [0,1]" | Sharp Rényi EPI (\(01\) | [1704.02634] | | "Sharp moment-entropy inequalities and capacity bounds for log-concave distributions" | Sharp lower bounds, applications | [1811.00345] | | "Sharp Bounds Between Two Rényi Entropies of Distinct Positive Orders" | Cross-order bounds, mutual information | [1605.00019] | | "Sharp continuity bounds for entropy and conditional entropy" | Continuity bounds | [1701.02398] | | "On Renyi Entropy Power Inequalities" | Improved multiparameter REPI | [1601.06555] | | "A tight uniform continuity bound for the Arimoto-Rényi conditional entropy ..." | Arimoto conditional Rényi bounds | [2007.05049] | | "Rigorous bounds for Renyi entropies of spherically symmetric potentials" | Quantum/geometry, sharp upper bounds | [1305.5334] | | "Bernoulli sums and Rényi entropy inequalities" | Discrete/variance, Fourier methods | [2103.00896] | | "Reversals of Rényi Entropy Inequalities under Log-Concavity" | Min-entropy, discrete log-concave | [2005.10930] | | "Rényi Bounds on Information Combining" | Conditional Rényi, polarization | [2004.14408] | | "Sharp bounds on $p$-norms for sums of independent uniform random variables, $0