Truncated Matrix Entropy: Theory & Applications

• Truncated matrix entropy is a framework that quantifies how entropy is preserved or reduced when matrices undergo deterministic or randomized truncation.
• Key methods include partial Hadamard matrices and stochastic polynomial approximations that enable efficient computation and preservation of both Rényi and von Neumann entropies.
• Practical applications span large-scale inference, compressed sensing, kernel methods, and quantum simulations, balancing computational speed with accuracy.

Truncated matrix entropy encompasses a family of results and methodologies concerning the entropy associated with matrices after the application of truncation or reduction schemes. In contemporary literature, this concept primarily appears in three distinct but related contexts: deterministic partial Hadamard matrices for discrete entropy preservation (Haghighatshoar et al., 2012), randomized polynomial and block-wise approximations for kernel matrix-based Rényi entropies (Gong et al., 2021), and the scaling of von Neumann entropy in products of truncated random unitary matrices (Beenakker, 19 Jan 2025). Each context emphasizes the behavior of entropy under structured or randomized truncation of a matrix, with results spanning from exact preservation to quantifiable loss depending on the setting.

1. Formal Definitions and Frameworks

1.1 Partial (Truncated) Hadamard Matrices

Let $N=2^n$; the $N\times N$ Hadamard matrix $$J_N = \left(\begin{smallmatrix}1 & 1 \ 1 & -1\end{smallmatrix}\right)^{\otimes n}$$. For a subset of row indices $\mathcal{K}_N$ with cardinality $m_N$, the partial (or truncated) Hadamard matrix is $P_N = (J_N)_{\mathcal K_N}$. For a discrete random vector $X^N \in \mathcal{X}^N$, entropy is $$H(X^N) = -\sum_{x \in \mathcal{X}^N}\Pr[X^N=x]\log_2 \Pr[X^N=x]$$. For continuous random vectors, differential entropy is defined analogously.

Preservation of entropy by a linear map $P_N$ is characterized by $H(X^N \mid P_NX^N) \leq N\epsilon$, or $h(Q(X^N) \mid P_NX^N) \leq N\epsilon$, where $Q$ denotes quantization.

1.2 Matrix-based Rényi Entropy

Given a normalized Gram matrix $G \in \mathbb{R}^{n \times n}$, the matrix-based Rényi entropy of order $\alpha>0$, $\alpha\neq1$, is

$$S_\alpha(G) = \frac{1}{1-\alpha} \log_2 \operatorname{tr}(G^\alpha) = \frac{1}{1-\alpha} \log_2 \left(\sum_{i=1}^n \lambda_i^\alpha\right),$$

where $\lambda_i$ are the eigenvalues of $G$.

1.3 Entropy in Products of Truncated Random Unitaries

For random matrix products $M = \tilde U_L \cdots \tilde U_1$ with each $\tilde U_j$ an $N \times N$ Haar-unitary with its first $\delta N$ rows and columns set to zero, the entropy of the normalized state $$\rho = \frac{MM^\dagger}{\operatorname{Tr}(MM^\dagger)}$$ is considered. Analysis focuses on the double-scaling limit $L, N \to \infty$ with $\tau = L\delta N / N$ fixed.

2. Main Results on Entropy Preservation under Truncation

2.1 Discrete Sources: Absorption Phenomenon (Partial Hadamard)

For discrete i.i.d. sources $X_1,\ldots,X_N$ over a finite subset of $\mathbb{Z}$, there exists a set of row indices $\mathcal{K}_N^{(\epsilon)}$ of vanishing fraction $|\mathcal{K}_N^{(\epsilon)}|/N \to 0$, such that

$H(X^N \mid P_N X^N) \leq N\epsilon,$

where $P_N$ selects only the rows in $\mathcal{K}_N^{(\epsilon)}$. Thus, most of the system's entropy can be "absorbed" by a vanishing subset of Hadamard rows as $N \to \infty$, and the entropy of the projected system is preserved up to $o(N)$.

2.2 Continuous Sources: No Sublinear Truncation (Partial Hadamard)

For continuous i.i.d. sources, any sublinear selection results in zero entropy preservation in a vanishing-error regime. Explicitly, if after quantization $Q$,

$H(Q(X^N) \mid P_N X^N) \leq N\epsilon,$

then necessarily $\limsup_{N\to\infty} m_N/N \to 1$ as $\epsilon \to 0$. No deterministic partial Hadamard matrix can compress a continuous source while preserving its differential entropy below rate 1.

2.3 Truncation in Kernel Matrices: Stochastic Polynomial and Block Approximations

Exact computation for $\operatorname{tr}(G^\alpha)$ requires $O(n^3)$ operations for full eigendecomposition. Stochastic polynomial approximations (Taylor, Chebyshev, Lanczos) yield unbiased, truncated estimates requiring only matrix-vector multiplications:

• Integer-order $\alpha$: $O(\alpha s \operatorname{nnz}(G))$
• Fractional-order $\alpha$: Taylor ($O(ms \operatorname{nnz}(G))$), Chebyshev ($O(ms\operatorname{nnz}(G))$), Lanczos ($O(ms\operatorname{nnz}(G))$, but higher storage).

Truncation in either degree $m$ or block structure (approximating $G$ as $\hat G$ with off-diagonal block SVDs) provides speedup with quantifiable bounds on entropy error.

2.4 Entropy Reduction in Truncated Random Unitary Products

The moments and entropy of $$\rho = \frac{MM^\dagger}{\operatorname{Tr}(MM^\dagger)}$$ obey, for $\tau = L\delta N/N$,

$$\langle \sigma^{2p} \rangle = \frac{1}{\Gamma(p)} e^{-p\tau} G_p(\tau), \;\; G_p(\tau) = (1-\tau)(p-1)! \sum_{k=0}^{p-1} \frac{(p\tau)^k}{k!} + \tau^p p^{p-1}$$

and

$${\cal S} = \ln N - \ln \tau + e^\tau(\tau - 1) \Gamma(0, \tau) - \gamma_E.$$

3. Proof Techniques and Key Inequalities

3.1 Polar Code Martingale Argument (Partial Hadamard)

Define a martingale $(I_n, \mathcal{F}_n)$ with $I_n = H(Y_{[w]_n} \mid \mathcal{F}_{n-1})$ and filtration by Kronecker tree bits. For discrete sources, the martingale converges almost surely—when conditional entropy is above a threshold, discrete EPI guarantees a minimal decrement until all nontrivial entropy is "polarized" to a vanishing subset.

3.2 Entropy Power Inequality over $\mathbb{Z}$

For any integer-valued distribution $p$, the discrete EPI

$H(p * p) - H(p) \geq g(H(p))$

with $g(\cdot)$ strictly increasing, underpins step size in the martingale, limiting the persistence of non-negligible conditional entropy.

3.3 Stochastic Trace Estimator and Polynomial Approximation (Matrix-based Rényi)

The trace is replaced by $(1/s)\sum_{i=1}^s v_i^T f(G)v_i$, where $f(G) = G^\alpha$ is approximated by polynomials (Taylor, Chebyshev) or quadrature via Lanczos tridiagonalization. Random probe count $s = O(1/\epsilon^2 \log(1/\delta))$ and polynomial degree $m$ (bounded by condition number and required accuracy) suffice to control error.

3.4 Analytical Continuation and Double-scaling Limit (Random Unitaries)

The singular value moments are derived by recursive averaging over Haar measure and allow analytic continuation to obtain the entropy via the Rényi family. The critical behavior is extracted in the $\tau \ll 1$ (linear) and $\tau \gg 1$ (logarithmic) regimes.

4. Regimes of Truncation and Entropy Scaling

Setting	Regime / Condition	Entropy Behavior / Bound
Discrete (Partial Hadamard)	$m_N/N \to 0$	$H(P_N X^N) = H(X^N) + o(N)$
Continuous (Partial Hadamard)	Any $m_N<N$	No entropy preservation in limit
Matrix-based Rényi (Randomized approx.)	$s=50\text{–}200$, degree $m$	Sub-percent MRE, polynomial time
Random Unitary Products	$\tau < 1$	${\cal S} \simeq \ln N - \tau$
Random Unitary Products	$\tau > 1$	${\cal S} \simeq \ln N - \ln \tau + (1-\gamma_E)$

The crossover at $\tau = 1$ marks a phase transition: for $\tau<1$, a finite subspace remains untruncated; for $\tau>1$, all singular values are strictly subunit, and entropy exhibits a shift governed by the incomplete gamma function.

5. Computational and Applied Implications

5.1 Complexity Advantages

Partial Hadamard truncation enables $O(N\log N)$ encoding/decoding via the fast Walsh–Hadamard transform or divide-and-conquer polar decoding (Haghighatshoar et al., 2012). Block low-rank and stochastic polynomial schemes reduce the $O(n^3)$ eigendecomposition in kernel entropy computation to as low as $O(n^2/c + nck)$.

5.2 Empirical Performance: Speed-Accuracy Trade-off

For matrix-based Rényi entropy

• Integer $\alpha$: Stochastic trace with $s=100$ achieves $<0.2\%$ mean relative error for $n=10^4$ in seconds (full SVD: $220$ s).
• Taylor/Chebyshev polynomials converge in 4–6 s for sub-$0.5\%$ error.
• Block low-rank approximation (e.g., $c=10\text{–}20$, $k=50\text{–}100$) doubles speedup with negligible error increase (Gong et al., 2021).

In real tasks, such as the Information Bottleneck on CIFAR-10 ($\alpha=2$), 5× to 10× speedups are observed with no significant loss in classification accuracy.

5.3 Connections to Information Dimension and Quantum Measurement

Discrete sources with zero Rényi information dimension allow vanishing-rate entropy-preserving measurements, whereas continuous sources, with unit information dimension, require full-rate (no compression), matching universality results in analog compression (Haghighatshoar et al., 2012). In quantum information, the phase transition in entropy from truncations of random unitary products models purification in monitored quantum circuits, connecting subspace geometry (via the von Neumann–Halperin theorem) to entropy reduction (Beenakker, 19 Jan 2025).

6. Broader Significance and Limitations

Truncated matrix entropy provides rigorous foundations for understanding entropy reduction and preservation under deterministic and randomized truncations in both classical and quantum settings. For discrete systems, strong absorption allows extreme compression; for continuous and high-dimensional quantum systems, strict impossibility bounds arise. The interplay of fast algorithms (Hadamard, block low-rank, randomized mat-vecs) with precise error control underscores the practical relevance for large-scale inference, compressed sensing, kernel methods, and quantum simulations. The universality of the scaling laws and phase transitions establishes truncated matrix entropy as a central notion in modern information theory and matrix analysis, with ongoing significance in empirical and theoretical research.