Computational Relative Entropy
- Computational relative entropy is a family of methods extending traditional KL divergence by incorporating resource constraints and algorithmic limitations.
- It employs numerical estimators, semidefinite programming, and variational techniques to analyze statistical, quantum, and field-theoretic data.
- By linking operational distinguishability with computational feasibility, it offers practical insights for hypothesis testing, channel capacity, and cryptographic protocols.
Computational relative entropy is not a single formalism but a family of computational viewpoints on relative entropy and related divergences. Across statistics, quantum information, field theory, holography, cryptography, and compression, it denotes either the explicit computation of relative entropy from data, operators, or path integrals, or a complexity-aware replacement of ordinary relative entropy in which distinguishability is restricted by computational resources. In its standard form, relative entropy is the Kullback–Leibler divergence or its quantum analogue,
while computational variants modify either the admissible tests, the optimization domain, or the numerical procedure used to evaluate these quantities (0808.4111, Koßmann et al., 2024, Meyer et al., 24 Sep 2025).
1. Definitions, scope, and basic distinctions
The classical discrete form used throughout statistics is
with finiteness iff is absolutely continuous with respect to . In finite-dimensional quantum theory the corresponding quantity is
and the max-relative entropy is
For quantum channels, the stabilized channel relative entropy is
with the Choi matrix (Koßmann et al., 2024).
A separate usage appears in complexity theory and cryptography. There, computational relative entropy does not merely mean “relative entropy computed numerically”; it is an operational quantity defined under bounded resources. One such formulation is the optimal type-II error exponent in asymmetric hypothesis testing when only polynomially many copies and polynomially many quantum gates are allowed:
where is the complexity-limited hypothesis-testing relative entropy (Meyer et al., 24 Sep 2025).
A third usage is methodological: the numerical estimation of KL or quantum relative entropy from samples, covariance operators, semidefinite relaxations, or path integrals. This includes trace-log formulas for Gaussian fields, MCMC-based evidence estimators, semidefinite approximations of matrix logarithms, quadrature-based quantum algorithms, and Euclidean replica constructions in CFT (Schröfl et al., 2023, Mehrabi et al., 2019, Fawzi et al., 2017, Lu et al., 13 Jan 2025, Lashkari, 2014).
This suggests that “computational relative entropy” has at least three stable meanings: resource-bounded distinguishability, computable surrogate formulations, and practical numerical algorithms. A plausible implication is that the term now functions as a unifying interface between operational information measures and the algorithmic constraints of specific domains.
2. Complexity-constrained relative entropy
In the complexity-theoretic formulation, the central object is not itself but the best asymptotic discrimination rate achievable by restricted tests. Measurements of gate complexity at most are modeled by effects in , and the corresponding single-shot quantity is
The resulting regularized quantity satisfies, up to negligible terms,
is additive under polynomial tensor powers, obeys data processing for poly-complexity channels, and has a computational Stein’s lemma identifying it with a computationally measured two-outcome relative entropy under a polynomial bound on measured max-relative entropy (Meyer et al., 24 Sep 2025).
The same framework yields computational analogues of Pinsker and Bretagnolle–Huber inequalities:
and a computational smoothing property,
so computationally indistinguishable states have equivalent complexity-aware information measures (Meyer et al., 24 Sep 2025).
Earlier cryptographic work developed related KL-based hardness notions for search problems. Hardness in relative entropy is defined through the KL divergence between the joint law of coins and outputs produced by an online generator and a simulated alternative:
If a distributional search problem is -hard, then this divergence is at least , and the same chain-rule structure yields next-block pseudoentropy and next-block inaccessible entropy (Agrawal et al., 2019).
Convex analysis gives a complementary perspective. A restricted or Editor's term “test-limited” relative entropy arises by restricting the variational characterization of KL to a class of efficient functions. In the paper’s convex-dual viewpoint,
and computational entropy becomes a separation problem between a target distribution and a convex entropy superlevel set, tested only by efficient distinguishers. For metric min-entropy, deterministic boolean and deterministic real-valued distinguishers coincide, whereas for finite Rényi order randomized or real-valued distinguishers are strictly stronger than deterministic boolean ones (Skórski, 2013).
These results establish that computational relative entropy is not merely an approximation to ordinary relative entropy. It is a distinct operational invariant whose value can collapse to zero even when the unbounded relative entropy is infinite, as in pseudorandom or one-way-function-based constructions (Meyer et al., 24 Sep 2025, Agrawal et al., 2019).
3. Convex, variational, and semidefinite formulations
A major line of work treats relative entropy as an optimization primitive. In finite-dimensional quantum information, many quantities reduce to minimization or maximization of over convex sets. Examples include the relative entropy of entanglement,
its PPT relaxation,
the Holevo information written as
and channel optimization problems expressed via conditional entropy or relative entropy of recovery (Fawzi et al., 2017).
The numerical obstacle is the matrix logarithm. A unified SDP approach replaces by semidefinite approximations built from the integral representation
followed by high-order quadrature and scaling-and-squaring. In the implementation described in CvxQuad, quantum_entr, trace_logm, quantum_rel_entr, quantum_cond_entr, and op_rel_entr expose these approximations to off-the-shelf semidefinite solvers. The paper reports absolute errors around for simple cq channels with default and uses the framework to compute relative entropy of entanglement, entanglement-assisted capacity, degradable-channel quantum capacity, and relative entropy of recovery (Fawzi et al., 2017).
For channel relative entropy, a distinct semidefinite framework starts from the Frenkel–Jencová operator-integral representation
Substituting and
produces lower- and upper-bound SDPs that sandwich with any desired precision. The lower-bound SDP optimizes over and auxiliary variables , while the upper-bound SDP uses variables , , and . The number of grid points needed for an -approximation scales as
with (Koßmann et al., 2024).
A related variational strategy appears for state estimation on quantum hardware. There, and Petz Rényi divergences are reduced to weighted sums of via quadrature formulas for and , and each is estimated through a variational minimization over an operator parameterized by shallow circuits. The method uses only copies of and , an ancilla, controlled unitaries, and an extended SWAP test, with total circuit width at most $2n+1$ qubits (Lu et al., 13 Jan 2025).
Across these formulations, a common pattern emerges: relative entropy is made computationally tractable by replacing nonlinear logarithms with integral, rational, or variational surrogates that preserve convexity or monotonicity sufficiently well for optimization.
4. Statistical estimation, Gaussian theories, and sample-based computation
In classical statistics, computational relative entropy begins with direct evaluation from counts. For i.i.d. discrete data with empirical frequencies ,
with complexity . The identity
makes KL the central functional of hypothesis testing, maximum likelihood, model selection, maximum entropy reconstruction, the EM algorithm, and Markov order determination (0808.4111).
For Gaussian scalar field theories, the computation is spectral. If and the measures are equivalent, then
and in an eigenbasis of with eigenvalues ,
When only the mass changes, with common Laplacian eigenvalues ,
The paper proves that for different masses with the same classical boundary conditions, is finite in finite volume iff , while Dirichlet versus Robin boundary conditions are mutually singular for all (Schröfl et al., 2023).
The same paper treats mutual information between disjoint regions as a KL divergence between the joint Gaussian field and the product of restrictions:
If , then the mutual information is finite; it satisfies an area law,
and becomes infinite for touching regions. In one dimension,
This is a fully explicit field-theoretic computation, but it is also a numerical recipe: discretize the Laplacian, build covariance blocks, and compute log-determinants (Schröfl et al., 2023).
Sample-based Bayesian estimation gives another computational regime. For posterior-to-prior information gain,
so one needs posterior samples, likelihood evaluations, and an evidence estimate. The paper estimates the evidence via a k-nearest-neighbor density-ratio method:
then forms
In the linear Gaussian model, the reported relative error is below for sample size larger than (Mehrabi et al., 2019).
Random-state calculations occupy an intermediate position between exact formulas and asymptotic estimation. For reduced states from the Wishart ensemble with aspect ratio , the large- average relative entropy is
derived from replica limits of hypergeometric expressions for and . This provides a closed-form estimator for chaotic many-body eigenstates and black-hole microstate distinguishability (Kudler-Flam, 2021).
5. Quantum many-body, field-theoretic, and holographic computations
In conformal and algebraic quantum field theory, computational relative entropy often means an explicit evaluation of Araki or Umegaki relative entropy through modular theory, replica limits, or quasi-free formulas.
For 1+1-dimensional CFT, the Euclidean path-integral construction computes sandwiched Rényi relative entropies via replicated manifolds. For a holomorphic vertex operator on a circle, the replicated correlator yields
and analytic continuation gives
For thermal states at temperatures and on an interval of length , with ,
The relative entropy is UV-finite because the short-distance divergences cancel in the replica ratios (Lashkari, 2014).
For chiral free fermions, Araki relative entropy reduces to a one-particle trace formula for quasi-free CAR states. The mutual information of two disjoint intervals can be written as
with built from and its compressions to the one-particle subspaces. The exact answer for free chiral fermions is
and for two intervals this becomes the standard cross-ratio logarithm. For finite-index subnets, the short-distance asymptotics picks up an additive constant controlled by the global dimension (Longo et al., 2017).
For fermionic QFT on globally hyperbolic spacetimes, the self-dual CAR framework yields a genuinely modular-theoretic formula. If is a faithful quasifree state, is the one-particle modular flow, and is a unitary field excitation with and , then
For a -KMS state on an ultrastatic spacetime,
and in a mode basis
This is the fermionic analogue of coherent-state relative entropy in free scalar QFT (Galanda, 2022).
In holography, relative entropy becomes geometrizable. For a boundary region ,
and the central semiclassical statement is
The modular Hamiltonian maps to bulk operators as
For ball-shaped regions in the vacuum,
and the bulk relative entropy reduces, in local modular-Hamiltonian cases, to canonical energy. The 2013 holographic analysis established at first order and positivity at second order for broad classes of perturbations, while the 2015 analysis promoted this to equality between boundary and bulk relative entropy inside the entanglement wedge (Blanco et al., 2013, Jafferis et al., 2015).
6. Communication, compression, and algorithmic applications
In relative entropy coding, computational relative entropy becomes a design principle for exact one-shot compression with shared randomness. A channel simulation algorithm communicates a sample using a code whose expected length is lower-bounded by mutual information:
An exact relative entropy coding algorithm attains
To sharpen the lower bound, the thesis introduces the Channel Simulation Divergence and proves
The computational complexity of selection samplers is controlled by :
A* sampling and Greedy Poisson Rejection Sampling attain code lengths within the optimal overhead (Flamich, 19 Jun 2025).
The same work gives a concrete continuous-space recipe. For Gaussian channels, if and , then
and A* or GPRS yields
This is presented as a quantization-free compression method that integrates naturally with reparameterizable probabilistic models (Flamich, 19 Jun 2025).
Bayesian implicit neural representations instantiate the same principle in learned codecs. With posterior and coding distribution , the training objective is
The empirical claim in the thesis is that REC with Bayesian INRs attains competitive or superior rate–distortion performance while using small, energy-efficient models (Flamich, 19 Jun 2025).
At a more abstract level, complete diagrammatic axiomatisations have recently been given for KL and Rényi divergences on categories of finite stochastic matrices. In this setting, KL on distributions is
and its column-wise extension to stochastic matrices is
The computational content comes from exact chain rules and max laws encoded as quantitative equations on string diagrams, yielding compositional calculation rules for probabilistic circuits (Sarkis et al., 4 Mar 2026).
This suggests that algorithmic applications of relative entropy now extend well beyond inference and discrimination. They include source coding on continuous spaces, compositional reasoning for stochastic kernels, and operational rates for privacy- or perception-constrained generative mechanisms.
7. Structural themes, misconceptions, and limitations
A common misconception is that computational relative entropy is synonymous with approximate KL evaluation. The literature instead supports two distinct claims. First, there are practical algorithms for estimating ordinary relative entropy from samples, matrices, channels, or path integrals. Second, there are genuinely new complexity-aware divergences whose operational content differs sharply from the unbounded theory (Mehrabi et al., 2019, Fawzi et al., 2017, Meyer et al., 24 Sep 2025).
Another misconception is that computational restrictions merely perturb relative entropy by small errors. Complexity-theoretic separations show otherwise. Under post-quantum one-way functions, there exist efficiently preparable classical states with computational relative entropy asymptotically zero while the ordinary relative entropy is infinite. Under classically-hard but quantumly-easy one-way functions, one obtains a quantum–classical gap: quantum observers can distinguish states with positive computational exponent whereas classical observers cannot (Meyer et al., 24 Sep 2025).
In numerical work, limitations are domain-specific. Gaussian-field formulas require equivalence of measures or regularized Fredholm determinants; in the different-mass KL divergence diverges. MCMC estimators suffer from curse-of-dimensionality effects in kNN density estimation and from poor overlap between posteriors. SDP methods scale poorly with Hilbert-space dimension because matrix-log approximations introduce lifted blocks of size on the order of . Channel-relative-entropy SDPs depend on , so ill-conditioned channel pairs require finer grids. Variational quantum estimators require expressive ansätze and sufficient shot budgets. Replica constructions in CFT are highly explicit in 1+1 dimensions but much harder in higher dimensions or on higher-genus replica manifolds (Schröfl et al., 2023, Mehrabi et al., 2019, Fawzi et al., 2017, Koßmann et al., 2024, Lu et al., 13 Jan 2025, Lashkari, 2014).
A further subtlety concerns monotonicity and cancellation. In gauge theories and gravity, local edge-mode or surface terms can change entanglement entropy, but relative entropy is often insensitive because those local contributions cancel between the two states being compared. This cancellation is central both in bulk–boundary relative entropy and in gauge-theoretic subregion constructions (Jafferis et al., 2015).
Taken together, these works define computational relative entropy as a broad research program rather than a single formula. Its unifying principle is operational: distinguishability is quantified not only by states and hypotheses, but also by what can be computed, optimized, or encoded under concrete structural constraints.