---
title: Relative Entropy Coding (REC)
url: https://www.emergentmind.com/topics/relative-entropy-coding-rec
type: topic
---

# Relative Entropy Coding (REC)

Searching arXiv for the cited REC literature to ground the article and confirm coverage.
Relative Entropy Coding (REC) denotes a family of coding frameworks in which relative entropy governs either robustness to source uncertainty or the expected length of a stochastic code. In one line of work, REC studies lossless or rate–distortion coding when the true source law is only known to lie in a relative-entropy ball around a nominal distribution. In another, now more common, line of work, REC is a channel-simulation problem: given \(X\sim P_X\), the encoder and decoder share randomness \(Z\), transmit a finite binary string, and require \(\dec(\enc(X,Z),Z)\sim P_{Y\mid X}\) exactly. Across both usages, the central object is a KL-type mismatch between a nominal law and a target law, but the operational questions differ: minimax robustness in the former, exact stochastic simulation at minimum expected rate in the latter [1102.2678; 2506.16309; 2602.07635].

## 1. Formal definitions and problem classes

In the uncertainty-ball formulation, the source alphabet is \(X=\{1,\dots,M\}\), \(P_0=(p_0(1),\dots,p_0(M))\) is a known nominal distribution, and the true law \(P=(p(1),\dots,p(M))\) is only known to satisfy
\[
D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.
\]
The admissible set is the relative-entropy ball
\[
\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},
\]
and the code is specified by a length vector \(\ell=(\ell_1,\dots,\ell_M)\), either real-valued or integer-valued, satisfying the \(D\)-ary Kraft inequality \(\sum_i D^{-\ell_i}\le 1\) [1102.2678].

In the stochastic-code formulation, REC generalises classical source coding by replacing the task of encoding a fixed message with the task of encoding a random or uncertain message. A stochastic code is a triple
\[
Z:\text{common randomness},\qquad \enc:\mathcal X\times \mathcal Z\to\{0,1\}^*,\qquad \dec:\{0,1\}^*\times \mathcal Z\to \mathcal Y,
\]
with \(Z\perp X\), such that for every \(x\),
\[
\dec(\enc(x,Z),Z)\sim P_{Y\mid X=x}.
\]
Its cost is the expected binary-string length
\[
L=\mathbb E[\,|\enc(X,Z)|\,].
\]
This formulation extends naturally to continuous spaces, since \(I(X;Y)\) can be finite even when \(H(Y\mid Z)=\infty\) or ordinary entropy coding is not directly applicable [2602.07635; 2506.16309].

A common misconception is that REC is merely entropy coding for continuous variables. The stochastic-code formulation is stricter: the decoder must generate a sample with the exact prescribed conditional law \(P_{Y\mid X=x}\), not merely reproduce a quantised surrogate. The uncertainty-ball formulation is different again: there the decoder is ordinary, but the code must be robust against source mismatch measured by relative entropy [1102.2678; 2602.07635].

## 2. Minimax REC over relative-entropy balls

For uncertain discrete sources, the basic minimax objectives are:
\[
\min_{\ell}\max_{P:\,D(P\|P_0)\le R} E_P[\ell(X)],
\]
\[
r(\ell,P)=E_P[\ell(X)]-H_D(P),
\qquad
\min_{\ell}\max_{P:\,D(P\|P_0)\le R} r(\ell,P),
\]
\[
\min_{\ell}\max_{P:\,D(P\|P_0)\le R}\Bigl(E_P[\ell(X)]-H_D(P)-(\ln D)^{-1}D(P\|P_0)\Bigr),
\]
and
\[
\min_{\ell}\max_{P:\,D(P\|P_0)\le R}\max_{i\in[X]}\bigl(\ell_i+\log_D p(i)\bigr).
\]
The first is minimax average length; the second is minimax average redundancy; the third is the Gawrychowski–Gagie shifted redundancy utility; the fourth is maximal minimax pointwise redundancy [1102.2678].

For minimax average redundancy, the maximising distribution lies in a tilted family
\[
P^{(\beta)}(i)=\frac{p_0(i)\,D^{-\beta \ell_i}}{\sum_j p_0(j)\,D^{-\beta \ell_j}},
\qquad \beta>0,
\]
with \(\beta\) chosen so that \(D(P^{(\beta)}\|P_0)=R\). Substituting this tilt reduces the code-design problem to an exponential-Huffman objective. In the \(\ell\)-domain the cost is
\[
\sum_i w_i\,D^{s\ell_i},\qquad w_i=p_0(i),\qquad s>0,
\]
which is exactly the exponential Huffman problem. The optimal real-valued lengths are
\[
\ell_i^*
=
-\frac{1}{s+1}\log_D p_0(i)
+
\log_D \sum_{j=1}^M p_0(j)^{1/(s+1)},
\]
while optimal integer prefix lengths are obtained by running the ordinary \(D\)-ary Huffman algorithm on modified weights
\[
\widetilde w_i=p_0(i)^{1/(s+1)}.
\]
As \(R\to 0\), one has \(\beta\to 0\), \(s\to 0\), and \(\ell_i^*\to -\log_D p_0(i)\), recovering the ordinary Shannon code for \(P_0\) [1102.2678].

The same uncertainty model extends to rate–distortion. With nominal source law \(\mu\), distortion \(\rho\), and uncertainty class \(M_R(\mu)=\{\mu': H(\mu'\|\mu)\le R\}\), the robust rate–distortion problem is
\[
R_+(D)=\inf_{q\in Q(D)}\sup_{\mu'\in M_R(\mu)} I(\mu';q),
\qquad
R_-(D)=\sup_{\mu'\in M_R(\mu)}\inf_{q\in Q(D)} I(\mu';q),
\]
and under mild compactness and convexity assumptions these coincide:
\[
R^*(D,R)=R_+(D)=R_-(D).
\]
The corresponding saddle-point representation is
\[
R^*(D,R)=\min_{\lambda\ge 0}\max_{s\le 0}
\left\{
sD+\lambda R+\lambda\log\int_A [Z_s(x)]^{-1/\lambda}\,\mu(dx)
\right\},
\]
with
\[
Z_s(x)=\int_{\hat A} e^{s\rho(x,y)}\,\nu^*(dy).
\]
At the saddle \((s^*,\lambda^*)\),
\[
q^*(x,dy)=\frac{e^{s^*\rho(x,y)}\nu^*(dy)}{Z_{s^*}(x)},
\qquad
\mu^*(dx)=\frac{[Z_{s^*}(x)]^{-\lambda^*}\mu(dx)}{\int_A [Z_{s^*}(u)]^{-\lambda^*}\mu(du)},
\]
and \(R\to 0\) reduces the expression to the classical rate–distortion function [1305.1230].

## 3. Stochastic REC and one-shot information limits

In channel-simulation REC, a standard lower bound is
\[
I(X;Y)\le H(Y\mid Z)\le \mathbb E[\,|\enc(X,Z)|\,].
\]
Classical constructions show that the lower bound is achievable within a logarithmic gap, and lower bounds show that this \(\log(I(X;Y)+1)\) gap is unavoidable in general. In particular, one standard achievability statement is
\[
H(Y\mid Z)\le I(X;Y)+\log(I(X;Y)+1)+4
\]
bits [2604.23076].

Recent work argues that mutual information is not the exact one-shot operational quantity for REC. The refinement is based on a width function and channel-simulation divergence. For \(P\ll Q\), the width at level \(\ell\ge 0\) is
\[
w_Q(\ell)=Q\{x:dP/dQ(x)\ge \ell\},
\]
and the channel-simulation divergence is
\[
D_{\mathrm{CS}}(P\|Q)= -\int_0^\infty w_Q(\ell)\log w_Q(\ell)\,d\ell.
\]
The associated functional information is
\[
I_F(X\to Y)
=
\mathbb E_Y\Bigl[D_{\mathrm{CS}}(P_{X\mid Y}\|P_X)\Bigr].
\]
One always has
\[
I(X;Y)\le I_F(X\to Y)\le I(X;Y)+\log(I(X;Y)+1)+1,
\]
so \(I_F\) refines \(I\) without exceeding it by more than a logarithmic term [2604.23076].

This refinement also appears in the broader CSD formalism, where the communication lower bound is expressed directly in terms of channel-simulation divergence rather than mutual information alone. In that formulation,
\[
\mathrm{CSD}(Q\|P)
=
-\int_0^\infty w_P(h)\ln w_P(h)\,dh,
\]
and any channel-simulation or REC scheme must satisfy
\[
\mathrm{CSD}\bigl(P_{\widetilde M\mid M}\|P_{\widetilde M}\bigr)\le \mathbb E[\,|\enc|\,].
\]
A plausible implication is that modern REC theory treats KL divergence as a first-order descriptor and width-based quantities as the sharp one-shot descriptors [2506.16309].

## 4. Algorithmic constructions and computational barriers

The canonical exact REC primitives are rejection sampling and Poisson/A*-type selection samplers. In the rejection-sampling code, proposals \((Y_k,U_k)\) are drawn from \(P_Y\times \mathrm{Unif}(0,1)\), one accepts the first \(k\) with \(U_k\le r_x(Y_k)/M_x\), where \(r_x(y)=dP_{Y\mid X=x}/dP_Y(y)\), and transmits an integer code for the accepted index \(k\). Its rate is controlled by \(\log \|r_x\|_\infty\), not directly by \(I(X;Y)\), so it is optimal only in special cases. In the Poisson functional representation, one draws \(Y_n\sim P\) with \(\mathrm{Exp}(1)\) arrival increments and selects
\[
N^*=\arg\min_{n\ge 1}\frac{T_n}{q(Y_n)/p(Y_n)},
\]
which ensures \(Y_{N^*}\sim Q\) [2602.07635; 2405.12203].

The main computational obstacle is that generic REC can be exponentially slow. In the Poisson functional representation analysis,
\[
\mathbb E[N^*]\approx \mathbb E_{X\sim Q}[p(X)/q(X)] = 2^{D_{\mathrm{KL}}[Q\|P]},
\]
so encoding time is on the order of
\[
O\bigl(2^{D_{\mathrm{KL}}[Q\|P]}\bigr).
\]
At a more abstract level, any selection sampler must satisfy
\[
\mathbb E[N]\ge \exp\!\bigl(D_\infty(Q\|P)\bigr),
\qquad
D_\infty(Q\|P)=\ln \esssup \frac{dQ}{dP},
\]
which is a general-purpose runtime barrier [2405.12203; 2506.16309].

Several algorithmic lines attack this barrier by exploiting structure. For continuous distributions over \(\mathbb R\) with unimodal density ratio, AS* coding has \(\mathcal O(D_\infty[Q\|P])\) expected runtime, AD* has the same behaviour empirically, and both achieve expected codelength \(\mathcal O(D_{\mathrm{KL}}[Q\|P])\). The approximate depth-limited variant DAD* fixes the code length and, in conjunction with the IsoKL VAE parameterisation, amortises per-latent overhead in VAE compression pipelines [2201.12857].

Greedy Rejection Coding (GRC) generalises rejection-based REC to arbitrary partitioning processes. Under sample-splitting on \(\mathbb R\) with unimodal density ratio \(dQ/dP\), GRCS has expected runtime upper bounded by
\[
\beta D_{\mathrm{KL}}[Q\|P]+O(1),
\qquad
\beta=\frac{2}{\log(4/3)}\approx 4.82,
\]
and its expected codelength is optimal:
\[
H[X\mid Z]\le D_{\mathrm{KL}}(Q\|P)+2\log(D_{\mathrm{KL}}(Q\|P)+1)+O(1).
\]
Under the same assumptions, GRCD is conjectured and experimentally observed to satisfy
\[
E[\text{runtime}] \le D_\infty(Q\|P)+O(1),
\qquad
H[X\mid Z]=D_{\mathrm{KL}}(Q\|P)+O(1)
\]
[2309.15746].

A different acceleration strategy is space partitioning. The support of \(P\) is partitioned into \(J\) bins \(B_1,\dots,B_J\) with \(P(B_j)=1/J\), one chooses bin weights \(\pi\), and defines an induced prior
\[
p'(x)=\sum_j \mathbf 1\{x\in B_j\}\,J\,\pi(j)\,p(x).
\]
For exact PFR, the recommended choice is
\[
\pi(j)\propto \sup_{x\in B_j}\frac{q(x)}{p(x)};
\]
for approximate ORC, it is
\[
\pi(j)=Q(B_j).
\]
The resulting expected draws satisfy
\[
\mathbb E[\#\text{ draws}]
\approx
2^{D_\infty[Q\|P']}
\approx
2^{D_\infty[Q\|P]-\log_2 J},
\]
that is, an exponential speedup factor \(J\) [2405.12203].

## 5. Singular channels and asymptotic refinements

A singular channel is one for which there exists a measurable \(g:\mathcal Y\to \mathbb R_+\) such that
\[
\frac{dP_{Y\mid X=x}}{dP_Y}(y)=g(y)
\qquad P_{X,Y}\text{-a.s.}
\]
Equivalently, the likelihood ratio depends only on \(y\), not on \(x\). Canonical examples given in the literature are the additive uniform noise channel \(Y=X+U\), \(U\sim \mathrm{Unif}(-1,1)\), and the binary erasure channel [2604.06055].

Singularity is the regime in which the usual logarithmic redundancy can collapse. For singular channels, one has
\[
I_F(X\to Y)=I(X;Y)
\]
exactly, and the ring-toss code achieves
\[
H(Y\mid Z)\le I(X;Y)+\log e < I(X;Y)+1.45.
\]
In general, the same construction yields
\[
H(Y\mid Z)
\le
I_F(X\to Y)+\log e
<
I(X;Y)+\log(I(X;Y)+1)+2.45.
\]
This is the tightest one-shot mutual-information-based upper bound stated in the cited material, and it establishes that rejection sampling is optimal for REC when the rate is measured by functional information [2604.23076].

The asymptotic picture is correspondingly sharp. For nonsingular channels satisfying mild moment conditions,
\[
I_F(X^n\to Y^n)=nI(X;Y)+\tfrac12\log n+O(1),
\]
so the logarithmic redundancy coefficient is \(1/2\). For singular channels,
\[
I_F(X^n\to Y^n)=nI(X;Y)
\]
exactly, so the redundancy vanishes asymptotically in that scale [2604.23076].

Bits-Back Rejection Sampling (BBRS) gives an implementable singular-channel construction with the same asymptotic efficiency as Sriramu and Wagner’s sampler. Its one-shot rate bound is
\[
\mathbb E[|\enc(X,Z)|]
\le
I[X;Y]+\log(H[\Gamma]+1)+2\Delta+O(1),
\]
where \(\Gamma\) is a quantised log-density ratio. For i.i.d. product channels, the analysis yields
\[
E[|\enc(X^n,Z)|]\le nI[X;Y]+o(\log n),
\]
hence the asymptotic logarithmic redundancy
\[
R^{\log}=0
\]
for singular channels [2604.06055].

## 6. Practical systems and applications

REC has been adopted most visibly in learned compression. A direct latent-space implementation is iREC, which encodes a single latent sample \(z\sim q(z\mid x)\) relative to a prior \(p(z)\) using a shared random stream of prior samples and beam search over auxiliary blocks. Its expected latent codelength is designed to satisfy
\[
E[\ell(z)]\approx \mathrm{KL}(q(z\mid x)\|p(z)),
\]
so that with residual coding the total expected description length is close to the negative ELBO. On Cifar10, ImageNet32, and Kodak, iREC outperforms all single-image bits-back variants reported in the paper and comes within \(\approx 20\%\) of the ELBO; for 32\(\times\)32 images, encoding speed is reported as \(\sim 1\) min per image, while decoding requires only RNG plus lookup and is real-time [2010.01185].

A* and GRC variants were developed partly to make REC practical for continuous-latent VAEs. On MNIST, A* coding with IKVAE and the depth-limited DAD* approximation yields one-shot lossless compression near the theoretically optimal limit. In the 50-latent IKVAE setting, AD* gives \(1.69\) bpp while DAD* gives \(1.60\) bpp, and the overhead for signalling code lengths drops from \(\approx 195\) bits for AD* to \(5\) bits for DAD* [2201.12857].

Space partitioning further improves practicability in neural compression. In the reported experiments, REC with space partitioning handles \(D_{\mathrm{KL}}[Q\|P]\) about three times greater than what previous methods can manage, and reduces the bitrate by approximately \(5\)–\(15\%\) in VAE-based lossless compression on MNIST and INR-based lossy compression on CIFAR-10, compared to previous methods. In the CIFAR-10 RECOMBINER setting, the achieved rates lie within \(2\%\) of the theoretical REC bound \(I+\log(I+1)+O(1)\) [2405.12203].

REC also supports privacy-preserving communication. In DP-REC, the sender clips the client update, forms Gaussian target and prior laws, draws \(K=2^b\) shared prior-noise vectors, computes importance weights
\[
\alpha_k = \frac{q_\phi(w_k)}{p_\theta(w_k)},
\qquad
\pi_k = \frac{\alpha_k}{\sum_j \alpha_j},
\]
samples an index \(k^*\sim \pi\), and transmits \(k^*\) together with the seed. For Gaussian \(p,q\), the Rényi divergence satisfies
\[
D_\lambda(\mathcal N(\Delta,\sigma^2 I)\|\mathcal N(0,\sigma^2 I))
=
\frac{\lambda}{2\sigma^2}\|\Delta\|_2^2
\le
\frac{\lambda C^2}{2\sigma^2},
\]
which feeds directly into privacy accounting. Reported communication gains range from \(18\times\) to \(4300\times\), and on FEMNIST the end-to-end communication is \(14\) GB versus \(259\) GB for DP-FedAvg at \(\epsilon=3\) [2111.05454].

The broader systems literature positions REC as a foundation for image, audio, video, and protein data compression using Bayesian implicit neural representations, and also for privacy-aware and realism-constrained compression. One thesis-level synthesis states that COMBINER matches or beats state-of-the-art while using \(10^2\!-\!10^4\times\) less compute, and that REC is a natural foundation for privacy-preserving or perceptual coding because it works directly with stochastic reconstruction laws rather than quantised codebooks [2506.16309; 2602.07635].

Source: https://www.emergentmind.com/topics/relative-entropy-coding-rec