Relative Entropy Coding (REC)

• Relative Entropy Coding (REC) is a framework that uses KL divergence to manage source uncertainty and enable precise channel simulation.
• It comprises two formulations—uncertainty-ball for robust lossless and rate–distortion coding, and stochastic codes for exact simulation with minimum expected lengths.
• REC underpins advanced compression systems including learned latent coding, privacy-preserving communication, and efficient VAE-based image processing.

Searching arXiv for the cited REC literature to ground the article and confirm coverage. Relative Entropy Coding (REC) denotes a family of coding frameworks in which relative entropy governs either robustness to source uncertainty or the expected length of a stochastic code. In one line of work, REC studies lossless or rate–distortion coding when the true source law is only known to lie in a relative-entropy ball around a nominal distribution. In another, now more common, line of work, REC is a channel-simulation problem: given $X\sim P_X$, the encoder and decoder share randomness $Z$, transmit a finite binary string, and require $\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$ exactly. Across both usages, the central object is a KL-type mismatch between a nominal law and a target law, but the operational questions differ: minimax robustness in the former, exact stochastic simulation at minimum expected rate in the latter [(Baer et al., 2011); (Flamich, 19 Jun 2025); (Flamich et al., 7 Feb 2026)].

1. Formal definitions and problem classes

In the uncertainty-ball formulation, the source alphabet is $X=\{1,\dots,M\}$, $P_0=(p_0(1),\dots,p_0(M))$ is a known nominal distribution, and the true law $P=(p(1),\dots,p(M))$ is only known to satisfy

$$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$

The admissible set is the relative-entropy ball

$$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$

and the code is specified by a length vector $\ell=(\ell_1,\dots,\ell_M)$, either real-valued or integer-valued, satisfying the $D$-ary Kraft inequality $Z$0 (Baer et al., 2011).

In the stochastic-code formulation, REC generalises classical source coding by replacing the task of encoding a fixed message with the task of encoding a random or uncertain message. A stochastic code is a triple

$Z$1

with $Z$2, such that for every $Z$3,

$Z$4

Its cost is the expected binary-string length

$Z$5

This formulation extends naturally to continuous spaces, since $Z$6 can be finite even when $Z$7 or ordinary entropy coding is not directly applicable (Flamich et al., 7 Feb 2026, Flamich, 19 Jun 2025).

A common misconception is that REC is merely entropy coding for continuous variables. The stochastic-code formulation is stricter: the decoder must generate a sample with the exact prescribed conditional law $Z$8, not merely reproduce a quantised surrogate. The uncertainty-ball formulation is different again: there the decoder is ordinary, but the code must be robust against source mismatch measured by relative entropy [(Baer et al., 2011); (Flamich et al., 7 Feb 2026)].

2. Minimax REC over relative-entropy balls

For uncertain discrete sources, the basic minimax objectives are: $Z$9

$\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$0

$\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$1

and

$\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$2

The first is minimax average length; the second is minimax average redundancy; the third is the Gawrychowski–Gagie shifted redundancy utility; the fourth is maximal minimax pointwise redundancy (Baer et al., 2011).

For minimax average redundancy, the maximising distribution lies in a tilted family

$\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$3

with $\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$4 chosen so that $\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$5. Substituting this tilt reduces the code-design problem to an exponential-Huffman objective. In the $\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$6-domain the cost is

$\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$7

which is exactly the exponential Huffman problem. The optimal real-valued lengths are

$\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$8

while optimal integer prefix lengths are obtained by running the ordinary $\dec(\enc(X,Z),Z)\sim P_{Y\mid X}$9-ary Huffman algorithm on modified weights

$X=\{1,\dots,M\}$0

As $X=\{1,\dots,M\}$1, one has $X=\{1,\dots,M\}$2, $X=\{1,\dots,M\}$3, and $X=\{1,\dots,M\}$4, recovering the ordinary Shannon code for $X=\{1,\dots,M\}$5 (Baer et al., 2011).

The same uncertainty model extends to rate–distortion. With nominal source law $X=\{1,\dots,M\}$6, distortion $X=\{1,\dots,M\}$7, and uncertainty class $X=\{1,\dots,M\}$8, the robust rate–distortion problem is

$X=\{1,\dots,M\}$9

and under mild compactness and convexity assumptions these coincide: $P_0=(p_0(1),\dots,p_0(M))$0 The corresponding saddle-point representation is

$P_0=(p_0(1),\dots,p_0(M))$1

with

$P_0=(p_0(1),\dots,p_0(M))$2

At the saddle $P_0=(p_0(1),\dots,p_0(M))$3,

$P_0=(p_0(1),\dots,p_0(M))$4

and $P_0=(p_0(1),\dots,p_0(M))$5 reduces the expression to the classical rate–distortion function (Rezaei et al., 2013).

3. Stochastic REC and one-shot information limits

In channel-simulation REC, a standard lower bound is

$P_0=(p_0(1),\dots,p_0(M))$6

Classical constructions show that the lower bound is achievable within a logarithmic gap, and lower bounds show that this $P_0=(p_0(1),\dots,p_0(M))$7 gap is unavoidable in general. In particular, one standard achievability statement is

$P_0=(p_0(1),\dots,p_0(M))$8

bits (Hill et al., 25 Apr 2026).

Recent work argues that mutual information is not the exact one-shot operational quantity for REC. The refinement is based on a width function and channel-simulation divergence. For $P_0=(p_0(1),\dots,p_0(M))$9, the width at level $P=(p(1),\dots,p(M))$0 is

$P=(p(1),\dots,p(M))$1

and the channel-simulation divergence is

$P=(p(1),\dots,p(M))$2

The associated functional information is

$P=(p(1),\dots,p(M))$3

One always has

$P=(p(1),\dots,p(M))$4

so $P=(p(1),\dots,p(M))$5 refines $P=(p(1),\dots,p(M))$6 without exceeding it by more than a logarithmic term (Hill et al., 25 Apr 2026).

This refinement also appears in the broader CSD formalism, where the communication lower bound is expressed directly in terms of channel-simulation divergence rather than mutual information alone. In that formulation,

$P=(p(1),\dots,p(M))$7

and any channel-simulation or REC scheme must satisfy

$P=(p(1),\dots,p(M))$8

A plausible implication is that modern REC theory treats KL divergence as a first-order descriptor and width-based quantities as the sharp one-shot descriptors (Flamich, 19 Jun 2025).

4. Algorithmic constructions and computational barriers

The canonical exact REC primitives are rejection sampling and Poisson/A*-type selection samplers. In the rejection-sampling code, proposals $P=(p(1),\dots,p(M))$9 are drawn from $$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$0, one accepts the first $$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$1 with $$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$2, where $$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$3, and transmits an integer code for the accepted index $$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$4. Its rate is controlled by $$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$5, not directly by $$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$6, so it is optimal only in special cases. In the Poisson functional representation, one draws $$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$7 with $$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$8 arrival increments and selects

$$D(P\|P_0)=\sum_{i=1}^M p(i)\ln \frac{p(i)}{p_0(i)} \le R.$$9

which ensures $$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$0 (Flamich et al., 7 Feb 2026, He et al., 2024).

The main computational obstacle is that generic REC can be exponentially slow. In the Poisson functional representation analysis,

$$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$1

so encoding time is on the order of

$$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$2

At a more abstract level, any selection sampler must satisfy

$$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$3

which is a general-purpose runtime barrier (He et al., 2024, Flamich, 19 Jun 2025).

Several algorithmic lines attack this barrier by exploiting structure. For continuous distributions over $$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$4 with unimodal density ratio, AS* coding has $$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$5 expected runtime, AD* has the same behaviour empirically, and both achieve expected codelength $$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$6. The approximate depth-limited variant DAD* fixes the code length and, in conjunction with the IsoKL VAE parameterisation, amortises per-latent overhead in VAE compression pipelines (Flamich et al., 2022).

Greedy Rejection Coding (GRC) generalises rejection-based REC to arbitrary partitioning processes. Under sample-splitting on $$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$7 with unimodal density ratio $$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$8, GRCS has expected runtime upper bounded by

$$\mathcal B(P_0,R)=\{P: D(P\|P_0)\le R,\ \sum_i p(i)=1,\ p(i)\ge 0\},$$9

and its expected codelength is optimal: $\ell=(\ell_1,\dots,\ell_M)$0 Under the same assumptions, GRCD is conjectured and experimentally observed to satisfy

$\ell=(\ell_1,\dots,\ell_M)$1

(Flamich et al., 2023).

A different acceleration strategy is space partitioning. The support of $\ell=(\ell_1,\dots,\ell_M)$2 is partitioned into $\ell=(\ell_1,\dots,\ell_M)$3 bins $\ell=(\ell_1,\dots,\ell_M)$4 with $\ell=(\ell_1,\dots,\ell_M)$5, one chooses bin weights $\ell=(\ell_1,\dots,\ell_M)$6, and defines an induced prior

$\ell=(\ell_1,\dots,\ell_M)$7

For exact PFR, the recommended choice is

$\ell=(\ell_1,\dots,\ell_M)$8

for approximate ORC, it is

$\ell=(\ell_1,\dots,\ell_M)$9

The resulting expected draws satisfy

$D$0

that is, an exponential speedup factor $D$1 (He et al., 2024).

5. Singular channels and asymptotic refinements

A singular channel is one for which there exists a measurable $D$2 such that

$D$3

Equivalently, the likelihood ratio depends only on $D$4, not on $D$5. Canonical examples given in the literature are the additive uniform noise channel $D$6, $D$7, and the binary erasure channel (Flamich et al., 7 Apr 2026).

Singularity is the regime in which the usual logarithmic redundancy can collapse. For singular channels, one has

$D$8

exactly, and the ring-toss code achieves

$D$9

In general, the same construction yields

$Z$00

This is the tightest one-shot mutual-information-based upper bound stated in the cited material, and it establishes that rejection sampling is optimal for REC when the rate is measured by functional information (Hill et al., 25 Apr 2026).

The asymptotic picture is correspondingly sharp. For nonsingular channels satisfying mild moment conditions,

$Z$01

so the logarithmic redundancy coefficient is $Z$02. For singular channels,

$Z$03

exactly, so the redundancy vanishes asymptotically in that scale (Hill et al., 25 Apr 2026).

Bits-Back Rejection Sampling (BBRS) gives an implementable singular-channel construction with the same asymptotic efficiency as Sriramu and Wagner’s sampler. Its one-shot rate bound is

$Z$04

where $Z$05 is a quantised log-density ratio. For i.i.d. product channels, the analysis yields

$Z$06

hence the asymptotic logarithmic redundancy

$Z$07

for singular channels (Flamich et al., 7 Apr 2026).

6. Practical systems and applications

REC has been adopted most visibly in learned compression. A direct latent-space implementation is iREC, which encodes a single latent sample $Z$08 relative to a prior $Z$09 using a shared random stream of prior samples and beam search over auxiliary blocks. Its expected latent codelength is designed to satisfy

$Z$10

so that with residual coding the total expected description length is close to the negative ELBO. On Cifar10, ImageNet32, and Kodak, iREC outperforms all single-image bits-back variants reported in the paper and comes within $Z$11 of the ELBO; for 32$Z$1232 images, encoding speed is reported as $Z$13 min per image, while decoding requires only RNG plus lookup and is real-time (Flamich et al., 2020).

A* and GRC variants were developed partly to make REC practical for continuous-latent VAEs. On MNIST, A* coding with IKVAE and the depth-limited DAD* approximation yields one-shot lossless compression near the theoretically optimal limit. In the 50-latent IKVAE setting, AD* gives $Z$14 bpp while DAD* gives $Z$15 bpp, and the overhead for signalling code lengths drops from $Z$16 bits for AD* to $Z$17 bits for DAD* (Flamich et al., 2022).

Space partitioning further improves practicability in neural compression. In the reported experiments, REC with space partitioning handles $Z$18 about three times greater than what previous methods can manage, and reduces the bitrate by approximately $Z$19–$Z$20 in VAE-based lossless compression on MNIST and INR-based lossy compression on CIFAR-10, compared to previous methods. In the CIFAR-10 RECOMBINER setting, the achieved rates lie within $Z$21 of the theoretical REC bound $Z$22 (He et al., 2024).

REC also supports privacy-preserving communication. In DP-REC, the sender clips the client update, forms Gaussian target and prior laws, draws $Z$23 shared prior-noise vectors, computes importance weights

$Z$24

samples an index $Z$25, and transmits $Z$26 together with the seed. For Gaussian $Z$27, the Rényi divergence satisfies

$Z$28

which feeds directly into privacy accounting. Reported communication gains range from $Z$29 to $Z$30, and on FEMNIST the end-to-end communication is $Z$31 GB versus $Z$32 GB for DP-FedAvg at $Z$33 (Triastcyn et al., 2021).

The broader systems literature positions REC as a foundation for image, audio, video, and protein data compression using Bayesian implicit neural representations, and also for privacy-aware and realism-constrained compression. One thesis-level synthesis states that COMBINER matches or beats state-of-the-art while using $Z$34 less compute, and that REC is a natural foundation for privacy-preserving or perceptual coding because it works directly with stochastic reconstruction laws rather than quantised codebooks (Flamich, 19 Jun 2025, Flamich et al., 7 Feb 2026).