Minimal Random Coding

• Minimal Random Coding is a set of techniques that minimize redundancy, communication cost, and representation size using stochastic constructions and statistical regularization.
• These methods employ random ensembles, constraint-based codeword selection, and distributional objectives to approach information-theoretic limits in error exponents and cost efficiency.
• Applications span channel coding, network coding, model compression (via bits-back coding), and random projection quantization, enhancing performance in communication and learning systems.

Minimal random coding encompasses a range of coding-theoretic techniques that optimize encoding efficiency—minimizing codeword redundancy, communication cost, and representation size—using stochastic constructions and statistical regularization. These methods are grounded in information theory, combinatorial coding, and empirical optimization, and span applications from channel coding to practical supervised learning, model compression, network coding, and cost optimization in constant-weight codes. Minimality is achieved by leveraging random ensembles, constraint-based generation, and distributional coding objectives so that coding overhead and error exponents approach information-theoretic limits.

1. Principles of Minimal Random Code Construction

Minimal random coding schemes employ random selection or sampling with additional constraints to ensure that each codeword minimally overlaps or correlates with others. In classical channel coding contexts, as in the generalized Random Gilbert–Varshamov (RGV) ensemble (Somekh-Baruch et al., 2018), codewords are constructed recursively:

• Fix a type class, i.e., all codewords share a specified empirical distribution.
• Impose a (possibly channel-dependent) distance constraint so that any new codeword is farther than a threshold $A$ from all previously chosen codewords.
• Ensure non-emptiness of feasible sets at every recursion by bounding the "ball volume"—the number of codewords within distance $A$ of any existing word.
• Recursive construction strictly preserves the minimal separation among codewords and supports rates up to the combinatorial bound induced by the distance constraint.

The approach generalizes to cost-constrained random coding for infinite alphabets, replacing type-class construction with cost-restricted sampling.

2. Achievable Error Exponents and Optimality

For transmission over a discrete memoryless channel (DMC), the error exponent quantifies the rate of exponential decay of the error probability in blocklength. Minimal random coding ensembles such as RGV yield exponents that recover and generalize the Csiszár–Körner bound (Somekh-Baruch et al., 2018), known to dominate both classical random coding and expurgated exponents:

$$E_{\mathrm{RGV}}(R,P,W,q,d,A) = \min_{V_{XX'Y} \in \mathcal{T}_{d,q,P}(A)} \left\{ D(V_{Y|X} \Vert W \mid P ) + | I_V(X;X',Y) - R |^+ \right\}$$

where:

• The constraint set $\mathcal{T}_{d,q,P}(A)$ enforces codeword distance ($d \geq A$), equality of marginals ($V_X = V_{X'} = P$), and a type-dependent decoding constraint.
• The optimal distance function $d^*(P_{XX'}) = -I_{P_{XX'}}(X;X')$ achieves universal optimality, tightly matching the Csiszár–Körner exponent for any DMC and decoder.

Additive metrics enable a dual Gallager-type representation for the exponent. For ML decoding, the Chernoff/Bhattacharyya distance recovers both random coding and expurgated bounds, unifying exponent derivations across mismatched decoders.

3. Minimal Random Code Learning and Bits-Back Coding Paradigm

Variational coding and model compression in machine learning (e.g., MIRACLE) are predicated on stochastic encoding that exploits the bits-back mechanism (Havasi et al., 2018). Instead of encoding deterministic parameters, a variational posterior $q(w)$ is learned under a prior $p(w)$, and a random sample $w^* \sim q(w)$ is encoded:

• The expected code length required is the KL divergence, $D_{KL}(q \Vert p)$.
• Constraints (Lagrangian or hard) on the KL term explicitly control the compression rate and trace the test error vs. memory Pareto frontier.
• Encoding achieves the information-theoretic lower bound to within a logarithmic additive term, exploiting the randomness in $q(w)$ so transmitted bits become recoverable at the receiver.
• Blockwise coding and shared randomness enable practical, distributed encoding of high-dimensional models.

Recent advances in Mean–KL parameterization (Lin et al., 2023) allow direct control over per-block or per-weight KL budgets, eliminating costly parameter annealing and accelerating convergence, while yielding heavier-tailed and more robust variational distributions.

4. Header Minimization in Random Linear Network Coding

In network coding contexts, minimal random coding targets header overhead reduction. The Small Set of Allowed Coefficients (SSAC) algorithm (Gligoroski et al., 2016) achieves this by:

• Constraining coded packets to combine only a small, fixed number $m \ll N$ of source packets with coefficients drawn from a minimal set $Q$ (size 2 or 3) in the ambient finite field $GF(q)$.
• Each header encodes just the active source indices and their coefficient selection, resulting in a header of length $m(\log_2|Q| + \log_2N)$, independent of $q$.
• Compression outperforms all previous schemes by a factor of 2–7, matching innovation and re-encoding requirements with orders-of-magnitude lower overhead.

This decouples network coding overhead from field size and enables efficient scalable deployment in resource-constrained wireless sensor networks.

5. Cost Minimization in Constant Weight Codes

Random constant-weight codes address trade-offs between codeword cost and reliability (Aceituno, 2021):

• Codebooks consist of length-$N$ codewords with fixed Hamming weight $a$ (relative weight $r = a/N$).
• Combinatorics (Stirling approximation) yield codebook size $\exp[N H(r)]$; under noise, the achievable rate is $G(r)$, derived by optimizing an overlap-based exponent against channel parameters.
• The minimal random-coding procedure solves:

$$\min_{r, N} C(rN, N) \quad \text{subject to} \quad \frac{1}{N}\ln |{\cal W}| < G(r)$$

where $C$ is codeword cost (e.g., linear in $a$ and $N$).

• Laplace, KL-based, and large deviation approximations make closed-form optimization tractable, yielding sharp prescriptions for code length, weight, and design that minimize overall communication cost.

6. Minimal Coding in Combinatorial and Graph Code Constructions

Extremal bounds on the number of minimal codewords in long codes reveal the power of random coding for saturating codebook entropy (Alahmadi et al., 2012):

• For binary linear $[n,k]$ codes, random generator-matrix constructions yield nearly all nonzero codewords minimal for rates $R = k/n \le 1/2$, with exponential supply ($\mu(R) \ge R$).
• Upper bounds by matroid theory are $\mu(R) \le H(R)$, tight for large rates.
• In graphical (cycle-code) cases, explicit constructions and multigraph families achieve the exact piecewise linear bound $\mu_g(R)$.
• Random coding thus emerges as both a lower and upper bound saturating strategy for minimal codeword supply across algebraic and combinatorial coding domains.

7. Minimal Coding in Random Projection and Quantization

In large-scale learning and retrieval, minimal random coding enables storage and computational economy via quantized Gaussian random projections (Li et al., 2013):

• Uniform quantization (binless, with fixed width $w$) strictly outperforms offset-based schemes for similarity estimation and linear classification.
• Nonuniform 2-bit schemes partition $\mathbb{R}$ into four intervals, providing robustness and near-optimal accuracy with minimal bits.
• Optimized variance analysis justifies 1–2 bits per projection as sufficient for most practical regimes, with collision probabilities and collision estimators derived analytically.
• Experiments on SVMs confirm negligible accuracy loss when using 2 bits versus full 32-bit admissions.

Minimal random coding here allows practitioners to match the error profile of full-precision machine learning workflows while achieving dramatic gains in transmission, storage, and runtime efficiency.

---

Minimal random coding constitutes a unified paradigm linking codebook generation, quantization, and statistical inference with information-theoretic and combinatorial optimality. It is foundational for modern advances in communication complexity, learning compression, network coding overhead reduction, and cost-efficient error-correcting code design.