Optimal Noisy Encoders: Capacity & Robustness

• The paper demonstrates that natural and permutation-based encoding schemes achieve capacity bounds in degraded broadcast channels through single-letter operations.
• Robust encoding methods integrate cost-constrained lossy computing and nonlinear, interference-aware strategies to minimize rate, distortion, and complexity.
• The study further extends to universal decoding and quantum encoding, highlighting optimal techniques that balance mutual information, resource constraints, and error exponents.

Optimal noisy encoders are encoding schemes that achieve information-theoretic optimality in the presence of channel noise, channel state uncertainty, interference, finite-precision, and other practical distortions. These schemes address capacity-achieving transmission, robustness, and complexity across diverse settings: degraded broadcast channels, multi-terminal networks, quantum channels, stochastic neural models, lossy computation with cost constraints, universal decoding under codebook noise, and privacy-preserving encoding. The concept of optimality varies but is precisely defined in terms of channel capacity, rate-distortion bounds, mutual information with side information, or specific coding-theoretic properties.

1. Capacity-Optimality and Natural Encoding Principles

In discrete degraded broadcast channels (DBC), the natural encoding (NE) scheme achieves the boundary of the capacity region by forming independent codebooks for each receiver and combining their symbols using the same single-letter operation as the channel noise model. For example, for the binary-symmetric channel (BSC), if the channel adds noise via $Y = X \oplus N$, then NE uses $X = f(X^{(1)}, X^{(2)})$ with $f$ as the channel’s “$\oplus$” operation (0811.4162). This principle generalizes: for the broadcast Z channel, NE uses the binary OR operation; for group-operation DBCs, NE reduces to group addition; for discrete multiplication DBCs, NE employs the multiplication law with “zero” as an erasing element.

Permutation encoding schemes extend NE to input-symmetric DBCs, where a subgroup of permutation matrices symmetrizes the channel input alphabet. Here the encoder combines codewords using permutation functions such that the overall input distribution is uniform and provably capacity-achieving.

Parametric expressions for the capacity region, such as

$$R_1 \leq s - H(Y|X), \quad R_2 \leq H(Z) - F^*(q, s)$$

and the definition of the conditional entropy bound function

$$F^*(q, s) = \min_{p(u,x): p_X=q, H(Y|U)=s, U\to X\to Y\to Z} H(Z|U)$$

enable explicit characterizations for families including binary-symmetric, broadcast Z, group-operation, and multiplication channels.

2. Robust and Cost-Constrained Encoding for Lossy Computing

In lossy distributed computation frameworks, the rate-distortion-cost function quantifies the minimum achievable transmission rate given joint constraints on distortion $D$ and measurement/action cost $C$ (Ahmadi et al., 2011, Ahmadi et al., 2011). The encoder must select messages and coordinate the decoder’s measurement actions (which control side information quality and incur resource cost) to minimize

$R(D,C) = \min_{P_{U|X},\,A=f(U)} I(X; U)$

subject to distortion and cost constraints. The action sequence $A^n$ can be set as a function of $U^n$ rather than greedily or independent of the message, yielding substantial rate benefits in the robust coding regime.

Concrete examples (binary sources, multiplication functions) demonstrate savings in rate for judicious allocation of measurement cost (for instance, selective sampling at the decoder), with explicit bounds detailed for different cost and distortion targets.

3. Zero-Delay and Nonlinear Encoding under Interference and Quantization

When transmitting sources in zero-delay regimes (scalar transmission per channel use), the presence of known interference or quantization constraints alters the optimal encoder structure. In the presence of known additive interference, linear interference cancellation (ICA) schemes are suboptimal. Instead, interference concentration (ICO) and one-dimensional lattice (1DL) schemes quantize the interference, shaping its impact on the channel input, and use companding maps for the source (Varasteh et al., 2016). Non-uniform quantization—where quantizer intervals for interference shrink toward the tails rather than the origin—outperforms uniform quantization, especially in strong interference regimes.

Necessary optimality conditions for the encoder under MMSE criterion are derived via variational calculus: $$h(v,s) = \frac{v}{2\lambda} \int g'(h(v,s)+s+w) f_W(w) \, dw$$ with numerical optimization (NOE) yielding encoder mappings that closely match the structured parameterized schemes (1DL–NU) in performance, but with lower complexity.

When the receiver employs a one-bit ADC and possesses correlated side information, optimal encoder maps under both MSE and distortion outage probability (DOP) criteria are periodic and shrink in support as correlation increases (Varasteh et al., 2017). Explicit necessary conditions are given for encoder optimality, coupling power constraint, source–side information correlation, and quantization noise.

4. Universal and Robust Decoding in the Presence of Codebook Noise

Universal decoding schemes address scenarios where the decoder only obtains noisy versions of the original codebook entries, as in biometrical identification systems or privacy-preserving applications (Merhav, 2016). Instead of using the “clean” codebook, the decoder employs a Lempel-Ziv (LZ) incremental parsing metric on noisy codewords and observed outputs: $$u(y, z) = \log P(y) + \sum_l c_e(y|z) \log c_e(y|z)$$ where $c_e(y|z)$ arises from joint LZ parsing. The average error probability of this universal decoder is shown to be as small as that of the optimal ML decoder, up to a sub-exponential factor in block length, and with identical error exponents when the ML decoder is exponentially reliable.

Robust Gray code constructions achieve rates arbitrarily close to the BSC channel capacity $1 - H_2(p) - \varepsilon$ while ensuring, for any block sent through the BSC, the decoded integer $\hat{j}$ is close to $j$ with error probability decaying as $\exp(-\Omega(t)) + \exp(-\Omega(d/\log d))$ for offsets $t$ (Con et al., 25 Jun 2024). The coding scheme deploys concatenation of Reed–Solomon and capacity-achieving codes, buffer markers decoded via majority rules, and interpolation rules to maintain the Gray code property under noise.

5. Quantum Optimal Encoding and Adaptive Strategies in Noisy Quantum Channels

In quantum dense coding over noisy channels, optimality depends on both the channel noise and the encoder’s strategy (unitary or non-unitary operations). For Pauli or depolarizing channels, the super dense coding capacity is: $$C_\text{Bell}^{(\text{one-sided})} = \log(d^2) - H(\{q_{mn}\})$$ for one-sided channels, and a threshold noise parameter $p_t \approx 0.345$ governs the transition from entangled (Bell state) to product state resource optimality (Shadman et al., 2010). For $p < p_t$, maximally entangled input is best; for $p > p_t$, separable states outperform. Non-unitary pre-processing further enhances capacity above this threshold.

In passive linear-optical quantum channels with finite-energy resource states and thermal environments, the maximum Holevo information is achieved by ensembles using uniform phase randomization and a finite set of channel attenuations (rings in phase space) (Tanggara et al., 2023). The optimal encoding is characterized by constraints on marginal information density; the output ensemble simplifies codebook construction, and these results are directly applicable to quantum reading of optical memory under noise.

6. Stochastic Encoding and Perceptual Constraints

Stochastic encoders (those that use shared or local randomness in the encoding process) may outperform deterministic encoders under constraints on reconstruction distribution—specifically, in the regime of perfect perceptual quality, where outputs must be drawn from the original source distribution (Theis et al., 2021). For example, encoding points on the unit circle at 1 bit/sample, the stochastic universal quantizer yields expected distortion $1 - (2/\pi)$, as opposed to $1 - (4/\pi^2)$ for the best deterministic quantizer, representing a 38.9% improvement. Such gains appear when reconstructions are required to maintain perceptual indistinguishability (as in neural compression or image coding).

7. Encoding in Networked, Multi-Terminal, and Control Systems

Optimal encoding schemes for real-time multi-terminal communication leverage finite-dimensional sufficient statistics and dynamic programming methods (0910.4955). Encoders act as filters that compress the observation history into a recursively updated statistic (for example, a posterior belief vector) and make decisions based on current observations and system state. In noisy networks, encoder structure adapts to both channel uncertainty and causality constraints. When communication channels are noise-free and common information is present, coordination is enhanced and causal optimality can be achieved; in noisy cases, robust encoder/decoder structures are required.

In joint source-channel coding for dynamical systems over Gaussian channels (with noisy feedback), optimal encoder and decoder pairs are shown to be linear finite-memory state-space filters (Gattami, 2015). The system exhibits a separation principle when encoder-side measurements are noisy, and necessary and sufficient conditions for stationary bounded error are derived connecting system instability to channel capacity.

8. Information-Theoretic and Thermodynamic Perspectives in Neural Systems

In stochastic latent-variable models of sensory encoding, restricted Boltzmann machines (RBMs) learn strategies that balance precision and noise robustness for stimuli of varying information content (Rule et al., 2018). High-information (rare) stimuli are encoded with suppressed variability; frequent (low-information) stimuli incur higher variability. Thermodynamic analysis reveals that statistical criticality emerges at model sizes sufficient to capture input statistics, associated with a phase transition and scale-free power law behavior in codeword frequencies (Zipf’s law). The Fisher information matrix detects these transitions, guiding network size for optimal encoding precision.

Key Summary Table

Setting / Channel Type	Optimal Encoding Principle	Main Performance Metric / Result
Degraded broadcast channel	Natural encoding (NE), permutation encoding	Capacity region, parametric formulas
Lossy computing with cost	Joint encoder/action optimization, robust coding	Rate-distortion-cost function
Zero-delay, interference	Nonlinear companding (ICO, 1DL), non-uniform quant.	MSE, complexity, optimality criteria
Universal decoding, codebook	LZ incremental parsing, decoding on noisy codebook	Error exponent (ML), sub-exp bounds
Quantum channel	Uniform phase randomization, discrete ring codes	Holevo information, channel capacity
Stochastic encoder	Universal quantization, shared randomness	Distortion (perfect percept. quality)
Networked/control systems	Sufficient statistic filters, dynamic programming	MSE, causal optimality, stationarity
Neural population codes	RBM, criticality, adaptive variability suppression	Energy–entropy balance, criticality

Optimal noisy encoders—whether natural, robust, stochastic, or quantum—represent principled solutions that judiciously balance information, noise, complexity, and resource constraints to optimally transmit, compute, or represent information in noisy environments. These schemes span foundational information theory, practical coding constructions, quantum communications, and biological computation.