Dyadic Probability Tree: Structure & Applications

• Dyadic probability tree is a combinatorial structure representing dyadic PMFs through full prefix-free code trees, where each probability is a negative power of two.
• It employs Geometric Huffman Coding to construct optimal dyadic PMFs that effectively minimize KL divergence from capacity-achieving distributions with O(m log m) complexity.
• The framework underpins modulation and distribution matching applications, mapping uniform bit streams to shaped channel symbols for near-capacity performance.

A dyadic probability tree is a combinatorial structure that encodes dyadic probability mass functions (PMFs) through its correspondence with full prefix-free code trees. Within the domain of discrete memoryless channels (DMCs) and memoryless discrete noiseless channels (DNCs), dyadic probability trees facilitate the generation and manipulation of PMFs that conform to dyadic constraints, i.e., each probability is a negative power of two. The framework directly links the design of such PMFs to prefix-free coding and enables efficient approaches to minimizing information-theoretic divergence from capacity-achieving input distributions. This construction is essential in scenarios where input symbols are mapped from streams of independent, equiprobable bits using modulator architectures.

1. Definition and Structure of Dyadic PMFs

A probability mass function $p=(p_1, \dots, p_m)$ is classified as dyadic if each $p_i$ can be expressed as $p_i = 2^{-\ell_i}$, where $\ell_i \in \mathbb{N}$ and $\sum_{i=1}^m 2^{-\ell_i} = 1$. There is a canonical bijection between dyadic PMFs and full prefix-free codes on $m$ symbols. In the associated prefix-free code tree, each symbol $i$ is assigned a codeword of length $\ell_i$, satisfying Kraft’s equality:

$\sum_{i=1}^m 2^{-\ell_i} = 1$

Interpreted probabilistically, traversing the tree until the emission of codeword $i$ occurs with probability $2^{-\ell_i}$. Thus, every full prefix tree induces the dyadic PMF $p_i = 2^{-\ell_i}$.

2. Capacity Gap and KL Divergence Minimization

Let $\mathsf{C}$ denote the channel capacity of a DMC and $p^* = \arg\max_p \,\mathcal{I}(p)$ be the unique capacity-achieving PMF. When employing a dyadic PMF $p$, the achieved mutual information obeys the relation (see Gallager '68):

$$\mathcal{I}(p) = \mathsf{C} - D(r\Vert r^*) \geq \mathsf{C} - D(p\Vert p^*)$$

where $r$ and $r^*$ are output PMFs, and $D(p\Vert q) = \sum_i p_i\,\log\frac{p_i}{q_i}$ denotes Kullback–Leibler divergence. To minimize the loss from non-optimal input distributions, the optimal dyadic PMF solves:

$\min_{\substack{p\text{ dyadic}\p_i=0 \text{ if } p^*_i=0}} D(p \Vert p^*)$

For DNCs, a weighted KL minimization arises:

$$\bar{H}(p) = \frac{H(p)}{\sum_i p_i w_i} = \mathsf{C}\,R - \frac{D(p\Vert (p^*)^R)}{\sum_i p_i w_i}$$

Hence, in both scenarios, optimal dyadic PMFs are characterized by minimizing suitable KL distances relative to (possibly weighted) capacity-achieving PMFs.

3. Geometric Huffman Coding for Dyadic PMFs

Traditional Huffman coding constructs prefix trees by sequentially merging the least probable symbols, minimizing expected codeword length for a given PMF. Geometric Huffman Coding (GHC) modifies the merge rule: instead of summing weights, it employs a geometric strategy. For $x_{m-1} \geq x_m$, GHC merges as follows:

$$x' = \begin{cases} x_{m-1}, & x_{m-1} \geq 4x_m \ 2\sqrt{x_{m-1}x_m}, & x_{m-1} < 4x_m \end{cases}$$

This procedure, applied recursively within a sorted list and encoded as a prefix tree, yields lengths $\ell_i$ whose induced PMF $p_i = 2^{-\ell_i}$ minimizes $D(p \Vert x)$ over dyadic PMFs.

Geometric Huffman Coding (GHC) Pseudocode:

1. Sort $x_1 \ge x_2 \ge \dots \ge x_m$.
2. While more than one symbol remains:
   • Let $(x_{m-1},x_m)$ be the two smallest.
   • Form $x'$ via the geometric rule above.
   • Remove $x_{m-1},x_m$, insert $x'$, and resort.
3. The depth of each leaf defines $\ell_i$, establishing $p_i=2^{-\ell_i}$.

Inductive proofs establish that the optimal merge requires assigning identical maximal length to $(x_{m-1},x_m)$ and replacing them with an effective cost $\tfrac{u_{m-1}+u_m}{2}-1$, where $u_i = -\log_2 x_i$.

4. Algorithmic Complexity

Each merge operation entails removal of two minimal elements and insertion of a new value into a sorted list of at most $m$ elements. Using a priority queue or heap, per-merge complexity is $O(\log m)$, and there are $m-1$ merges, yielding overall GHC runtime $O(m\log m)$. This matches the computational order of classical Huffman coding.

5. Block Coding and Asymptotic Capacity

Dyadic probability trees can be extended to block coding. For blocks of length $k$, the joint capacity PMF is $p^{(k)*}=p^*\times\cdots\times p^*$, and GHC is applied to $p^{(k)*}$ to obtain the dyadic approximation $p^{(k)}$. The divergence satisfies:

$D(p^{(k)}\Vert p^{(k)*}) \leq \log 2$

and thus:

$$\frac{1}{k} D(p^{(k)}\Vert p^{(k)*}) \xrightarrow{k\rightarrow\infty} 0$$

resulting in

$$\frac{1}{k}\,\mathcal{I}(p^{(k)}) \longrightarrow \mathsf{C}$$

as $k \to \infty$, guaranteeing the asymptotic capacity-achieving property of the construction.

6. Applications in Modulation and Distribution Matching

In digital modulation schemes where nonuniform input distributions (e.g., shaped QAM or APSK constellations) are desirable, dyadic probability trees provide a tractable means to induce symbol distributions approximating the channel’s capacity-optimizing PMF. The process turns a stream of uniformly distributed bits into appropriately distributed channel symbols. Construction of a full prefix tree, with leaves mapped to constellation points, yields dyadic PMFs that closely approach the theoretical optimum for mutual information or entropy rate. The structural and computational properties of GHC—$O(m\log m)$ complexity and asymptotic optimality—make dyadic probability trees highly effective for practical distribution matching in coded modulation systems (Böcherer et al., 2010).