Distribution Matching Loss (DMD)

• Distribution Matching Loss (DMD) is a measure that quantifies how well a model's output distribution aligns with a target distribution using divergence metrics like KL divergence.
• It underpins fixed-length, one-to-one, invertible mapping schemes such as CCDM and optimal codebooks, ensuring near-optimal energy efficiency in communications.
• The scaling laws indicate that while normalized divergence per symbol vanishes with increasing block length, the unnormalized divergence grows logarithmically, highlighting trade-offs in system design.

Distribution Matching Loss (DMD) quantifies and steers the alignment between a model-generated distribution and a target distribution in scenarios ranging from communications and generative modeling to domain adaptation and graph compression. The term is used for diverse but precisely formulated objectives that drive distributions of outputs (symbols, features, samples, etc.) toward a desired structure, measured via an explicit divergence, distance, or discrepancy. DMD forms the foundation of modern practical distribution matching, especially where invertibility or strict one-to-one mapping is required.

1. Mathematical Formulation and Codebook Design

A canonical instantiation of DMD is the task of fixed-length distribution matching with a one-to-one, invertible mapping. Given a source of uniformly distributed input bits (denoted $\mathbb{B}^m$), the distribution matcher defines an injective map $f: \mathbb{B}^m \to \mathcal{A}^n$ to a (typically binary) output sequence that approximates a specified target distribution $P_A^{\otimes n}$. The codebook $C = f(\mathbb{B}^m)$ contains $|C| = 2^m$ codewords. The output distribution $U_C$ is uniform on $C$ by construction.

The DMD for codebook $C$ with respect to the target $P_A$ is the informational divergence

$$D(U_C \| P_A) = \sum_{x \in C} \frac{1}{|C|} \log_2 \frac{1/|C|}{P_A(x)}.$$

Introducing the average letter distribution $P_c$ over $C$ allows decomposing this divergence (see Equation (15) and (17)): $$D(U_C \| P_A) = -\log_2|C| + n H(P_c) + n D(P_c \| P_A),$$ where $H(P_c)$ is the entropy of the average letter distribution and $D(P_c \| P_A)$ is the Kullback-Leibler divergence for the single-symbol marginals. In the binary case, writing $p = P_A(1)$ and $p_c = P_c(1)$,

$$D(U_C \| P_A) = -\log_2|C| + n H(p_c) + n D(p_c \| p).$$

Codebook construction strategies profoundly affect minimization of DMD:

• Constant Composition Distribution Matcher (CCDM): All codewords have the same type (composition). CCDM is efficiently implementable (often via arithmetic coding) and achieves low, well-characterized divergence.
• Optimal Codebooks: Consist of type sets with lowest weights (for $p<1/2$), i.e., the union of codewords with the minimum number of ones. This is a union-of-type-sets construction, found via sorting codewords by likelihood under $P_A$ (see Lemma 5).

2. Divergence Scaling and Asymptotic Properties

A central result is the scaling law for DMD as a function of the output block length $n$ (Schulte et al., 2017):

• Unnormalized divergence $D(U_C \| P_A)$ grows at least logarithmically with $n$.
  • For CCDM (single-type codebook), the upper bound is $D(U_C \| P_A) \lesssim \log_2(n) + \text{const}$.
  • For the optimal codebook (union-of-type-sets), the lower bound is

  $$\liminf_{n \to \infty} \left(D(U_C \| P_A) - 0.5\log_2 n\right) \geq \text{const}. \quad \text{(Equation (34))}$$

• Normalized divergence per symbol, $D(U_C \| P_A)/n$, vanishes as $n \to \infty$ for both CCDM and optimal constructions. This is essential for high-rate, energy-efficient communication, as it ensures the transmitted distribution approaches the target in the limit.

The following table summarizes key relationships:

Codebook Construction	Normalized Divergence ($/n$)	Unnormalized Divergence	Practical Complexity
CCDM (single type)	$\to 0$ as $n\to\infty$	$\sim \log_2 n$	Simple, implementable
Union of type sets	$\to 0$ as $n\to\infty$	$\geq 0.5\log_2 n$	Higher, less practical

3. Trade-offs: Energy Efficiency, Stealth, and Detectability

The scaling law for DMD has immediate practical interpretation:

• Energy-efficient Communications (e.g., PAS): Vanishing normalized divergence ensures achievable rates approach channel capacity. CCDM performs nearly optimally here; the gap to the absolute optimal codebook is a constant.
• Stealth Communication: Often requires absolute (unnormalized) divergence to vanish with $n$, i.e., $D(U_C \| P_A) \to 0$. The logarithmic scaling of DMD makes this unattainable with one-to-one, fixed-length DMs—even the optimal construction fails to yield undetectable communication for arbitrarily large block sizes.
• Trade-off: The unavoidable logarithmic growth of DMD forces a balance between energy efficiency and undetectability in systems where both properties are desired.

A plausible implication is that to achieve vanishing DMD in the total sense for stealth, alternative strategies—such as randomized, one-to-many mappings—may be required, although such schemes fall outside the strict invertibility regime analyzed in (Schulte et al., 2017).

4. Practical Implications and Implementability

The decomposition of DMD into codebook entropy and average composition divergence underscores significant implementational consequences:

• CCDM: Achieves DMD within a constant of the optimum, is invertible via arithmetic coding, and is robust to scaling.
• Optimal Codebooks: Offer minimal theoretical DMD but are often less practical due to exponentially larger or more irregular codebook structures.
• Complexity vs. Divergence: CCDM's slight practical gap to the optimum (bounded, independent of $n$) is generally negligible for energy efficiency but could be critical for highly constrained applications.

The explicit formula

$$D(U_C \| P_A) = -\log_2|C| + n H(p_c) + n D(p_c \| p)$$

allows system designers to tradeoff codebook size, rate, and divergence analytically for any target distribution.

5. Connection to Modern Distribution Matching Approaches

The theoretical framework and scaling laws established for the binary, fixed-length, one-to-one DM are foundational for a wide range of subsequent developments:

• Extensions to non-binary alphabets, probabilistic amplitude shaping, and coded modulation.
• Implementation of multi-level and parallel architectures (e.g., PA-DM, hierarchical DM) motivated by practical throughput and latency constraints.
• Advances in fast, hardware-oriented distribution matchers that balance storage, lookup complexity, and DMD.
• Methodological influence on stochastic or one-to-many DMs (not analyzed in (Schulte et al., 2017)), which relax the invertibility or fixed-length constraints to further reduce DMD when required.

Such extensions may alter the scaling of DMD, but the impossibility of absolutely vanishing unnormalized divergence under fixed one-to-one, invertible mapping persists as a fundamental constraint.

6. Summary and Theoretical Significance

The analysis in (Schulte et al., 2017) establishes:

• Rigorous upper and lower bounds on the DMD for fixed-length, one-to-one, binary-output distribution matching.
• The logarithmic scaling law is both tight and unavoidable in invertible schemes.
• Practical constructions such as CCDM are near-optimal, with divergences remaining within a fixed constant of the best achievable value.
• The results underpin the core trade-offs in physical-layer coding, secret or covert communications, and general distribution matching problems where strict invertibility is required.

These insights are central to both the engineering and information-theoretic understanding of distribution matching loss and its unavoidable scaling in practical applications.