---
title: Authentic Discrete Diffusion (ADD) Model
url: https://www.emergentmind.com/topics/authentic-discrete-diffusion-add
type: topic
---

# Authentic Discrete Diffusion (ADD) Model

Authentic Discrete Diffusion (ADD) is a diffusion framework designed to work directly on discrete data such as class labels and text tokens, while preserving the core structure of continuous denoising diffusion probabilistic models; in a broader information-theoretic and CTMC usage, an ADD model is a discrete diffusion model whose training losses and inference procedures are exactly tied to the data log-likelihood, mutual information, and the true reverse continuous-time Markov chain, rather than to heuristic masking rules or continuous relaxations [2510.01047] [2510.24088] [2607.05381].

## 1. Definition, motivation, and scope

Standard diffusion models operate on continuous vectors with Gaussian noise and MSE losses. Discrete data live on a categorical simplex with non-Euclidean geometry and mutually exclusive outcomes, making naive extensions of DDPM problematic. The ADD formulation in "Authentic Discrete Diffusion Model" [2510.01047] is explicitly positioned against two families that the paper describes as inadequate for this setting: embedding-based diffusion, which maps discrete tokens to continuous embeddings and runs Gaussian diffusion there, and masked or BERT-style “discrete diffusion,” which the paper characterizes as pseudo discrete diffusion because it lacks a well-defined Gaussian Markov forward process and is not a true diffusion process in the sense of Sohl-Dickstein et al.

In that formulation, ADD is introduced to be “authentic” in two senses. First, it preserves the defining characteristics of diffusion models: a Gaussian forward noising process with a variance schedule and an iterative reverse denoising process. Second, it operates directly on float-encoded one-hot categorical vectors, not on continuous embeddings or masked tokens, and uses explicit discretization, `argmax → one-hot`, at each step [2510.01047].

A broader theoretical usage appears in "Information-Theoretic Discrete Diffusion" [2510.24088], which states that an “Authentic Discrete Diffusion (ADD)” model would be a discrete diffusion model whose training losses and inference procedures are not heuristic, but are exactly tied to the data log-likelihood and mutual information. "What Does a Discrete Diffusion Model Learn?" sharpens this criterion further: an authentic method is mathematically aligned with the true reverse CTMC induced by a chosen forward noising process, with a model output that is unambiguously interpreted as reverse jump rates or an equivalent coordinate such as denoiser, cavity, or score [2607.05381]. This suggests a two-level description: ADD is both a concrete one-hot-space diffusion architecture and a criterion of mathematical faithfulness for discrete diffusion more generally.

## 2. One-hot-space diffusion model

For a \(K\)-class categorical variable, ADD represents a label as a one-hot vector
\[
\mathbf{y}_0 \in \{0,1\}^K,\quad \sum_{k=1}^K y_0^{(k)} = 1.
\]
For text, a sentence of length \(N\) is represented as
\[
\mathbf{Y}_0 = [\mathbf{y}_{0,1}, \dots, \mathbf{y}_{0,N}],\quad \mathbf{y}_{0,i}\in \{0,1\}^K.
\]
These one-hot vectors are treated as real-valued vectors in \(\mathbb{R}^K\) so that Gaussian noise can be added [2510.01047].

The forward process uses the same functional form as DDPM, but applied directly to one-hot vectors:
\[
q(\mathbf{y}_t \mid \mathbf{y}_0) = \mathcal{N}\!\big(\mathbf{y}_t;\, \sqrt{\bar{\alpha}_t}\,\mathbf{y}_0,\,(1-\bar{\alpha}_t)\mathbf{I}\big).
\]
For text, the same corruption is applied independently to each token:
\[
q(\mathbf{y}_{t,i} \mid \mathbf{y}_{0,i}) = \mathcal{N}\!\big(\mathbf{y}_{t,i}; \sqrt{\bar{\alpha}_t}\,\mathbf{y}_{0,i}, (1-\bar{\alpha}_t)\mathbf{I}\big),\quad i=1,\dots,N.
\]
The reverse process during training predicts a categorical distribution over the clean one-hot label:
\[
p_\theta(\mathbf{y}_0 \mid \mathbf{y}_t, c) = \operatorname{Softmax}\big(f_\theta(\mathbf{y}_t, t, c)\big).
\]
At inference, ADD enforces discrete structure via an `argmax→one-hot` projection at every step:
\[
\hat{\mathbf{y}}_0 = \operatorname{onehot}\!\Big(\arg\max_k\, p_\theta(y_0^{(k)} \mid \mathbf{y}_t, c)\Big),
\]
followed by re-noising,
\[
\mathbf{y}_{t-1} \sim \mathcal{N}\!\big(\sqrt{\alpha_{t-1}}\,\hat{\mathbf{y}}_0,\,(1-\alpha_{t-1})\mathbf{I}\big).
\]
Iterating this loop from a large \(T\) down to \(t=1\) yields progressively sharper categorical predictions. The paper states that this combination yields an authentic diffusion in one-hot space and forms a direct bridge between discriminative tasks and generative tasks [2510.01047].

## 3. Timestep-conditioned cross-entropy and the discriminative–generative bridge

A central novelty of ADD is the timestep-conditioned cross-entropy objective, which replaces the usual MSE noise regression of DDPMs:
\[
\mathcal{L}_{\text{CE}} = -\mathbb{E}_{t \sim \mathcal{U}[1,T]} \,\bar{\alpha}_t \sum_{k=1}^K y_0^{(k)} \log p_\theta(y_0^{(k)} \mid \mathbf{y}_t, c).
\]
For multi-token text, the paper gives
\[
\mathcal{L}_{\text{text}} = -\sum_{i=1}^N \sum_{k=1}^K y_{0,i}^{(k)} \log p_\theta(y_{0,i}^{(k)} \mid \mathbf{y}_{t,i}, c).
\]
Uniform sampling over timesteps trains the model to handle all noise levels, while the coefficient \(\bar{\alpha}_t\) is motivated as a way to prevent the diffusion network from over-relying on the conditioning features \(c\) [2510.01047].

The paper treats this loss as the mechanism by which denoising becomes classification. In the discriminative view, the model predicts \(y_0\) given a noisy label \(\mathbf{y}_t\) and features \(c\). In the generative view, the same predictor is applied repeatedly through the diffusion chain. This is the sense in which the method “establishes a bridge between discriminative and generative learning” [2510.01047].

The ablations reported in the paper are unusually sharp. An ADD variant with regressive, MSE-like loss for noise prediction reaches Top-1 \(= 0.13\%\), whereas ADD with timestep-conditioned cross-entropy reaches Top-1 \(= 82.72\%\) at 400 epochs. The paper also reports that classifier-free guidance improves Top-1 from \(82.36\%\) to \(82.82\%\), and that argmax followed by one-hot projection improves Top-1 from \(82.35\%\) to \(82.82\%\). These results are used to support the claim that discrete cross-entropy on one-hot labels is essential, and that timestep weighting, classifier-free guidance, and argmax-plus-one-hot discretization all contribute measurable gains while maintaining diffusion behavior [2510.01047].

## 4. Information-theoretic likelihood interpretation

"Information-Theoretic Discrete Diffusion" [2510.24088] supplies a different notion of authenticity. Its starting point is a continuous-time Markov chain forward diffusion
\[
\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p_{\text{data}},
\]
with reverse rate matrix
\[
Q_t(y,x) = \begin{cases} \displaystyle\frac{p_t(y)}{p_t(x)} Q_t(x,y) & x\neq y, \\[0.5em] -\sum_{\tilde{y}\neq x} Q_t(\tilde{y},x) & x=y. \end{cases}
\]
The central object is the marginal ratio
\[
s_t^\star(x)_y := \frac{p_t(y)}{p_t(x)},
\]
which plays the role analogous to the score \(\nabla \log p_t\) in continuous diffusion [2510.24088].

For general CTMC diffusion, the paper introduces the Denoising Score Entropy loss and proves that its optimum is the true ratio \(s_t^\star\). It then establishes the Information-Minimum Denoising Score Entropy relation:
\[
\frac{d}{dt} I(x_0; x_t) = - mdse(t),
\]
together with an exact negative log-likelihood decomposition,
\[
-\log p_0(x_0) = \int_0^\infty mdse(x_0,t)\,dt,
\]
in the full-noise limit under ergodicity. The paper’s explicit conclusion is that integrated DSE is equal to the true NLL in the score-optimal case; DSE is not merely a variational upper bound, but a tight and principled estimator of log-likelihood [2510.24088].

For masked or absorbing diffusion, the paper gives a parallel theory based on Denoising Cross-Entropy. With
\[
DCE(\mathbf{x}_0,\mathbf{x},c) := \sum_{i=1}^L 1[x^i=\square] \log \frac{1}{c(\mathbf{x})_{i,x_0^i}},
\]
it proves both DSE–DCE equivalence for absorbing processes and an Information-Minimum Denoising Cross-Entropy identity:
\[
\frac{d}{d\lambda} I(\mathbf{x}_0;\mathbf{x}_\lambda) = -\frac{1}{\lambda}\, mdce(\lambda).
\]
The corresponding NLL identity is
\[
-\log p_0(\mathbf{x}_0) = \int_0^1 \frac{1}{\lambda} mdce(\mathbf{x}_0,\lambda)\,d\lambda.
\]
The paper states this explicitly: DCE is an exact estimator of NLL for absorbing diffusion, not a mere variational bound. It further derives time-free likelihood formulas, conditional likelihood estimators for prompt–response tasks, and coupled Monte Carlo estimators of likelihood ratios [2510.24088].

Within this information-theoretic vocabulary, ADD denotes a discrete diffusion model whose losses are distribution–loss matched in the same sense that Gaussian noise pairs with MSE and I-MMSE. This use of the term does not depend on one-hot Gaussian diffusion specifically; it depends on exact likelihood matching.

## 5. CTMC alignment, ELBO exactness, and coordinate semantics

A more general state-space account appears in "Foundations of Diffusion Models in General State Spaces" [2512.05092]. There, a genuine discrete diffusion is described as a Markov chain or CTMC on a finite state space, with forward one-step kernel
\[
q(x_t \mid x_{t-1}) = \mathrm{Cat}(x_t;\, \tilde Q_t e_{x_{t-1}}),
\]
continuous-time rate matrix
\[
R_t = \frac{\alpha_t'}{\alpha_t}\left(I - \pi \mathbf 1^\top\right),
\]
and a reverse process derived from time reversal rather than introduced ad hoc. For masking or absorbing diffusion, the ELBO reduces to a weighted masked-language-modeling cross-entropy, which the paper presents as an ELBO-based maximum likelihood method rather than a heuristic masking objective [2512.05092].

"What Does a Discrete Diffusion Model Learn?" develops the strongest formal criterion of authenticity [2607.05381]. It derives the CTMC ELBO with boundary terms and proves the Oracle Distance theorem: the negative ELBO is exactly equal to the data entropy plus the path KL from the oracle reverse process to the learned one. It also proves that the unique optimizer is the conditional expectation of the true reverse jump rate given the current noisy state, and that its irreducible cost is
\[
-\frac{d}{dt}I(Z_0; Z_t).
\]
In this framework, denoiser, cavity, and score are “one object in different coordinates,” and reading a neural network in the wrong coordinate changes the process being trained and sampled [2607.05381].

For token-factorizing noise, the paper gives three exact coordinates for the optimizer: denoiser, cavity, and score, with closed-form conversions among them. It also proves that denoiser and cavity coincide for masked diffusion but not for uniform diffusion, and that a denoiser parameterization makes the uniform ELBO diverge at initialization while the bridge plug-in stays finite. A plausible implication is that authenticity, in the strict CTMC sense, requires not only a legitimate forward process and an ELBO, but also a precise semantic contract for what the network outputs represent during both training and sampling [2607.05381].

## 6. Architectures, empirical results, applications, and nomenclature

In its direct architectural instantiation, ADD is implemented as a hybrid of a Transformer feature extractor for conditioning and a diffusion module that takes noisy one-hot vectors plus conditioning and outputs logits over classes or tokens. The full model reported in "Authentic Discrete Diffusion Model" has about \(111\)M parameters, with about \(87\)M in the Transformer encoder for images and about \(24\)M in the diffusion module. Training uses AdamW with learning rate \(1\times 10^{-4}\), weight decay \(0.3\), 20 epochs of warmup followed by cosine decay, effective batch size \(4096\) over \(8\times\)A100 GPUs, PyTorch AMP, and global gradient clipping at norm \(3.0\) [2510.01047].

The paper reports ImageNet Top-1 results of \(82.8\%\) and \(83.0\%\) for ViT-Base + ADD variants, compared with \(82.3\%\) for a standard ViT-Base, \(82.6\%\) for a standard ViT-Large, and \(83.1\%\) for a standard ViT-Huge at \(448\) resolution. For MS COCO captioning, the reported CLIP Scores are \(0.30\) for ground-truth captions, \(0.16\) for shuffled captions, \(0.18\) for PDD-generated captions, and \(0.25\) for ADD-generated captions. The paper interprets these results as showing that the same one-hot diffusion backbone can serve both classification and multi-token generation [2510.01047].

The broader ADD perspective has already appeared in task-specific settings. In speech enhancement, ADDSE uses absorbing discrete diffusion over neural codec codes, with a special absorbing state \(M=K+1\), a denoising cross-entropy objective weighted by \(1/\lambda\), and an RQDiT architecture combining techniques from RQ-Transformer and diffusion Transformers for non-autoregressive modeling [2602.22417]. In MIMO detection, GD4 performs denoising directly in the discrete symbol space and enables fast inference with one or a few denoising evaluations, in contrast to existing diffusion-based detectors that operate in a continuous relaxed space [2605.00423]. In discrete diffusion language models, watermarking work describes an authenticity-oriented notion in which outputs are made authentically attributable through a distribution-preserving Gumbel-max scheme seeded by sequence index, yielding a detector with an exponentially decaying probability of false detection in the token sequence length [2511.02083].

The acronym itself is not fully stable across the literature. The following usages are explicitly documented:

| Paper | Meaning of “ADD” | Relation |
|---|---|---|
| "Authentic Discrete Diffusion Model" [2510.01047] | Authentic Discrete Diffusion | Direct one-hot-space framework |
| "Information-Theoretic Discrete Diffusion" [2510.24088] | “Authentic” as exact likelihood matching | Broader theoretical criterion |
| "Add and Thin: Diffusion for Temporal Point Processes" [2311.01139] | No separate algorithm formally called “ADD” | “Add” refers to addition of Poisson events |
| "Adversarial Diffusion Distillation" [2311.17042] | Adversarial Diffusion Distillation | Unrelated acronym |

This nomenclatural variation matters because several papers use “authentic” to mean mathematically faithful to discrete structure, but not all uses of “ADD” refer to the same method. The direct one-hot framework of [2510.01047], the information-theoretic likelihood interpretation of [2510.24088], and the CTMC exactness criteria of [2607.05381] are nevertheless mutually reinforcing. Taken together, they describe ADD as a family of discrete diffusion methods that preserve core diffusion structure directly in discrete representations, align losses with exact likelihood or ELBO identities, and require consistent interpretation of reverse-process parameterizations during training and sampling.

Source: https://www.emergentmind.com/topics/authentic-discrete-diffusion-add