---
title: 'FLDD: Forward-Learned Discrete Diffusion'
url: https://www.emergentmind.com/topics/forward-learned-discrete-diffusion-fldd
type: topic
---

# FLDD: Forward-Learned Discrete Diffusion

Searching arXiv for the cited FLDD-related papers to ground the article in current arXiv records.
arXiv search: "2305.11089 Blackout Diffusion Generative Diffusion Models in Discrete-State Spaces"
Forward-Learned Discrete Diffusion (FLDD) denotes a class of diffusion methods in which the forward corruption process in a discrete or discretized domain is itself parameterized and learned, rather than fixed a priori. Across recent arXiv work, the term spans several closely related uses: a non-Markovian discrete diffusion framework with learnable forward marginals and posteriors [2605.18204], an exact discrete-state perspective in which a learned generator \(Q_\theta\) is supported by closed-form time reversal [2305.11089], and task-specific realizations such as relay-enabled masked diffusion for language [2605.22967], per-element molecular masking schedules [2505.16790], and learned latent forward trajectories for text [2601.16220]. This suggests that FLDD is best understood as a broader design principle—learning how data are noised so that denoising becomes easier, more accurate, or more efficient—rather than as a single canonical algorithm.

## 1. Terminological scope and conceptual identity

In the narrowest sense, FLDD refers to the method introduced in "Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster" [2605.18204]. There, the central claim is that conventional discrete diffusion uses a fixed forward process that induces a reverse target that is typically not factorized, whereas the generative model is usually parameterized by factorized reverse conditionals for efficiency. FLDD addresses this mismatch by learning the forward noising process while leaving the reverse model in the same efficient factorized family.

A broader usage appears in several adjacent lines of work. "Blackout Diffusion: Generative Diffusion Models in Discrete-State Spaces" develops an exact theory of discrete-state forward and reverse diffusion and explicitly states that this framework immediately supports learning \(Q\) or \(K_t\) in an FLDD-style construction [2305.11089]. "Learned Relay Representations for Forward-Thinking Discrete Diffusion Models" uses FLDD as a direct synonym for relay-enabled, forward-thinking discrete diffusion trained via truncated BPTT, emphasizing latent state propagation across denoising steps rather than explicit learning of a Markov generator [2605.22967]. "Learning Flexible Forward Trajectories for Masked Molecular Diffusion" presents MELD as an FLDD instance in which per-element corruption schedules are learned to reduce state-clashing in molecular graphs [2505.16790]. "Towards Latent Diffusion Suitable For Text" realizes the same principle in latent space through Neural Flow Diffusion Models, where the forward process is learned from data and induces a discrete process through decoding [2601.16220].

A common misconception is that FLDD names a single universally agreed architecture. The literature instead uses the term for a family of methods united by one design move: the forward trajectory is no longer treated as a fixed nuisance schedule. Another common misconception is that learning the forward process requires abandoning parallel, factorized reverse sampling. The 2026 FLDD formulation is explicit that the reverse model remains factorized and the sampler is unchanged [2605.18204].

## 2. Exact discrete-state foundations

The most general mathematical foundation in the supplied corpus is the exact discrete-state framework of Blackout Diffusion. The per-coordinate state space is a finite set \(\Omega = \{0,1,\ldots,M\}\), the full data space is \(\Omega^N\), and the forward process may be written either as a discrete-time Markov chain \(x_{k+1} \sim K_k(\cdot \mid x_k)\) or as a continuous-time Markov chain with generator \(Q\) [2305.11089]. In continuous time, the forward marginals satisfy the Kolmogorov forward equation
\[
\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.
\]

For nearest-neighbor transitions, the forward Master equation can be written in a step-operator form, and the framework generalizes to arbitrary banded or nonlocal transitions by decomposing \(Q\) into banded components. The resulting perspective replaces Gaussian additive noise by discrete jump noise, which may be local, global, structured, masking-based, or absorbing.

The central exact reversal formula is the continuous-time reverse generator
\[
q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,
\]
with diagonals determined by negative row sums. In matrix form,
\[
Q^*_t = \mathrm{Diag}(\pi_t)^{-1} Q^\top \mathrm{Diag}(\pi_t).
\]
The discrete-time analogue is the Bayes-form reversal
\[
\mathbb{P}\{X_k=i \mid X_{k+1}=j\} = \frac{K_k(j|i)\, p_k(i)}{p_{k+1}(j)}.
\]

This exact reversal is the discrete counterpart of the reverse-time correction in continuous Gaussian diffusion. The Blackout Diffusion analysis relates the jump-process correction to the score term in the reverse SDE, and identifies a discrete-state score
\[
s_{\mathrm{dis},\sigma}(m,s) \propto \nu_\sigma(m'_\sigma)\,\frac{p_{(m'_\sigma,s)\vert(o,0)} - p_{(m,s)\vert(o,0)}}{p_{(m,s)\vert(o,0)}}.
\]
In the dense-state limit, this approaches \(\partial_x \log p_t(x)\) up to scaling.

The exactness of this formulation matters for FLDD because it yields closed-form reverse targets once the forward process is specified. The required reversibility condition is that \(\pi_t(i) > 0\) on the relevant support; otherwise reverse rates involving state \(i\) are undefined. Detailed balance is not required, although it is compatible with the framework.

## 3. Canonical FLDD: learnable forward marginals, posteriors, and total correlation

The 2026 FLDD paper formalizes the forward-learning idea in a deliberately non-Markovian way [2605.18204]. The data lie in \(x_0 \in \{1,\ldots,K\}^d\), and the reverse model remains the standard factorized reverse chain
\[
p_\theta(x_{t-1} \mid x_t) = \prod_{i=1}^d \mathrm{Cat}(x_{t-1}^i; v_\theta^i(x_t,t)).
\]
The forward process is replaced by a learnable family consisting of forward marginals \(q_\phi(x_t \mid x_0)\) and forward posteriors \(q_\phi(x_{t-1} \mid x_t,x_0)\), with joint factorization
\[
q_\phi(x_{0:T} \mid x_0) = q_\phi(x_T \mid x_0)\prod_{t=1}^T q_\phi(x_{t-1} \mid x_t,x_0).
\]

The forward marginals are factorized across coordinates,
\[
q_\phi(x_t \mid x_0)=\prod_{i=1}^d q_\phi(x_t^i \mid x_0), \qquad q_\phi(x_t^i \mid x_0)=\mathrm{Cat}(x_t^i; u_\phi^i(x_0,t)),
\]
but each \(u_\phi^i(x_0,t)\) depends on the full \(x_0\), not only on \(x_0^i\). This is the key mechanism by which FLDD can perform data-dependent destruction of information while retaining a factorized form.

Consistency between marginals and posteriors is enforced through a per-coordinate maximal coupling. For coordinate \(i\), let \(a_t = u_\phi^i(x_0,t)\) and \(a_s = u_\phi^i(x_0,s)\) with \(s=t-1\). The conditional is
\[
q_\phi(x_{t-1}^i = j \mid x_t^i = k, x_0) =
\begin{cases}
\min(a_s^k,a_t^k)/a_t^k, & \text{if } j=k,\\[4pt]
\frac{a_t^k-\min(a_s^k,a_t^k)}{a_t^k}\, r_j, & \text{if } j\neq k,
\end{cases}
\]
where
\[
r_j = \frac{\max(a_s^j-a_t^j,0)}{\sum_\ell \max(a_s^\ell-a_t^\ell,0)}.
\]
The full posterior factorizes across coordinates given \((x_t,x_0)\).

The paper’s main theoretical insight is that the reverse target induced by a fixed forward process,
\[
q_\phi(x_{t-1}\mid x_t)=\mathbb{E}_{q_\phi(x_0\mid x_t)}[q_\phi(x_{t-1}\mid x_t,x_0)],
\]
is generally not factorized. For any discrete vector \(y\), the best factorized approximation in KL is obtained by matching the marginals, and the minimum KL equals the total correlation:
\[
\min_{p \text{ factorized}} D_{\mathrm{KL}}\!\left(q \,\middle\|\, \prod_i p_i\right)
=
D_{\mathrm{KL}}\!\left(q \,\middle\|\, \prod_i q(y^i)\right)
=
TC(q).
\]
Applied to \(q_\phi(x_{t-1}\mid x_t)\), this means that the irreducible gap to the factorized reverse family is exactly the total correlation of the target reverse distribution. FLDD learns \(\phi\) to reduce that total correlation for a fixed number of steps \(T\), thereby making few-step generation feasible without changing the reverse sampler.

Training remains variational. The main diffusion loss is
\[
L(\theta,\phi)=
\mathbb{E}_{p_{\mathrm{data}}(x_0)}
\left[
\sum_{t=1}^T
\mathbb{E}_{q_\phi(x_t\mid x_0)}
\,
\mathrm{KL}\!\left(
q_\phi(x_{t-1}\mid x_t,x_0)\,\|\, p_\theta(x_{t-1}\mid x_t)
\right)
\right].
\]
Optimization uses a Concrete warm-up followed by an unbiased REINFORCE estimator for \(\phi\), while \(\theta\) is trained by standard backpropagation. Sampling is unchanged: draw \(x_T\) from the prior, then sample each reverse step in parallel from the factorized categorical model.

## 4. Representative instantiations across discrete domains

The exact discrete-state theory of Blackout Diffusion provides one foundational instantiation. Its forward corruption is a pure-death process per pixel or channel,
\[
m \xrightarrow{\gamma m} m-1,
\]
with \(\gamma = 1\) by time scaling, and per-coordinate solution
\[
X_t \sim \mathrm{Binom}(X_0,e^{-t}), \qquad
p_{(m,t)\vert(o,0)} = \binom{o}{m} e^{-mt}(1-e^{-t})^{o-m}.
\]
As \(t \to \infty\), all mass concentrates at \(0\), so the prior is the singular blackout state. The reverse process is birth-only, and training uses exact likelihood-derived losses rather than a variational bound. On CIFAR-10, the reported scores include FID \(4.77\) and IS \(9.01\) for instantaneous loss with a Binomial bridge; on Binarized MNIST the reported FID is \(0.02\), and on CelebA-64 it is \(3.21\) [2305.11089].

MELD instantiates FLDD for molecular graphs by learning per-element masking schedules for atoms and bonds. Standard masked diffusion with a uniform schedule is reported to suffer from a state-clashing problem in which distinct molecules collapse into a common masked state, creating multimodal posteriors that are difficult for a factorized unimodal reverse model to learn [2505.16790]. MELD therefore replaces an element-agnostic schedule by learned \(\beta_{t,\phi}^i\) and \(\beta_{t,\phi}^{ij}\), generated by a scheduling network
\[
f_\phi(h,\ell)=\mathrm{MLP}_\psi(W_\ell h)\in \mathbb{R}^K,
\]
with element embeddings used to break symmetries. The learned schedule is parameterized through
\[
\bar y_t^i=\frac{\sum_{k=1}^K w_k^i t^k}{\sum_{k=1}^K w_k^i}, \qquad
\gamma_t^i=\bar y_t^i(\gamma_{\max}-\gamma_{\min})+\gamma_{\min}, \qquad
\alpha_{t,\phi}^i=\sigma(-\gamma_t^i),
\]
with \(\gamma_{\max}=5\) and \(\gamma_{\min}=-13\). Training jointly optimizes \((\theta,\phi)\) through a continuous-time weighted cross-entropy objective and uses Straight-Through Gumbel-Softmax to maintain gradient flow through the discrete forward noising path. On ZINC250K, the paper reports validity \(93.21\) for MELD, compared with \(14.58\) for a vanilla MDM with cosine schedule and \(27.78\) for a vanilla MDM with polynomial schedule.

NFDM realizes the learned-forward principle for text in latent space rather than by learning an explicit categorical kernel. A neural flow defines learned forward marginals
\[
z_t = F_\phi(\epsilon,t,E_\phi(x_0))
= \mu_\phi(E_\phi(x_0),t)+\sigma_\phi(E_\phi(x_0),t)\epsilon,
\qquad \epsilon \sim \mathcal{N}(0,I),
\]
thereby learning a multivariate forward process from data [2601.16220]. The learned latent diffusion induces a discrete process through a decoder \(p_\phi(x\mid z_0)\), and training minimizes a simulation-free variational upper bound consisting of reconstruction, diffusion, and prior terms. On ROCstories, NFDM reports \(3.12 \pm 0.05\) bpc versus \(3.05\) bpc for a similarly sized autoregressive GPT-J baseline, while a Diffusion-LM reimplementation reports \(5.94 \pm 0.41\) bpc.

These examples show that FLDD can appear as an exact CTMC with a learnable generator, as learnable absorbing-mask schedules in graphs, or as a learned latent-space trajectory for discrete text. The shared principle is forward adaptation to the statistical structure and inductive constraints of the reverse model.

## 5. Forward-thinking FLDD and learned relay representations

A distinct but related usage of FLDD appears in "Learned Relay Representations for Forward-Thinking Discrete Diffusion Models" [2605.22967]. In that work, FLDD is defined as relay-enabled discrete diffusion trained end-to-end via truncated backpropagation through time. The forward-learning component is not an explicit learned Markov noising kernel; instead, the model learns how to propagate latent information forward across denoising rounds.

The setting is masked diffusion for sequences. Standard masked diffusion models iteratively refine a partially masked sequence but discard hidden computation between steps. Relay adds a continuous per-token channel \(h_k\) that is carried across refinement steps:
\[
h_{k+1}
=
f_\theta\!\big(\mathrm{emb}(x_{t_k}) + R_\phi(h_k)\big),
\qquad
\ell_k=\mathrm{unemb}(h_{k+1}).
\]
In the Sudoku-scale implementation, \(R_\phi\) is an affine LayerNorm applied to \(h_k\); in the Fast-dLLM v2 adaptation, the relay LayerNorm is initialized with \(\gamma=0,\beta=0\) so that training begins near zero injection.

Training performs an on-policy rollout of length \(K\), with teacher forcing on the committed tokens and cross-entropy accumulated only over masked positions:
\[
\mathcal{L}(\theta)=
\mathbb{E}_{x_0,\xi_{0:K-1}}
\Bigg[
\sum_{k=0}^{K-1}
\sum_{i\in \mathcal{M}(x_{t_k})}
-\log p_\theta^i(x_0^i \mid x_{t_k},h_k)
\Bigg].
\]
Gradients propagate through the relay path but not through the unmasking policy \(u\). The adjoint recurrence used for truncated BPTT is
\[
\lambda_k
=
(\partial_{h_k}\ell_k)^\top \nabla_{\ell_k}L_k
+
(\partial_{h_k}h_{k+1})^\top \lambda_{k+1},
\qquad
\lambda_K=0.
\]

The reported empirical results are explicitly framed as an accuracy-latency trade-off. On Sudoku-Extreme at \(\tau=0.15\), Relay (full, tied) reaches \(62.67\%\) exact match at mean NFE \(7.43\), compared with \(58.42\%\) for Relay (sg), \(38.70\%\) for Rollout, and \(20.27\%\) for MLM. In the Fast-dLLM v2 scaling experiment, Relay (full) reports HumanEval Base \(42.1\%\), Plus \(37.2\%\), NFE \(88.3\), and MBPP Base \(46.6\%\), Plus \(41.5\%\), NFE \(78.8\). The paper also states that relay-enabled inference reduces latency by up to \(32\%\) relative to vanilla supervised fine-tuning and that peak memory with \(K=2\) BPTT is approximately \(20.1\) GiB versus approximately \(21.2\) GiB for vanilla SFT.

This usage of FLDD is terminologically broader than the explicit learned-noising formulation of [2605.18204]. A plausible implication is that the term is beginning to cover any discrete diffusion system in which future denoising steps are improved by learned forward-side state design, whether that state is a noising distribution, a relay memory, or a latent flow.

## 6. Empirical profile, design trade-offs, and limitations

Across the cited papers, FLDD-style methods are motivated by a common bottleneck: the reverse model is often constrained to be factorized, unimodal, or otherwise computationally efficient, while a fixed forward process induces reverse targets with stronger dependencies or multimodality than the reverse parameterization can represent. The learned forward is used to reduce that mismatch [2605.18204], to avoid state-clashing [2505.16790], to fit language trajectories more closely in latent space [2601.16220], or to amortize computation across denoising rounds [2605.22967].

The reported empirical pattern is consistent. In the explicit FLDD paper, text quality degrades gracefully as the number of steps is reduced from \(T=100\) to \(T=10\): on ROCStories, MAUVE drops from \(0.538\) to \(0.511\), while on QM9 the reported metrics remain strong at \(T=10\), with Valid \(99.08\%\), Unique \(99.95\%\), and FCD \(0.385\) [2605.18204]. In MELD, learning the forward schedule transforms a setting where vanilla masked diffusion reports ZINC250K validity of \(14.58\) or \(27.78\) into one with reported validity \(93.21\), and in conditional polymer generation the reported MAE is \(0.769\), compared with \(0.921\) for the next best GraphDiT baseline [2505.16790]. In NFDM, learned forward trajectories markedly tighten the likelihood gap to autoregressive language models, but the best quality still uses \(2000\) denoising steps and few-step generation degrades rapidly without trajectory straightening [2601.16220]. In Relay, the gain is not a tighter likelihood bound but a shift in the performance-latency Pareto frontier through hidden-state reuse [2605.22967].

Several practical constraints recur. Exact discrete-state reversal requires \(\pi_t(i)>0\) on the training support, so absorbing-terminal priors such as blackout must be handled at finite times rather than literally at \(T=\infty\) [2305.11089]. Large state spaces motivate factorized or sparse \(Q_\theta\), local jump neighborhoods, and matrix-exponential approximations such as Krylov subspace methods. In reverse sampling, \(\tau\)-leaping with Poisson samplers offers an approximation, while exact bridges are available only for special processes such as Blackout Diffusion. In molecular FLDD, Straight-Through Gumbel-Softmax is needed so that the learned forward process remains differentiable through discrete samples [2505.16790]. In non-Markovian FLDD, REINFORCE is required after the Concrete warm-up, which introduces optimization variance [2605.18204]. In Relay, longer truncation horizons than \(K=2\) are identified as a possible source of memory and stability challenges [2605.22967].

The main point of disagreement across the literature is not whether learning the forward process helps, but what exactly should be learned. The spectrum ranges from exact transition generators and posteriors, to per-element masking rates, to latent continuous trajectories, to relay memories across denoising steps. This suggests that FLDD is evolving into a general research program: discrete diffusion becomes substantially more flexible when the forward path is treated as an object of learning rather than as a fixed schedule inherited from the earliest diffusion constructions.

Source: https://www.emergentmind.com/topics/forward-learned-discrete-diffusion-fldd