FLDD: Forward-Learned Discrete Diffusion

• FLDD is a diffusion framework that learns the forward noising process to improve the design of reverse denoising targets.
• It covers various methods, including learnable forward marginals, per-element masking schedules, and latent trajectories for text.
• FLDD addresses the mismatch between fixed forward processes and efficient factorized reverse models, enhancing generation performance.

Searching arXiv for the cited FLDD-related papers to ground the article in current arXiv records. arXiv search: "(Santos et al., 2023) Blackout Diffusion Generative Diffusion Models in Discrete-State Spaces" Forward-Learned Discrete Diffusion (FLDD) denotes a class of diffusion methods in which the forward corruption process in a discrete or discretized domain is itself parameterized and learned, rather than fixed a priori. Across recent arXiv work, the term spans several closely related uses: a non-Markovian discrete diffusion framework with learnable forward marginals and posteriors (Bartosh et al., 18 May 2026), an exact discrete-state perspective in which a learned generator $Q_\theta$ is supported by closed-form time reversal (Santos et al., 2023), and task-specific realizations such as relay-enabled masked diffusion for language (Rozonoyer et al., 21 May 2026), per-element molecular masking schedules (Seo et al., 22 May 2025), and learned latent forward trajectories for text (Midavaine et al., 7 Jan 2026). This suggests that FLDD is best understood as a broader design principle—learning how data are noised so that denoising becomes easier, more accurate, or more efficient—rather than as a single canonical algorithm.

1. Terminological scope and conceptual identity

In the narrowest sense, FLDD refers to the method introduced in "Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster" (Bartosh et al., 18 May 2026). There, the central claim is that conventional discrete diffusion uses a fixed forward process that induces a reverse target that is typically not factorized, whereas the generative model is usually parameterized by factorized reverse conditionals for efficiency. FLDD addresses this mismatch by learning the forward noising process while leaving the reverse model in the same efficient factorized family.

A broader usage appears in several adjacent lines of work. "Blackout Diffusion: Generative Diffusion Models in Discrete-State Spaces" develops an exact theory of discrete-state forward and reverse diffusion and explicitly states that this framework immediately supports learning $Q$ or $K_t$ in an FLDD-style construction (Santos et al., 2023). "Learned Relay Representations for Forward-Thinking Discrete Diffusion Models" uses FLDD as a direct synonym for relay-enabled, forward-thinking discrete diffusion trained via truncated BPTT, emphasizing latent state propagation across denoising steps rather than explicit learning of a Markov generator (Rozonoyer et al., 21 May 2026). "Learning Flexible Forward Trajectories for Masked Molecular Diffusion" presents MELD as an FLDD instance in which per-element corruption schedules are learned to reduce state-clashing in molecular graphs (Seo et al., 22 May 2025). "Towards Latent Diffusion Suitable For Text" realizes the same principle in latent space through Neural Flow Diffusion Models, where the forward process is learned from data and induces a discrete process through decoding (Midavaine et al., 7 Jan 2026).

A common misconception is that FLDD names a single universally agreed architecture. The literature instead uses the term for a family of methods united by one design move: the forward trajectory is no longer treated as a fixed nuisance schedule. Another common misconception is that learning the forward process requires abandoning parallel, factorized reverse sampling. The 2026 FLDD formulation is explicit that the reverse model remains factorized and the sampler is unchanged (Bartosh et al., 18 May 2026).

2. Exact discrete-state foundations

The most general mathematical foundation in the supplied corpus is the exact discrete-state framework of Blackout Diffusion. The per-coordinate state space is a finite set $\Omega = \{0,1,\ldots,M\}$, the full data space is $\Omega^N$, and the forward process may be written either as a discrete-time Markov chain $x_{k+1} \sim K_k(\cdot \mid x_k)$ or as a continuous-time Markov chain with generator $Q$ (Santos et al., 2023). In continuous time, the forward marginals satisfy the Kolmogorov forward equation

$$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$

For nearest-neighbor transitions, the forward Master equation can be written in a step-operator form, and the framework generalizes to arbitrary banded or nonlocal transitions by decomposing $Q$ into banded components. The resulting perspective replaces Gaussian additive noise by discrete jump noise, which may be local, global, structured, masking-based, or absorbing.

The central exact reversal formula is the continuous-time reverse generator

$$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$

with diagonals determined by negative row sums. In matrix form,

$Q$0

The discrete-time analogue is the Bayes-form reversal

$Q$1

This exact reversal is the discrete counterpart of the reverse-time correction in continuous Gaussian diffusion. The Blackout Diffusion analysis relates the jump-process correction to the score term in the reverse SDE, and identifies a discrete-state score

$Q$2

In the dense-state limit, this approaches $Q$3 up to scaling.

The exactness of this formulation matters for FLDD because it yields closed-form reverse targets once the forward process is specified. The required reversibility condition is that $Q$4 on the relevant support; otherwise reverse rates involving state $Q$5 are undefined. Detailed balance is not required, although it is compatible with the framework.

3. Canonical FLDD: learnable forward marginals, posteriors, and total correlation

The 2026 FLDD paper formalizes the forward-learning idea in a deliberately non-Markovian way (Bartosh et al., 18 May 2026). The data lie in $Q$6, and the reverse model remains the standard factorized reverse chain

$Q$7

The forward process is replaced by a learnable family consisting of forward marginals $Q$8 and forward posteriors $Q$9, with joint factorization

$K_t$0

The forward marginals are factorized across coordinates,

$K_t$1

but each $K_t$2 depends on the full $K_t$3, not only on $K_t$4. This is the key mechanism by which FLDD can perform data-dependent destruction of information while retaining a factorized form.

Consistency between marginals and posteriors is enforced through a per-coordinate maximal coupling. For coordinate $K_t$5, let $K_t$6 and $K_t$7 with $K_t$8. The conditional is

$K_t$9

where

$\Omega = \{0,1,\ldots,M\}$0

The full posterior factorizes across coordinates given $\Omega = \{0,1,\ldots,M\}$1.

The paper’s main theoretical insight is that the reverse target induced by a fixed forward process,

$\Omega = \{0,1,\ldots,M\}$2

is generally not factorized. For any discrete vector $\Omega = \{0,1,\ldots,M\}$3, the best factorized approximation in KL is obtained by matching the marginals, and the minimum KL equals the total correlation: $\Omega = \{0,1,\ldots,M\}$4 Applied to $\Omega = \{0,1,\ldots,M\}$5, this means that the irreducible gap to the factorized reverse family is exactly the total correlation of the target reverse distribution. FLDD learns $\Omega = \{0,1,\ldots,M\}$6 to reduce that total correlation for a fixed number of steps $\Omega = \{0,1,\ldots,M\}$7, thereby making few-step generation feasible without changing the reverse sampler.

Training remains variational. The main diffusion loss is

$\Omega = \{0,1,\ldots,M\}$8

Optimization uses a Concrete warm-up followed by an unbiased REINFORCE estimator for $\Omega = \{0,1,\ldots,M\}$9, while $\Omega^N$0 is trained by standard backpropagation. Sampling is unchanged: draw $\Omega^N$1 from the prior, then sample each reverse step in parallel from the factorized categorical model.

4. Representative instantiations across discrete domains

The exact discrete-state theory of Blackout Diffusion provides one foundational instantiation. Its forward corruption is a pure-death process per pixel or channel,

$\Omega^N$2

with $\Omega^N$3 by time scaling, and per-coordinate solution

$\Omega^N$4

As $\Omega^N$5, all mass concentrates at $\Omega^N$6, so the prior is the singular blackout state. The reverse process is birth-only, and training uses exact likelihood-derived losses rather than a variational bound. On CIFAR-10, the reported scores include FID $\Omega^N$7 and IS $\Omega^N$8 for instantaneous loss with a Binomial bridge; on Binarized MNIST the reported FID is $\Omega^N$9, and on CelebA-64 it is $x_{k+1} \sim K_k(\cdot \mid x_k)$0 (Santos et al., 2023).

MELD instantiates FLDD for molecular graphs by learning per-element masking schedules for atoms and bonds. Standard masked diffusion with a uniform schedule is reported to suffer from a state-clashing problem in which distinct molecules collapse into a common masked state, creating multimodal posteriors that are difficult for a factorized unimodal reverse model to learn (Seo et al., 22 May 2025). MELD therefore replaces an element-agnostic schedule by learned $x_{k+1} \sim K_k(\cdot \mid x_k)$1 and $x_{k+1} \sim K_k(\cdot \mid x_k)$2, generated by a scheduling network

$x_{k+1} \sim K_k(\cdot \mid x_k)$3

with element embeddings used to break symmetries. The learned schedule is parameterized through

$x_{k+1} \sim K_k(\cdot \mid x_k)$4

with $x_{k+1} \sim K_k(\cdot \mid x_k)$5 and $x_{k+1} \sim K_k(\cdot \mid x_k)$6. Training jointly optimizes $x_{k+1} \sim K_k(\cdot \mid x_k)$7 through a continuous-time weighted cross-entropy objective and uses Straight-Through Gumbel-Softmax to maintain gradient flow through the discrete forward noising path. On ZINC250K, the paper reports validity $x_{k+1} \sim K_k(\cdot \mid x_k)$8 for MELD, compared with $x_{k+1} \sim K_k(\cdot \mid x_k)$9 for a vanilla MDM with cosine schedule and $Q$0 for a vanilla MDM with polynomial schedule.

NFDM realizes the learned-forward principle for text in latent space rather than by learning an explicit categorical kernel. A neural flow defines learned forward marginals

$Q$1

thereby learning a multivariate forward process from data (Midavaine et al., 7 Jan 2026). The learned latent diffusion induces a discrete process through a decoder $Q$2, and training minimizes a simulation-free variational upper bound consisting of reconstruction, diffusion, and prior terms. On ROCstories, NFDM reports $Q$3 bpc versus $Q$4 bpc for a similarly sized autoregressive GPT-J baseline, while a Diffusion-LM reimplementation reports $Q$5 bpc.

These examples show that FLDD can appear as an exact CTMC with a learnable generator, as learnable absorbing-mask schedules in graphs, or as a learned latent-space trajectory for discrete text. The shared principle is forward adaptation to the statistical structure and inductive constraints of the reverse model.

5. Forward-thinking FLDD and learned relay representations

A distinct but related usage of FLDD appears in "Learned Relay Representations for Forward-Thinking Discrete Diffusion Models" (Rozonoyer et al., 21 May 2026). In that work, FLDD is defined as relay-enabled discrete diffusion trained end-to-end via truncated backpropagation through time. The forward-learning component is not an explicit learned Markov noising kernel; instead, the model learns how to propagate latent information forward across denoising rounds.

The setting is masked diffusion for sequences. Standard masked diffusion models iteratively refine a partially masked sequence but discard hidden computation between steps. Relay adds a continuous per-token channel $Q$6 that is carried across refinement steps: $Q$7 In the Sudoku-scale implementation, $Q$8 is an affine LayerNorm applied to $Q$9; in the Fast-dLLM v2 adaptation, the relay LayerNorm is initialized with $$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$0 so that training begins near zero injection.

Training performs an on-policy rollout of length $$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$1, with teacher forcing on the committed tokens and cross-entropy accumulated only over masked positions: $$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$2 Gradients propagate through the relay path but not through the unmasking policy $$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$3. The adjoint recurrence used for truncated BPTT is

$$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$4

The reported empirical results are explicitly framed as an accuracy-latency trade-off. On Sudoku-Extreme at $$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$5, Relay (full, tied) reaches $$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$6 exact match at mean NFE $$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$7, compared with $$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$8 for Relay (sg), $$\frac{d}{dt}\,\pi_t = \pi_t Q, \qquad \pi_t = \pi_0 e^{tQ}.$$9 for Rollout, and $Q$0 for MLM. In the Fast-dLLM v2 scaling experiment, Relay (full) reports HumanEval Base $Q$1, Plus $Q$2, NFE $Q$3, and MBPP Base $Q$4, Plus $Q$5, NFE $Q$6. The paper also states that relay-enabled inference reduces latency by up to $Q$7 relative to vanilla supervised fine-tuning and that peak memory with $Q$8 BPTT is approximately $Q$9 GiB versus approximately $$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$0 GiB for vanilla SFT.

This usage of FLDD is terminologically broader than the explicit learned-noising formulation of (Bartosh et al., 18 May 2026). A plausible implication is that the term is beginning to cover any discrete diffusion system in which future denoising steps are improved by learned forward-side state design, whether that state is a noising distribution, a relay memory, or a latent flow.

6. Empirical profile, design trade-offs, and limitations

Across the cited papers, FLDD-style methods are motivated by a common bottleneck: the reverse model is often constrained to be factorized, unimodal, or otherwise computationally efficient, while a fixed forward process induces reverse targets with stronger dependencies or multimodality than the reverse parameterization can represent. The learned forward is used to reduce that mismatch (Bartosh et al., 18 May 2026), to avoid state-clashing (Seo et al., 22 May 2025), to fit language trajectories more closely in latent space (Midavaine et al., 7 Jan 2026), or to amortize computation across denoising rounds (Rozonoyer et al., 21 May 2026).

The reported empirical pattern is consistent. In the explicit FLDD paper, text quality degrades gracefully as the number of steps is reduced from $$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$1 to $$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$2: on ROCStories, MAUVE drops from $$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$3 to $$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$4, while on QM9 the reported metrics remain strong at $$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$5, with Valid $$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$6, Unique $$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$7, and FCD $$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$8 (Bartosh et al., 18 May 2026). In MELD, learning the forward schedule transforms a setting where vanilla masked diffusion reports ZINC250K validity of $$q^*_{ij}(t) = q_{ji}\,\frac{\pi_t(j)}{\pi_t(i)}, \qquad i \neq j,$$9 or $Q$00 into one with reported validity $Q$01, and in conditional polymer generation the reported MAE is $Q$02, compared with $Q$03 for the next best GraphDiT baseline (Seo et al., 22 May 2025). In NFDM, learned forward trajectories markedly tighten the likelihood gap to autoregressive LLMs, but the best quality still uses $Q$04 denoising steps and few-step generation degrades rapidly without trajectory straightening (Midavaine et al., 7 Jan 2026). In Relay, the gain is not a tighter likelihood bound but a shift in the performance-latency Pareto frontier through hidden-state reuse (Rozonoyer et al., 21 May 2026).

Several practical constraints recur. Exact discrete-state reversal requires $Q$05 on the training support, so absorbing-terminal priors such as blackout must be handled at finite times rather than literally at $Q$06 (Santos et al., 2023). Large state spaces motivate factorized or sparse $Q$07, local jump neighborhoods, and matrix-exponential approximations such as Krylov subspace methods. In reverse sampling, $Q$08-leaping with Poisson samplers offers an approximation, while exact bridges are available only for special processes such as Blackout Diffusion. In molecular FLDD, Straight-Through Gumbel-Softmax is needed so that the learned forward process remains differentiable through discrete samples (Seo et al., 22 May 2025). In non-Markovian FLDD, REINFORCE is required after the Concrete warm-up, which introduces optimization variance (Bartosh et al., 18 May 2026). In Relay, longer truncation horizons than $Q$09 are identified as a possible source of memory and stability challenges (Rozonoyer et al., 21 May 2026).

The main point of disagreement across the literature is not whether learning the forward process helps, but what exactly should be learned. The spectrum ranges from exact transition generators and posteriors, to per-element masking rates, to latent continuous trajectories, to relay memories across denoising steps. This suggests that FLDD is evolving into a general research program: discrete diffusion becomes substantially more flexible when the forward path is treated as an object of learning rather than as a fixed schedule inherited from the earliest diffusion constructions.