---
title: Hierarchical Discrete Diffusion Model
url: https://www.emergentmind.com/topics/hierarchical-discrete-diffusion-model
type: topic
---

# Hierarchical Discrete Diffusion Model

Hierarchical discrete diffusion models are denoising-based generative models in which the underlying variables are discrete and the generative process is explicitly organized across levels of abstraction, scales, or structured partitions. In the literature represented here, hierarchy appears in several forms: clean-to-group-to-mask state spaces, multi-level residual vector quantization token streams, coarse-to-fine graph constructions, semantic vocabularies, and discrete-time latent trajectories across scales. Representative instances include Hierarchical Koopman Diffusion for images, HiCoDiT for video-to-speech, ADDSE/RQDiT for speech enhancement, MolHIT for molecular graphs, HDLM for language modeling, and hierarchical discrete flow matching for graphs [2510.12220] [2604.15923] [2602.22417] [2602.17602] [2510.08632] [2604.00236].

## 1. Definition and conceptual scope

In the narrowest sense, a hierarchical discrete diffusion model extends flat discrete diffusion by replacing a single categorical state space with multiple semantic or structural levels. MolHIT makes this explicit by enlarging the token space from clean states to mid-level states and then to a mask state, so that the forward process becomes a two-stage corruption chain \( \mathcal{S}_0 \to \mathcal{S}_1 \to \mathcal{S}_2 \) rather than a direct jump from clean token to mask [2602.17602]. HDLM adopts the same principle for language: words map surjectively to cluster tokens and then to a mask token, so the forward process performs semantic abstraction and the reverse process performs “next semantic scale prediction” [2510.08632].

A second usage places hierarchy in the discrete representation rather than only in the transition kernel. HiCoDiT models 12 RVQ codec levels, with low-level tokens encoding coarse speaker-aware semantics and high-level tokens capturing fine-grained prosody, while ADDSE models a \(L \times D\) grid of RVQ codes with an architecture that separates temporal and depth structure [2604.15923] [2602.22417]. In these systems, diffusion remains discrete, but the hierarchy is inherited from the codec itself.

A third usage couples hierarchy to multiscale structure in the generated object. Hierarchical discrete flow matching for graph generation defines a sequence of coarse graphs and fine graphs, with generation factorized across levels as \(p(\mathcal{G})=\prod_{\ell=0}^{L} p_{\theta_\ell}(\mathcal{G}^\ell \mid \mathcal{H}^\ell)\), where each \(\mathcal{H}^\ell\) is a spanning supergraph induced by the next coarser level [2604.00236]. Hierarchical Koopman Diffusion is presented as exactly an instance of what can be called a hierarchical discrete diffusion model because it models discrete-time diffusion evolution in a Koopman-lifted latent space and does so hierarchically across spatial scales [2510.12220].

A separate line of work locates hierarchy entirely in the forward process. “Improving Discrete Diffusion Models via Structured Preferential Generation” introduces a structured masking process in which token categories are destroyed in an information-based order, so the reverse model preferentially regenerates some categories before others [2405.17889]. This suggests that hierarchy in discrete diffusion need not require a multilevel backbone; it can be induced by the corruption schedule alone.

## 2. Mathematical constructions

The most common starting point is absorbing or masking diffusion on discrete variables. HiCoDiT uses the SEDD framework, where each token dimension follows a continuous-time discrete Markov chain with a special absorbing token \([MASK]\), and the reverse model learns concrete probability ratios via Denoising Score Entropy [2604.15923]. ADDSE uses absorbing discrete diffusion with token space \(\{1,\dots,K+1\}\), where the \((K+1)\)-th symbol is a mask token \(M\), and the reverse process becomes repeated masked-token infilling trained by denoising cross-entropy [2602.22417]. Prism operates on the same masked discrete diffusion language-model setting, with per-token transition \(q_{t|0}(z_{t,i}\mid z_{0,i})=\mathrm{Cat}(z_{t,i}; \alpha_t \mathbf{e}(z_{0,i}) + (1-\alpha_t)\mathbf{e}_m)\) [2602.01842].

MolHIT generalizes this flat setup by defining a forward kernel
\[
Q_t = \alpha_t I + (\beta_t - \alpha_t) Q^{(1)} + (1-\beta_t) Q^{(2)},
\]
where \(Q^{(1)}\) projects clean states to group states and \(Q^{(2)}\) projects to the mask state [2602.17602]. The semigroup-consistent family \(Q_{t|s}\) preserves closed-form marginals and posterior expressions, so the model remains a valid discrete diffusion process while traversing intermediate chemically meaningful abstractions.

HDLM uses a continuous-time Markov chain over a hierarchical vocabulary. For a token \(x\), the forward marginal is
\[
q_t(z_t\mid x)=\mathrm{Cat}\!\bigl(z_t;\,\alpha_t \mathbf{x} + \beta_{t,c}\mathbf{c}(\mathbf{x}) + \beta_{t,m}\mathbf{m}\bigr),
\]
with \(\beta_{t,c}+\beta_{t,m}=1-\alpha_t\) [2510.08632]. The reverse model predicts a word-level distribution \(\mathbf{x}_\theta\), while cluster-level posteriors are obtained by Bayesian marginalization through the surjective map \(\Gamma\). The resulting ELBO decomposes into a cluster-level cross-entropy term for mask inputs and a within-cluster word-level cross-entropy term for cluster inputs.

Hierarchical discrete flow matching replaces explicit forward Markov chains with a continuous-time probability path
\[
p_t(z^i \mid z_0^i, z_1^i)=\alpha(t)\,[z^i=z_1^i] + (1-\alpha(t))\,[z^i=z_0^i],
\]
and a probability velocity
\[
u_t^i(z^i, Z_t)=\frac{\dot{\alpha}(t)}{1-\alpha(t)}\left(p_{1|t}(z^i\mid Z_t)-[z^i=z_t^i]\right),
\]
so generation becomes a discrete denoising process implemented through flow matching rather than through a long discrete-time chain [2604.00236].

Hierarchical Koopman Diffusion departs from categorical-state corruption but still instantiates a discrete-time hierarchical diffusion viewpoint. It encodes an image into multiscale latent observables \( \{z_t^{(l)}\}_{l=1}^L \) and assumes linear latent dynamics
\[
\frac{d z_t^{(l)}(i,j)}{dt}=A^{(l)}(i,j) z_t^{(l)}(i,j),
\qquad
z_{t+\Delta t}=e^{A\Delta t} z_t,
\]
so the entire denoising trajectory has a closed-form evolution operator at each scale [2510.12220].

## 3. Hierarchy mechanisms and representative architectures

The literature suggests that no single hierarchy mechanism dominates. Some models place hierarchy in the state space, some in the tokenization, some in graph coarsening, and some in inference over denoising trajectories.

| Model | Discrete object | Hierarchy mechanism |
|---|---|---|
| HKD [2510.12220] | Diffusion trajectory states in Koopman latent space | Scale-specific Koopman subspaces and operators across spatial resolutions |
| HiCoDiT [2604.15923] | 12 RVQ codec token streams | Low-level \(r_{1:2}\) and high-level \(r_{3:12}\) blocks with dual-scale conditioning |
| ADDSE / RQDiT [2602.22417] | \(L \times D\) RVQ code grid | Frame-DiT over time and Depth-DiT over RVQ depth |
| MolHIT [2602.17602] | Atom and bond categories | Clean \(\to\) chemical-group \(\to\) mask hierarchy plus decoupled atom encoding |
| HDLM [2510.08632] | Word tokens | Word \(\to\) cluster \(\to\) mask semantic vocabulary |
| HDFM [2604.00236] | Node and edge labels | Coarse graph \(\to\) spanning supergraph \(\to\) fine graph factorization |

A distinct architecture-free variant is structured preferential generation, where the hierarchy is a token-type ordering in the forward process. Different categories are masked at different times according to an information-based schedule, so the reverse process regenerates some token types earlier than others [2405.17889].

Prism occupies a different position. It is built on masked discrete diffusion language models, but it is not a new generative architecture; it is a test-time scaling framework layered on top. Its hierarchy lies in inference: exploration, thinning, and refinement stages over denoising trajectories, with pruning, partial remasking, and self-verification [2602.01842]. This is still relevant because it shows that hierarchy can enter discrete diffusion at decoding time rather than only in training-time model design.

## 4. Objectives, samplers, and controllability

Despite the diversity of forward processes, the training objectives often reduce to denoising losses with structure-aware conditioning. HKD is trained from trajectories produced by a well-trained diffusion teacher and uses a reconstruction loss for the one-step map together with a trajectory consistency loss
\[
\mathcal{L}=\mathcal{L}_{t\text{-consist}}+\mathcal{L}_{\text{recon}},
\]
which forces any teacher state \(x_t\) to evolve to the same target clean image \(x_\epsilon\) after Koopman propagation [2510.12220].

HiCoDiT uses multi-level Denoising Score Entropy,
\[
\mathcal{L}_{\text{score}}=\sum_{i=1}^{12}\mathcal{L}_{\text{DSE}}(\bm{x}^{r_i},t,\bm{c}),
\]
plus an identity alignment term
\[
\mathcal{L}_{\text{id}}=\ell_1(\bm{c}_{\text{id}},\bm{c}_{\text{GE2E}}),
\]
and total loss
\[
\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{score}}+\lambda \mathcal{L}_{\text{id}},\qquad \lambda=100.
\]
It also applies conditioning dropout so that lip, identity, and emotion inputs can be dropped during training to enable predictor-free guidance at inference [2604.15923].

ADDSE uses noisy-speech-conditioned denoising cross-entropy
\[
\mathcal{L}_{\text{DCE}}(\theta)=
\mathbb{E}\left[\sum_{(i,j):\, c_\lambda^{i,j}=M}
-\frac{1}{\lambda}\log q_\theta\big(c^{i,j}\mid \mathbf{c}_\lambda,\tilde{\mathbf{c}}\big)\right],
\]
with a time-independent network that predicts \(p_0(\cdot \mid \mathbf{c}_t,\tilde{\mathbf{c}})\). Because the network is time-independent, predictions can be reused when no new code becomes unmasked, reducing average network function evaluations; the paper reports average NFE \(\approx 545\) for \(N_{\text{steps}}=1024\) [2602.22417].

MolHIT derives a continuous-time NELBO for HDDM, but in practice it trains with the standard DiGress-style cross-entropy on clean nodes and edges. Its distinctive feature appears at sampling time through the Project-and-Noise sampler: sample a clean graph \(\hat{G}_0\) from the model’s logits, then re-apply the forward kernel to obtain the next noisy graph. Temperature scaling and top-\(p\) truncation are applied to atom predictions, and classifier-free guidance is used for property-conditioned generation [2602.17602].

HDLM derives closed-form ELBOs in which the mask-level term is a cluster classification loss and the cluster-level term is a within-cluster word classification loss. Practical training then adds force-transition constraints during decoding, loss-weight clipping for numerical stability, and stochastic cluster perturbations with probability \(\xi<1\) to improve self-correction under mismatched contexts [2510.08632].

HDFM uses simple cross-entropy over clean node and edge labels at each hierarchy level,
\[
\mathcal{L}_\theta
= \mathbb{E}\left[\gamma \sum_i -\log p_\theta(x_i \mid Z_t,\mathcal{H}^\ell)
+ (1-\gamma)\sum_j -\log p_\theta(a_j \mid Z_t,\mathcal{H}^\ell)\right],
\]
which is notable because the hierarchical discrete diffusion effect comes primarily from the coarse-to-fine graph factorization and sparse supergraph conditioning rather than from a complicated loss [2604.00236].

## 5. Empirical regimes and applications

In image generation, HKD is positioned as a one-step model that retains trajectory access. On CIFAR-10, it reports FID-50k \(3.30\) in one step, compared with \(3.77\) for DMD and \(3.55\) for CD (LPIPS), while on FFHQ \(64\times 64\) it reports \(5.70\) in one step versus \(5.99\) for ECM. Its qualitative contribution is equally central: because the Koopman dynamics are explicit, intermediate states can be reconstructed analytically, spectral modes can be masked or injected, and “frequency-aware” editing can modify fine details while leaving coarse structure intact [2510.12220].

In speech and audio, hierarchical discrete diffusion is closely tied to residual vector quantization. HiCoDiT exploits the hierarchy of 12 codec levels, aligning lip motion and identity with low-level blocks and emotion with high-level blocks through dual-scale adaptive instance layer normalization, and the paper reports improved fidelity and expressiveness over baselines in video-to-speech generation together with stronger out-of-domain performance on film data [2604.15923]. ADDSE uses a hierarchical discrete latent space for speech enhancement, where non-intrusive metrics plateau around \(8\)–\(16\) diffusion steps and performance is particularly strong at low signal-to-noise ratios [2602.22417].

In language modeling, HDLM reports consistently lower validation and generative perplexity than masked or flat discrete diffusion baselines; for example, HDLM-small-64 achieves validation perplexity \(\le 23.36\) and generative perplexity \(144.2\), compared with MDLM-small at \(\le 27.39\) and \(163.7\), and HDLM-base-128 reaches validation perplexity \(\le 19.22\) and generative perplexity \(139.9\) [2510.08632]. Prism addresses a different problem—test-time scaling for masked diffusion language models—and shows that hierarchical trajectory search can match or exceed best-of-16 performance with much lower NFE; for LLaDA 8B Instruct on GSM8K, Prism with \(N=16, S=2, K=8\) reaches \(85.30\%\) at \(1048\) NFE, compared with best-of-16 at \(87.50\%\) and \(4096\) NFE [2602.01842].

In graph and molecular generation, the hierarchy is both structural and computational. HDFM reports large speed gains by combining sparse hierarchical supergraphs with discrete flow matching: on ZINC250k, HDFM requires \(394\) seconds at \(128\) NFE and \(168\) seconds at \(32\) NFE, versus \(5517\) seconds for DiGress at \(1000\) NFE and \(9826\) seconds for SparseDiff at \(2000\) NFE [2604.00236]. MolHIT, which explicitly names Hierarchical Discrete Diffusion Models as its foundation, reports MOSES validity \(99.1\%\), quality \(94.2\), and FCD \(1.03\), and further shows strong downstream performance in multi-property guided generation and scaffold extension [2602.17602].

## 6. Related research, misconceptions, and outlook

A recurring misconception is to equate hierarchy exclusively with multiresolution image pyramids. The surveyed literature places hierarchy over token categories, RVQ levels, graph coarsenings, semantic vocabularies, chemical groupings, and even inference trajectories. Another is to treat every hierarchical diffusion model as discrete. Several influential neighbors are hierarchical but continuous: Hierarchical Diffuser uses Gaussian diffusion for temporally abstract planning with jumpy sub-goals [2401.02644]; SIHD builds multi-scale continuous trajectory diffusions from structural entropy and state communities in offline RL [2509.21942]; Groupwise Diffusion Model uses continuous flow matching with non-overlapping group schedules and a frequency-domain hierarchy [2310.01400]; and Diffuse-TreeVAE conditions a continuous DDPM on a learned latent tree with a discrete leaf index [2407.06124].

These adjacent works matter because they supply design templates even when the noise process is not discrete. Groupwise Diffusion Model shows that grouping and order of generation are first-class design choices, and its frequency-domain version demonstrates a coarse-to-fine hierarchy in which low-frequency groups encode high-level semantics while higher bands refine local details [2310.01400]. The phase-transition analysis of diffusion on a Random Hierarchy Model goes further: it studies a fully discrete hierarchical latent tree with Gaussian leaf diffusion and finds a critical time beyond which high-level features such as class labels abruptly become unrecoverable while low-level features decay smoothly. This provides a theoretical explanation for why diffusion time can behave like a scale parameter in hierarchical data [2402.16991].

The main limitations are equally diverse. Some models depend on hand-designed or pretrained hierarchies, as in MolHIT’s chemically curated grouping and HDLM’s embedding-based semantic clustering [2602.17602] [2510.08632]. Others rely on teacher trajectories or two-stage pipelines, as in HKD and Diffuse-TreeVAE [2510.12220] [2407.06124]. Inference wrappers such as Prism introduce hyperparameter sensitivity in pruning windows, survivor counts, and remasking policies [2602.01842]. Hierarchical graph generators can become harder to train in low-data regimes, and continuous analogues such as SIHD and Hierarchical Diffuser still face the usual offline-RL dependence on dataset support [2509.21942] [2401.02644].

The outlook suggested by these papers is toward richer learned hierarchies, not merely deeper ones. HDLM explicitly points to learned semantic mappings and more than one intermediate level; MolHIT highlights applicability beyond molecules to text and images; HKD proposes higher-resolution images, videos, and 3D data; and HDFM suggests that coarse-to-fine discrete denoising on sparse structures can scale to broader discrete domains [2510.08632] [2602.17602] [2510.12220] [2604.00236]. A plausible implication is that “hierarchical discrete diffusion model” is best understood not as a single architecture, but as a general design principle: denoising should proceed through semantically or structurally meaningful intermediate discrete states rather than through a flat categorical space alone.

Source: https://www.emergentmind.com/topics/hierarchical-discrete-diffusion-model