Sticky Jump Diffusions: A Unifying View of Masked, Continuous, and Hybrid Diffusion

Abstract: We introduce Sticky Jump Diffusions (SJDs), continuous-time Markov processes on $\mathbb R<sup>d$ whose discrete anchors are token embeddings. In forward time, anchors release their mass at a hazard rate and the released mass diffuses in the continuous ambient space; time reversal couples a score-driven SDE with a sticky jump kernel whose rate and destination are fixed by flux balance with the forward law. We estimate the score and the per-anchor reverse hazards from a single denoising classifier via Denoising Hazard Matching, the hazard analogue of denoising score matching, with simulation-free cross-entropy training. SJD recovers masked diffusion, continuous diffusion, and hybrid diffusion as limits. Its reversal explains features that each family treats as given: the mask of masked diffusion carries no evidence about the source token because the unsticking kernel of every anchor collapses to the same absorbing point; the terminal projection of continuous diffusion is required due to the absence of atoms in its forward marginal, without which flux balance yields no reverse jumps; and the update rules of hybrid diffusion (commit rate, destination, and drift) all follow from flux balance rather than from separate design. Beyond these limits, the unsticking kernel becomes a design space: a cross-position blending corrupts each position toward a blend of its neighbors' clean values or embeddings, turning dependency structure such as spatial locality or a constraint graph into an inductive bias of the corruption itself, and improves over the identity-kernel hybrid on CIFAR-10, Text8, and Sudoku.

• The paper introduces Sticky Jump Diffusions as a framework unifying discrete, continuous, and hybrid diffusion models using analytic reverse-time dynamics derived from flux balance.
• It details a novel Denoising Hazard Matching procedure that leverages a single classifier for both reverse-time allocation and hazard rate estimation.
• Empirical results across image, text, and Sudoku tasks demonstrate improved sample quality, stability, and efficiency compared to existing methods.

Sticky Jump Diffusions: A Unifying Framework for Discrete, Continuous, and Hybrid Diffusion Models

Introduction

Sticky Jump Diffusions (SJDs) present a unifying framework for discrete, continuous, and hybrid diffusion models by defining a class of continuous-time jump-diffusion processes on $\mathbb{R}^d$ with a finite anchor set representing embedded tokens. The SJD framework places masked diffusion, continuous diffusion on embeddings, and their hybrids as analytic limits within a single forward-time process. Its generative reverse-time process, rigorously derived from flux balance, employs a score-based SDE in the continuous region and a sticky jump kernel that projects off-anchor mass back onto discrete anchors.

This essay details (1) the construction and mathematical grounding of SJDs, (2) the associated training and inference methods—including the Denoising Hazard Matching (DHM) procedure, (3) connections to existing diffusion literature, and (4) empirical results spanning image, text, and combinatorial data.

The SJD Forward/Reverse Construction

Forward Dynamic: Mass Unsticking and Diffusion

An SJD is defined by a continuous state space $\mathsf{X} = \mathbb{R}^d$ and a countable anchor set $A\subset\mathbb{R}^d$. The process begins at anchors (tokens), which unstick with a prescribed hazard $\vec{\lambda}_t$ into the continuous complement, where standard Itô dynamics operate:

$$d\vec{X}_t = \vec{b}_t(\vec{X}_t)\, dt + g_t d\vec{W}_t,$$

with no forward jumps into anchors.

The forward marginal comprises a Dirac mixture (mass at anchors, decaying according to $\vec{\lambda}_t$) and an absolutely continuous component across the continuous region. For sequence data, the kernel controlling unstick location can depend nonlocally on the entire clean sequence, parameterized as a blending of positional embeddings:

Figure 1: Reverse-time trajectories over each model's path space. Masked diffusion is discrete but gradient-free; continuous diffusion is gradient-based but requires a terminal projection; SJD couples ambient gradients with sticky jumps onto anchors.

Reverse-Time: Generative Sticky Jump Diffusions

The time-reversal of SJD yields a generative process composed of:

• Continuous SDE: Standard score-based denoising dynamics on the continuous region.
• Sticky Jump Kernel: Mass on the continuous region stochastically commits to discrete anchors at rate and allocation analytically determined by forward-process flux balance:

$$J_t(y, \{a\}) = \frac{m_t(a)}{p_t(y)}\, \lambda_t\, r_t(y, a),$$

where $m_t(a)$ is the current anchor mass, and $r_t(y, a)$ is the forward unsticking kernel evaluated at location $y$.

Unified View of Discrete, Continuous, and Hybrid Diffusion

By varying SJD parameters (hazard schedule, unsticking kernel, forward SDE coefficients, and blending structure), classical families of discrete data diffusion arise as special cases:

• Masked diffusion as the degenerate-kernel limit—anchors unstick to a single absorbing point with vanishing continuous region.
• Continuous diffusion by setting all unsticking rates to zero—no discrete component, pure score-based SDE.
• Hybrid diffusion when the unsticking kernel is per-position isotropic with matching VP schedule and the blending matrix is identity.

The SJD time-reversal elucidates the features of each limit: for instance, why masked-diffusion commit events carry no information about the originating token (all mass unsticks to the same point), or why continuous models require a terminal projection.

Denoising Hazard Matching (DHM): Learning SJD Samplers

A single denoising classifier $\mathsf{X} = \mathbb{R}^d$0, trained with cross-entropy on SJD-corrupted pairs, suffices for both reverse-time decisions: the allocation (“where”) and the total commit rate (“when”). The reverse hazard is written as a conditional expectation (DHM identity), mirrored on denoising score matching:

Figure 2: Effect of $\mathsf{X} = \mathbb{R}^d$1 on the state-dependent hazard target $\mathsf{X} = \mathbb{R}^d$2. Smaller $\mathsf{X} = \mathbb{R}^d$3 shrinks commitment regions toward the anchor, delaying commitments for mass farther from anchors.

$\mathsf{X} = \mathbb{R}^d$4

with the DHM analytic target involving only forward-time known quantities. The classifier posterior is used for Rao-Blackwellization, maximizing estimation efficiency.

All SJD-specific computations (including integral mixtures over unstick times) are deferred to post-hoc quadratures at inference, not to the classifier network.

The SJD Unsticking Kernel as Design Axis

Unlike previous hybrid models, SJD makes the unsticking kernel (i.e., cross-position blending) an explicit inductive bias, allowing spatial or constraint-graph correlations to be injected into the corruption dynamics. For image, text, and constrained-logic data, empirical results show this axis is powerful:

Empirical Results

Image Generation (CIFAR-10)

Blurring pixel values before embedding (a “value-space blur”) as the SJD corruption kernel improves FID, outperforming both identity-kernel hybrids and matched-ID continuous/masked baselines—demonstrating the kernel’s role as a sample-quality controller.

Text Generation (Text8)

Using a local 1D Gaussian blend for each character position, SJD consistently achieves higher valid-word ratios at length thresholds over CANDI and Masked Diffusion LLMs (MDLM), with Pareto improvements along the quality-diversity frontier.

Figure 3: Text8 valid-word counts at length thresholds $\mathsf{X} = \mathbb{R}^d$5 and $\mathsf{X} = \mathbb{R}^d$6 as a function of the blending kernel $\mathsf{X} = \mathbb{R}^d$7 (NFE budgets). SJD with $\mathsf{X} = \mathbb{R}^d$8 dominates hybrid and masked baselines.

Figure 4: Quality--diversity frontiers on Text8, sweeping temperature and kernel parameters. SJD family dominates high-diversity regime.

Constraint Satisfaction (Sudoku)

On Sudoku, SJD’s cross-position constraint-graph kernel (encoding row/column/box structure) enables faster and more reliable model convergence, higher solution accuracy, and avoids the unstable learning behavior of identity-kernel hybrids.

Theoretical Implications

SJDs provide a principled, process-level framework for designing and interpreting discrete/continuous generative models. The flux-balance-determined jump intensities obviate heuristic schedule choices, ensuring the reverse-time sampler aligns strictly with the forward dynamics; model decisions at each step (commitment time and destination) are specified by the process, not architecture.

The ability to specify arbitrary cross-position blends creates a rich design space for encoding inductive biases (e.g., spatial, syntactic, or logical) directly at the process level rather than in neural architecture.

Further, the simulation-free, cross-entropy-only training with analytic Rao-Blackwellization ensures robustness, variance reduction, and the avoidance of auxiliary regression heads common in hybrid models.

Future Directions

Potential continuations include:

• End-to-end training of state-dependent, adaptive hazard schedules for scheduling commit events, perhaps leveraging downstream task feedback.
• Expansion to combinatorially structured or graphical data with learnable constraint-graph kernels.
• Theoretical investigation of kernel design trade-offs on sample efficiency, expressivity, and reasoning capacity.
• Extending SJD-style process invariances to more general state spaces beyond Euclidean token embeddings.

Conclusion

SJDs rigorously subsume and explain both discrete and hybrid diffusion methods as limits, while providing a flexible design axis via the corruption kernel. The framework's reverse-sampler is analytically determined by flux conservation, not by hand-tuned scheduling, and harnesses the denoising classifier for both allocation and hazard rate estimation via DHM. Practically, SJD shows measurable empirical gains by leveraging cross-position blending in the forward process. The methodology encourages further architectural and process-level innovation for finite-support generative models.