Diffinity: Continuous Diffusion Regex Guidance

• Diffinity is a method that enforces formal syntactic constraints on continuous diffusion language models by leveraging regex-derived finite automata.
• It transforms regex patterns into tokenizer-aligned automata and computes differentiable acceptance probabilities to steer latent denoising.
• Empirical results show that Diffinity greatly improves constraint satisfaction in natural language tasks compared to autoregressive constrained decoding.

Diffinity is a training-free guidance method for enforcing formal syntactic constraints in continuous diffusion LLMs, specifically constraints expressible as regular expressions / finite automata. It targets the structural mismatch between continuous diffusion generation and standard constrained decoding: a continuous diffusion LM denoises a continuous latent representation of the entire sequence, so there is no explicit discrete prefix during generation, unlike autoregressive LMs. Diffinity addresses this by analytically computing how likely a current latent state is to decode to a string accepted by a target regex, differentiating that probability with respect to the latent, and injecting the resulting gradient into the reverse diffusion update. Implemented on top of PLAID, it is presented as a classifier-guidance-style method in which the “classifier” is replaced by an analytic acceptance probability computation derived from the regex itself (Kim et al., 12 Feb 2026).

1. Definition and target setting

Diffinity is designed for continuous text diffusion models such as PLAID. In this setting, generation proceeds by denoising latent embeddings rather than discrete tokens, and the decoded text only appears at the end. The decoder maps a latent to positionwise token distributions, not to one hard string. Formal syntax constraints such as regexes are therefore difficult to impose, because validity is discrete, combinatorial, and nonlocal, while the model state is continuous and global.

The contrast with autoregressive constrained decoding is central. In an autoregressive LM, one maintains an automaton state over the current prefix and masks next-token choices that cannot lead to a valid completion. Continuous diffusion lacks the corresponding primitive: at intermediate timesteps there is no canonical prefix string, and each timestep manipulates a noisy continuous latent for all positions jointly. Prior diffusion constrained-decoding methods worked mainly for discrete masking diffusion models, where there is still an interpretable partially discrete structure. Diffinity instead targets continuous diffusion directly (Kim et al., 12 Feb 2026).

The method is framed as a training-free mechanism with five steps: it takes a regex constraint, turns it into a finite automaton aligned with the model tokenizer, analytically computes the probability that the current latent state will decode to some valid string in the regex language, differentiates that probability with respect to the latent, and injects the gradient into the reverse diffusion update. The method is training-free with respect to the constraint: no auxiliary classifier is trained and no finetuning is performed.

2. Guidance objective and probabilistic formulation

The underlying reverse denoising step is the continuous DDPM-style update

$$x_{t-1} = \mu_\theta(x_t, t) + \sigma_t \epsilon \tag{1}$$

where $x_t$ is the current noisy latent, $x_{t-1}$ is the next less-noisy latent, $\epsilon \sim \mathcal{N}(0,I)$, $\sigma_t$ is the timestep variance, and $\mu_\theta(x_t,t)$ is the model’s predicted reverse-process mean.

Diffinity modifies this update to steer sampling toward latent regions that are likely to decode to a string in the regular language $\mathcal{L}$: $$x_{t-1} = \mu_\theta(x_t, t) + \gamma \sigma_t^2 \nabla_{x_t} \log \mathbb{E}_{s \sim \operatorname{Dec}(x_t)} [s \in \mathcal{L}] + \sigma_t \epsilon \tag{2}$$ Here $\mu_\theta(x_t,t)$ is the unconditional reverse update from the base diffusion model, $$\mathbb{E}_{s \sim \operatorname{Dec}(x_t)} [s \in \mathcal{L}]$$ is the probability that a string sampled from the decoder-induced unigram distribution satisfies the regex, $x_t$0 pushes the latent toward higher regex-validity probability, $x_t$1 is the guidance scale, and $x_t$2 is the standard classifier-guidance scaling by noise level (Kim et al., 12 Feb 2026).

The method is motivated by the standard score decomposition

$x_t$3

and replaces the learned classifier term with the analytic proxy

$x_t$4

This proxy depends on two explicit assumptions. First, the decoder-induced unigram assumption: $x_t$5 is treated as a matrix of independent per-position token distributions. Second, the approximation above is exact near $x_t$6, where $x_t$7, and only approximate at noisy timesteps, especially large $x_t$8.

The acceptance probability is differentiable as a function of the decoder probabilities $x_t$9. Since the decoder is differentiable with respect to $x_{t-1}$0, Diffinity computes

$x_{t-1}$1

by backpropagating through the decoder and through the automaton dynamic program. In implementation, Diffinity “takes the gradient of this transformer composed with our implementation of the expected probability to obtain gradients on the word2vec latent space.”

3. Regular languages, tokenizer alignment, and analytic acceptance probability

Diffinity uses regular expressions as the user-facing constraint language and compiles them to a deterministic finite automaton. The appendix defines a DFA as

$x_{t-1}$2

where $x_{t-1}$3 is the alphabet, $x_{t-1}$4 the states, $x_{t-1}$5 the initial state, $x_{t-1}$6 the transition function, and $x_{t-1}$7 the accepting states.

A technical complication arises because regexes are naturally defined over characters, whereas the diffusion LM emits tokens from a vocabulary $x_{t-1}$8, and many strings admit multiple tokenizations. Diffinity therefore introduces a function $x_{t-1}$9 that converts a character-level DFA into a tokenizer-aligned automaton $\epsilon \sim \mathcal{N}(0,I)$0. The algorithm splits the DFA into a character-level automaton $\epsilon \sim \mathcal{N}(0,I)$1, splits each vocabulary token $\epsilon \sim \mathcal{N}(0,I)$2 into characters, checks from each automaton state whether consuming those characters follows a valid path, adds a token-labeled transition $\epsilon \sim \mathcal{N}(0,I)$3 when it does, and removes character transitions not in the vocabulary. The appendix theorem states that $\epsilon \sim \mathcal{N}(0,I)$4 accepts a sequence of tokens $\epsilon \sim \mathcal{N}(0,I)$5 iff $\epsilon \sim \mathcal{N}(0,I)$6 for all $\epsilon \sim \mathcal{N}(0,I)$7 and $\epsilon \sim \mathcal{N}(0,I)$8 (Kim et al., 12 Feb 2026).

Once the aligned automaton is built, Diffinity computes the regex acceptance probability under the decoder-induced unigram distribution using dynamic programming over automaton states. For sequence length $\epsilon \sim \mathcal{N}(0,I)$9, it constructs a transition matrix $\sigma_t$0 for each position $\sigma_t$1, with

$\sigma_t$2

The dynamic program is initialized by

$\sigma_t$3

propagates according to

$\sigma_t$4

and returns the final acceptance probability

$\sigma_t$5

Theorem 1 states that Algorithm 1 returns

$\sigma_t$6

equivalently

$\sigma_t$7

The paper’s illustrative example uses the regex c(a|u)t, whose automaton has paths for cat and cut. Given per-position token probabilities from $\sigma_t$8, the acceptance probability becomes the total probability mass of all accepted token paths, for example

$\sigma_t$9

The finite-state component is therefore not a decoding heuristic; it is the differentiable conditioning signal used by the sampler.

4. Implementation on PLAID and empirical results

Diffinity is implemented on top of PLAID. The relevant assumptions are explicit: PLAID is a continuous diffusion model; it uses a 32-dimensional word2vec latent space; it has a 1.3B transformer that interprets latent points as unigram probability distributions over tokens; decoding to text happens once at the end, not at every denoising step; and final output uses argmax from the final unigram distribution. Diffinity inserts guidance directly into PLAID’s reverse DDPM sampling equation at every denoising step, without altering training (Kim et al., 12 Feb 2026).

The experimental setup uses 180 regex constraints in two groups. The JSON benchmark contains 70 regexes derived from JSONSchemaBench: the construction started with 100 schemas, converted schemas to regex when possible, excluded 10 non-regular schemas, and excluded 20 more whose tokenizer-aligned automata were too large, namely more than 230 states or more than 7.5M transitions. The natural-language benchmark contains 110 synthetic regex templates in six categories: Prefix, Suffix, Appearance, Between-$\mu_\theta(x_t,t)$0, Between (unbounded), and Word Length. Sequence length is fixed at 64 for all benchmarks. JSON generation uses 256 timesteps; natural language uses 1024 timesteps. Natural language uses 20 samples per regex; JSON uses 10 samples per regex. The baselines are autoregressive grammar-constrained decoding using the Guidance library on GPT2-Small, GPT2-Medium, and GPT2-Large. The metrics are constraint satisfaction rate, Pass@10, perplexity using Llama-3.1-8B as reference LM, and fluency using Claude Sonnet 4.5, scored 0–100 over matching samples only.

For JSON, Diffinity uses padded schema regexes of the form

$\mu_\theta(x_t,t)$1

because diffusion outputs fixed-length token sequences; the GPT baselines use the schema regex directly without padding. For natural language, the headline result in the text is that Diffinity reaches 92.9% average satisfaction, versus 20.5% for GPT2-Large-GCD (Kim et al., 12 Feb 2026).

Setting	Diffinity	Comparator highlights
JSON satisfaction / pass@10	68.4% / 91.4%	GPT2-Small-GCD: 79.3% / 98.0%
Natural-language average satisfaction	92.9%	GPT2-Large-GCD: 20.5%
Prefix satisfaction	95.7%	GPT2-Large-GCD: 43.5%
Suffix satisfaction	96.8%	GPT2-Large-GCD: 17.5%
Appearance satisfaction	92.5%	GPT2-Large-GCD: 3.2%
Between-$\mu_\theta(x_t,t)$2 satisfaction	85.5%	GPT2-Large-GCD: 0.5%
Between (unbounded) satisfaction	93.8%	GPT2-Large-GCD: 3.5%
Word Length satisfaction	95.0%	GPT2-Large-GCD: 89.5%

The JSON result is competitive but somewhat worse than constrained GPT-2: Diffinity achieves 68.4% satisfaction and 91.4% pass@10, compared with 79.3% and 98.0% for GPT2-Small-GCD, 75.3% and 97.1% for GPT2-Medium-GCD, and 77.0% and 98.6% for GPT2-Large-GCD. The paper attributes this partly to the difficulty of highly structured formats for a relatively small diffusion LM.

The natural-language results are substantially stronger. Per category, Diffinity reports Prefix 95.7 / 100.0 / 66.5 / 32.2, Suffix 96.8 / 100.0 / 59.3 / 36.8, Appearance 92.5 / 100.0 / 58.5 / 34.7, Between-$\mu_\theta(x_t,t)$3 85.5 / 95.0 / 58.7 / 33.3, Between (unbounded) 93.8 / 100.0 / 57.4 / 33.3, and Word Length 95.0 / 100.0 / 60.7 / 43.3, where the four numbers are satisfaction, pass@10, perplexity, and fluency. GPT2-Medium-GCD and GPT2-Large-GCD are dramatically worse on Prefix, Suffix, Appearance, Between-$\mu_\theta(x_t,t)$4, and Between (unbounded), while Word Length remains relatively easy for autoregressive constrained decoding.

The paper also compares Diffinity with unconditioned PLAID at guidance scale 2.5 on natural-language benchmarks. Diffinity has average perplexity 60.0 and average fluency 35.8; unconditioned PLAID has perplexity 61.6 and fluency 46.0. The perplexity gap is described as “virtually nonexistent,” while the fluency drop is moderate. On the subset of tasks representable both by Diffinity and PLAID native guidance—Prefix, Suffix, and Appearance—PLAID native guidance has 59.4% average satisfaction, whereas Diffinity has 92.9%. For minimum-perplexity samples on these tasks, Diffinity reports average minimum perplexity 10.4, versus 25.7 for PLAID native guidance.

5. Guidance-scale behavior, global-sequence advantages, and comparison with autoregressive decoding

A guidance-scale ablation is reported at scales 1.0, 1.5, 2.0, and 2.5. Natural-language satisfaction rises from 77.3 to 89.1 to 90.8 to 93.0, while JSON satisfaction rises from 30.4 to 50.6 to 62.7 to 68.4. Natural-language perplexity decreases from 73.56 to 69.2 to 65.47 to 59.95, while natural-language fluency changes from 39.4 to 39.3 to 37.3 to 35.8. Stronger guidance therefore improves satisfaction and also improves perplexity, while only mildly reducing fluency (Kim et al., 12 Feb 2026).

The paper’s explanation for the strong natural-language performance centers on the infinite repetition trap in autoregressive constrained decoding. If a regex requires a word such as cat in a certain future position but the model has already entered a low-probability prefix, the autoregressive system may keep extending earlier material rather than committing to the required token, eventually exhausting the max token budget. Because Diffinity denoises the entire sequence jointly, it can move globally toward a full satisfying sentence rather than greedily extending a problematic prefix. This is presented as especially important for Appearance, Between-$\mu_\theta(x_t,t)$5, and Between (unbounded) constraints.

The appendix includes a distribution-preservation experiment on regexes forcing a choice between sentence prefixes like “The …” and “It …”. Diffinity’s relative frequencies track external LM likelihood much better, while GPT2-Small-GCD remains dominated by first-token biases. This suggests that the global, non-causal denoising process can preserve the base distribution more faithfully under certain formal constraints, although the paper presents this as a targeted empirical observation rather than a general theorem.

6. Computational costs, assumptions, and scope

The principal limitation is computational cost. The automaton score computation scales approximately linearly with the number of transitions because Algorithm 1 line 6 computes a dense sum over automaton transitions, and gradient computation uses torch.autograd. The reported latency examples are 33.3 s/sample for unconstrained PLAID at batch size 1, 55.9 s/sample for PLAID conditional generation, and 145.6 s/sample for the smallest Diffinity benchmark shown, which has 71,735 transitions, also at batch size 1. For 14 representative benchmarks, time increases roughly linearly with transition count. Batch sizes 4 and 16 help substantially, indicating good GPU parallelism, but the overhead remains significant (Kim et al., 12 Feb 2026).

The method’s scope is also sharply delimited by its assumptions. It requires a differentiable mapping from latents to token probabilities, which is natural in PLAID but may not hold for arbitrary diffusion LMs. It supports regular constraints only: non-regular JSON schema constructs, nested objects, and unbounded list structures were excluded. Automaton blowup after tokenizer alignment is a practical bottleneck, as reflected in the filtering of JSON regexes larger than 230 states or 7.5M transitions. The term “training-free” is correspondingly narrow: Diffinity is training-free with respect to new constraints, but it still depends on a pretrained model architecture with a suitable latent-decoder structure.

The paper proposes several directions for reducing the runtime burden: more memory-efficient automaton data structures, custom gradient kernels, and compact automata representations such as symbolic automata. A plausible implication is that the finite-state guidance idea is less limited by principle than by current automaton and autograd efficiency.

Diffinity’s conceptual contribution is to show that continuous diffusion LLMs can obey formal syntax without exposing discrete prefixes and without training auxiliary classifiers. Its core construction is finite-state and analytic: compile a regex to a DFA, align the DFA with the tokenizer, compute acceptance probability under decoder marginals by dynamic programming, backpropagate that probability through the decoder into latent space, and inject the gradient into reverse diffusion. In this sense, Diffinity recasts syntax control in continuous diffusion as differentiable probabilistic conditioning over regular languages rather than token masking or left-to-right constrained decoding (Kim et al., 12 Feb 2026).