Masked Language Flow Models

• MLFMs are a generative framework that merges masked diffusion with continuous flow-based embedding refinement to enable efficient, high-quality language generation.
• They bridge partially masked and clean text sequences via stochastic interpolation in embedding space, facilitating adaptive token promotion and context-guided sampling.
• Lightweight adaptation techniques, including LoRA integration and AdaLN conditioning, fine-tune large pretrained models to achieve state-of-the-art results in reasoning and instruction following.

Masked Language Flow Models (MLFMs) are a generative modeling framework for language that unifies the discrete, position-wise flexibility of masked diffusion models with the continuous, joint flow-based refinement of embedding trajectories. MLFMs provide a mechanism to bridge partially masked and clean text sequences via stochastic interpolation in embedding space, enabling improved efficiency and quality in both parallel and multi-step language generation. This framework supports the adaptation of pretrained masked diffusion models through lightweight architectural modifications and enables novel sampling strategies that synergistically combine continuous denoising and discrete token commitments. MLFMs achieve state-of-the-art results in instruction following and reasoning tasks under constrained decoding budgets (Azangulov et al., 26 Jun 2026, Wang et al., 27 Jun 2026).

1. Theoretical Foundations

MLFMs originate from critical advances in two preceding generative paradigms: Masked Diffusion Models (MDMs) and Flow LLMs (FLMs).

Masked Diffusion Models (MDMs) define a forward process where clean token sequences $X \in \mathcal{V}^L$ are gradually masked per token at rate $s \in [0,1]$. The reverse process unmasking is parallelized but factorized across positions, modeled by $p_\theta^\ell(X^\ell \mid X_s^{\mathcal U_s})$. This factorization breaks contextual dependencies in few-step generations, degrading quality.

Flow LLMs (FLMs) embed tokens into Euclidean space and learn a continuous flow transporting noise toward clean sequences, parameterized by an interpolation parameter $t \in [0,1]$. While FLMs maintain all tokens in a latent state until termination, this approach lacks the ability to incrementally commit tokens for multi-step, chain-of-thought reasoning.

MLFMs integrate these perspectives by introducing a two-parameter forward process:

• Masking index $s$: determines masked vs. observed positions.
• Noising index $t$: governs the continuous stochastic interpolation between mask and clean embedding for masked positions.

The stochastic interpolant is defined for each token position $\ell$ as:

$$z_{s,t}^\ell = \begin{cases} E_{X^\ell}, & \ell \in \mathcal U_s \ (1-t) m + t E_{X^\ell} + \mathcal{N}(0, \sigma^2 t(1-t) I_H), & \ell \in \mathcal{M}_s \end{cases} \quad t \in [0,1]$$

where $m$ is the mask embedding. The model is trained by cross-entropy loss only over masked positions:

$$\mathcal{L}_{\mathrm{MLFM}}(\theta) = \mathbb{E}_{X \sim p,\,s \sim \pi_s,\,t \sim \pi_t} \left[ \frac{1}{|\mathcal{M}_s|} \sum_{\ell \in \mathcal{M}_s} -\log p_\theta^\ell(X^\ell \mid z_{s,t}, t) \right]$$

Posterior-mean embeddings and velocity fields on masked positions are used to define an ODE in embedding space, supporting joint, continuous updates:

$s \in [0,1]$0

Integration of this velocity field produces a flow from masked to clean states (Azangulov et al., 26 Jun 2026, Wang et al., 27 Jun 2026).

2. Model Architecture and Adaptation

A key strength of MLFMs is the capacity for lightweight adaptation of large pretrained masked diffusion models (e.g., SMDM 1.028B parameters) into flow-based generators:

• Embedding and Backbone Freezing: The token-embedding matrix and Transformer blocks are frozen.
• LayerNorm Time Conditioning: Each LayerNorm is augmented with AdaLN time-conditioning on the flow parameter $s \in [0,1]$1.
• Low-Rank Adaptation: LoRA adapters (rank 256) are inserted into all attention, MLP, and output head layers, with only these and AdaLN parameters (≈319M) trained under the MLFM objective.

Typical hyperparameters include LoRA rank of 256, bridge noise scale $s \in [0,1]$2, mask scheduling via Clipped MaskGIT cosine policies, and time $s \in [0,1]$3 sampled from mixtures over intervals and fitted distributions.

This modular adaptation strategy allows rapid fine-tuning without extensive re-training and preserves the majority of pretrained capabilities for contextual modeling (Azangulov et al., 26 Jun 2026).

3. Decoding, Sampling, and Policy Mechanisms

MLFM sampling interleaves continuous flow steps akin to DDPM samplers with discrete “promotion” (unmasking) of tokens whose predictive probability exceeds a confidence threshold:

• DDPM-Like Flow Steps: For masked positions, the embeddings are updated via a Gaussian bridge informed by the model’s posterior mean.
• Context-Corrupted Classifier-Free Guidance (CCFG): Guidance is achieved by constructing a reference flow where only the already-determined context (“clean” positions) is re-noised, producing a guided velocity

$s \in [0,1]$4

with guidance scale $s \in [0,1]$5. This leverages classifier-free denoising to sharpen sample quality.

• Online Token Promotion (OTP): At every time step, masked tokens with maximum posterior probability above $s \in [0,1]$6 are deterministically unmasked and fixed, using their clean embedding as context in the subsequent flow.
• Early Promotion Bound: Premature promotion with tolerance $s \in [0,1]$7 introduces at most a total variation error of $s \in [0,1]$8.

This hybrid of continuous denoising and discrete committal aligns with, and generalizes, the masked x-prediction flow decoders developed for masked diffusion LLMs (Nagatomi et al., 26 Jun 2026, Azangulov et al., 26 Jun 2026).

Continuous Decoding and Partial-Belief Accumulation (as in (Wang et al., 27 Jun 2026)) further refines this approach: each token position is updated asynchronously according to a learned per-token progress scalar $s \in [0,1]$9. Tokens maintain soft, revisable embeddings and advance via velocities proportional to per-token confidence. Decoding is asynchronous and driven by a policy network which selects step fractions dynamically based on contextual signals, confidence, and entropy.

Policy Network and RL Training: A lightweight MLP is trained by Grouped Rollout Policy Optimization (GRPO), optimizing a compound reward based on task completion (e.g., HumanEval pass@1) and progress completion per token.

4. Algorithmic Implementation

A typical MLFM sampler executes the following procedure:

1. Initialization: Masked tokens are set to the mask embedding $p_\theta^\ell(X^\ell \mid X_s^{\mathcal U_s})$0; observed tokens use their clean embedding.
2. Iterative Flow Updates: For a discretized schedule $p_\theta^\ell(X^\ell \mid X_s^{\mathcal U_s})$1, repeat:
   • Evaluate posterior-mode token predictions and per-token confidence.
   • Promote confident tokens to the clean set and update embeddings.
   • Apply context-corrupted guidance if enabled.
   • Advance masked positions using predicted velocities in embedding space through stochastic updates.
3. Termination: Proceed until all tokens are promoted or $p_\theta^\ell(X^\ell \mid X_s^{\mathcal U_s})$2. Final decoding samples remaining ambiguous tokens.

Explicit pseudocode is provided in (Azangulov et al., 26 Jun 2026, Wang et al., 27 Jun 2026), allowing unambiguous implementation.

The x-prediction flow decoder (as formalized in (Wang et al., 27 Jun 2026)) similarly operates by updating continuous progress scalars $p_\theta^\ell(X^\ell \mid X_s^{\mathcal U_s})$3 for each token, accumulating partial beliefs, and applying synchronous or asynchronous corrections (including hard commitment and re-editing). Policy-directed update magnitudes provide adaptive, per-token advancement based on model confidence and context.

5. Empirical Results and Component Analysis

MLFMs offer substantial improvements in efficiency and sample quality under limited decoding budgets across language modeling, code generation, and reasoning benchmarks.

Code Generation (LLaDA-8B-Instruct and LLaDA2.0-mini):

With only 25% of the diffusion steps, x-prediction flow decoders recover 97% of full-budget performance. On HumanEval (8B), accuracy improves from 33.54% (standard mask-prediction at 1/4 budget) to 45.12% (+11.6%) with x-prediction flow; similar gains appear on MBPP (Wang et al., 27 Jun 2026).

Reasoning and Instruction Tasks (GSM8K, MT-Bench):

MLFM samplers match or outperform SMDM and AR baselines for MT-Bench at 128 vs 256 steps (2.27 vs 1.60 and 1.57 GPT-4o scores). On GSM8K, MLFM achieves 31.24% (vs 58.5% SMDM); further progress is expected with task-specific fine-tuning (Azangulov et al., 26 Jun 2026).

Ablation Studies:

Component	HumanEval (8B, 1/4 budget)	Δ
Full system	45.12	—
– Hard commitment	41.46	–3.66
– Re-editing	26.83	–18.29
– x-pred alignment	17.07	–28.05

These ablations demonstrate the criticality of each algorithmic component. Policy-driven adaptive updates, token-wise progress accumulation, and the combined use of discrete and continuous reasoning all contribute materially to final performance.

6. Significance and Future Directions

MLFMs expand the expressive and practical reach of flow-based generation by combining any-position conditioning, partial observability, and efficient parallel/compositionality with embedding-space trajectories that enable continuous revision and re-evaluation. This synthesis makes MLFMs particularly well-suited for tasks requiring both flexible parallel generation and multi-step chain-of-thought reasoning, including code synthesis, mathematical deduction, and multi-turn instruction following.

While existing results confirm scalability to billion-parameter models and strong instruction-following, further work is required to close performance gaps on tasks with high reasoning complexity without extensive fine-tuning. A plausible implication is that advances in adaptive time and masking schedules, as well as richer policy learning signals, could further accelerate convergence and sample quality.

MLFMs represent a convergence of denoising diffusion, flow-based modeling, and policy-driven asynchronous refinement, offering a unified framework potentially extensible to non-autoregressive, multimodal, and non-textual generative tasks (Azangulov et al., 26 Jun 2026, Wang et al., 27 Jun 2026).