ELF: Embedded Language Flows

Abstract: Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos. Their success has attracted growing interest in applying them to language modeling. Unlike their image-domain counterparts, today's leading diffusion LLMs (DLMs) primarily operate over discrete tokens. In this paper, we show that continuous DLMs can be made effective with minimal adaptation to the discrete domain. We propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space based on continuous-time Flow Matching. Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network. This formulation makes it straightforward to adapt established techniques from image-domain diffusion models, e.g., classifier-free guidance (CFG). Experiments show that ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results suggest that ELF offers a promising path toward effective continuous DLMs.

• The paper introduces a continuous diffusion model that operates in the embedding space, eliminating intermediate token-level supervision.
• It employs flow matching with x-prediction and uses SDE-based samplers to reduce perplexity and enhance generation quality.
• Empirical results demonstrate that ELF outperforms discrete and other continuous models on tasks such as translation and summarization.

Embedded Language Flows (ELF): A Continuous Diffusion Model for Language Generation

Motivation and Context

Recent advances in diffusion and flow-based models have established continuous-time generative paradigms as state-of-the-art for images and other continuous modalities. However, their adaptation to language modeling has bifurcated into discrete DLMs—operating directly on token spaces—and continuous DLMs—denoising in embedding or latent spaces. Empirically, discrete DLMs have been dominant, but the underlying question is whether the gap reflects intrinsic modality differences or simply suboptimal algorithmic choices in the continuous regime.

ELF (Embedded Language Flows) introduces a minimalist yet rigorously continuous framework for diffusion language modeling. By operating almost entirely within a continuous embedding space and postponing discretization to the final generation step, ELF breaks from the prevalent paradigm of intermediate per-step token supervision and decoder reliance found in previous continuous DLMs.

Figure 1: Conceptual illustration of ELF as a diffusion trajectory in embedding space, with discretization only at $t=1$.

Methodological Formulation

Embedding Space Construction

ELF maps discrete token sequences to contextual continuous embeddings via a frozen pretrained T5 encoder. The encoder is solely active during training; inference employs only the denoising network and unembedding layer, eliminating decoder overhead.

Continuous-Time Flow Matching

ELF's generation path is formulated via Flow Matching—leveraging linear (rectified flow) interpolation from Gaussian noise to data embeddings, parameterized as $z_t = t x + (1-t)\,\epsilon$ for $t \in [0,1]$. The network predicts clean embeddings ($x$-prediction) rather than velocity ($v$-prediction), aligning the denoising and discretization objectives and empirically outperforming velocity-based approaches, especially at higher dimensionalities.

Figure 2: ELF training pipeline: tokens encode to clean embeddings, corrupted to $z_t$, and ELF predicts $\hat{x}$ (clean embeddings); decoding occurs only at $t=1$.

Discretization and Decoding

At $t=1$, ELF projects clean embeddings back to discrete tokens via a learnable unembedding matrix, optimizing a cross-entropy objective. Crucially, the denoising and decoding modes share network weights, reinforced by the binary "mode" token and control tokens for time and CFG scale.

Sampling and Guidance

Inference proceeds via ODE or SDE samplers; stochasticity via SDE improves low-perplexity regimes. ELF natively incorporates classifier-free guidance (CFG) by extrapolating between unconditional and self-conditioned predictions, implemented efficiently via training-time CFG techniques. Conditional generation is realized by prepending clean embeddings from input prompts, leveraging self-attention and CFG for robust context adherence.

Figure 3: Key design ablations (embedding choice, decoding strategy, sampler): pretrained contextual embeddings and SDE-inspired samplers yield improved efficiency and quality.

Empirical Analysis

Unconditional Generation

On OpenWebText, ELF-B (105M) achieves generative perplexity 24 in only 32 sampling steps, surpassing discrete DLMs (MDLM, Duo) and contemporary continuous DLMs (FLM, LangFlow) trained on much larger token budgets. ELF rivals distilled baselines requiring additional training, while maintaining a minimalist architecture.

Figure 4: ELF-B outperforms discrete and continuous DLMs, rivals distilled variants, and uses substantially fewer training tokens.

Conditional Tasks

On WMT14 German-to-English translation and XSum summarization, ELF exhibits superior BLEU and ROUGE scores compared to autoregressive and diffusion baselines of similar scale. Qualitative outputs demonstrate contextually fluent and semantically aligned generations.

Figure 5: ELF-B qualitative samples for unconditional, translation, and summarization tasks with strong automatic metrics.

Denoising Trajectory

ELF's continuous sampling transforms initial noise into fluent and grammatical sentences as $t$ progresses, visualizing the emergence of meaningful language from embedding space.

Figure 6: ELF-B denoising trajectory: progressive refinement from ungrammatical to grammatical text as $z_t = t x + (1-t)\,\epsilon$0 increases.

Ablation and Design Analysis

• Prediction Target: $z_t = t x + (1-t)\,\epsilon$1-prediction is robust across embedding dimensionalities and exhibits stable performance, supporting the hypothesis of data lying on low-dimensional manifolds.
• Bottleneck Dimension: Moderate bottlenecks (e.g., 128d) optimize the perplexity-diversity trade-off; extremes induce degeneracy or loss of diversity.
• Denoising Mode Probability: Allocation of 0.8 (denoising) stabilizes training, yielding the best trade-off.
• Conditioning Strategy: In-context, token-based conditioning reduces parameter count and outperforms adaLN-Zero.
• Optimizer: Muon outperforms AdamW for training efficiency and generative perplexity.
• Sampling Schemes: Logit-normal schedules and SDE stochasticity significantly improve inference efficiency and generation quality; optimal SDE noise scale $z_t = t x + (1-t)\,\epsilon$2 finely tunes the perplexity-diversity frontier.

  Figure 7: $z_t = t x + (1-t)\,\epsilon$3-prediction remains effective as embedding dimension increases, while $z_t = t x + (1-t)\,\epsilon$4- and $z_t = t x + (1-t)\,\epsilon$5-prediction degrade.

  

  Figure 8: Bottleneck dimension analysis: 128d yields best balance, extremes reduce generation quality or diversity.

  

  Figure 9: Denoising mode probability: 0.8/0.2 (denoise/decode) optimizes trade-off.

  

Figure 10: In-context conditioning improves performance and reduces parameter overhead.

Figure 11: Logit-normal time schedule and SDE noise scale $z_t = t x + (1-t)\,\epsilon$6 ablation.

Figure 12: CFG scale sweep: moderate guidance ($z_t = t x + (1-t)\,\epsilon$7) optimizes conditional task performance.

Practical, Theoretical Implications, and Future Directions

ELF demonstrates that a rigorously continuous approach—eschewing intermediate token-level supervision and extra decoding architectures—renders continuous DLMs competitive with discrete counterparts. Its compatibility with sophisticated guidance (CFG), efficient sampling schemes (SDE), and reduced dependence on training budget signal practical advantages for scalable and controllable language modeling.

Theoretically, ELF substantiates the view that language modeling can profitably adopt continuous-time, embedding-centric generative paradigms characteristic of vision diffusion models. Its empirical robustness to embedding dimension and architectural minimalism suggest that future work in diffusion-based language modeling should further explore continuous-time flow models, alternative embedding spaces, and advanced conditioning and guidance strategies.

Given its design, ELF offers promising avenues for low-resource generative models, efficient conditional language generation, and cross-modal extensions leveraging shared continuous representations. Ongoing developments may focus on scaling ELF architectures, expanding their conditional repertoire, and integrating multimodal prompts and outputs.

Conclusion

ELF (Embedded Language Flows) establishes a continuous diffusion LLM paradigm, leveraging Flow Matching in embedding space with minimal adaptation for discrete data. By denoising entirely in continuous space and discretizing only at the final step, ELF achieves strong generation quality and data efficiency, surpassing discrete and continuous DLMs across tasks and ablation benchmarks. These results reinforce the viability of continuous DLMs and motivate further exploration of flow-based generative modeling for language and beyond.