---
title: 'Dream-7B: Diffusion LLM'
url: https://www.emergentmind.com/topics/dream-7b-c6df8262-070c-4319-a9c2-0fd3e6eb7050
type: topic
---

# Dream-7B: Diffusion LLM

Searching arXiv for the relevant Dream-7B papers and closely related work.
Dream-7B is a 7-billion-parameter diffusion large language model (DLLM) introduced as an open alternative to standard autoregressive (AR) large language models. Its defining departure from AR modeling is that it generates by iteratively denoising a masked sequence in parallel, rather than emitting tokens strictly left-to-right by next-token prediction. The model is presented as a general-purpose base language model, with public variants **Dream-Base** and **Dream-Instruct**, and is subsequently used as the backbone family for code and multimodal descendants such as Dream-Coder 7B, Dream-VL, and Dream-VLA [2508.15487]. The name should not be conflated with two unrelated systems that also use the acronym DREAM: the biomedical autonomous research framework in "Autonomous self-evolving research on biomedical data: the DREAM paradigm" and the dense retrieval method in "DREAM: Dense Retrieval Embeddings via Autoregressive Modeling" [2407.13637] [2606.24667].

## 1. Identity, scope, and disambiguation

Dream-7B denotes the diffusion LLM family introduced in "Dream 7B: Diffusion Large Language Models" [2508.15487]. In that work, the model is described as a strong open diffusion language model designed to narrow the performance gap between discrete diffusion language models and modern 7B-class AR baselines such as Qwen2.5-7B, while preserving diffusion-specific capabilities including arbitrary-order generation, infilling, and a tunable quality-speed trade-off.

A recurring source of ambiguity is that several unrelated papers use the string **DREAM**. The biomedical paper from 2024 uses DREAM as shorthand for **Data-driven self-Evolving Autonomous systeM**, a modular autonomous biomedical research system rather than a 7B checkpoint [2407.13637]. The 2026 retrieval paper uses DREAM as **Dense Retrieval Embeddings via Autoregressive Modeling**, a method for training dense retrievers via frozen-LLM supervision rather than a generative language model [2606.24667]. In the context of generative modeling, Dream-7B refers specifically to the diffusion LLM line.

| Name | arXiv id | Scope |
|---|---:|---|
| Dream 7B | [2508.15487] | Diffusion large language model |
| DREAM | [2407.13637] | Autonomous biomedical research system |
| DREAM | [2606.24667] | Dense retrieval embedding method |

This naming distinction matters because later multimodal and coding papers explicitly build on the Dream-7B backbone family, whereas the biomedical and retrieval papers are separate research programs with different objectives, architectures, and evaluation regimes [2509.01142] [2512.22615].

## 2. Diffusion formulation and training objective

Dream-7B is framed against the conventional AR factorization. For a token sequence \(x \coloneqq (x^1,\dots,x^N)\), the paper writes the AR model as
$$
p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),
$$
which enforces progressive left-context prediction [2508.15487]. Dream instead adopts a masked discrete diffusion formulation in which a clean sequence \(x_0 \coloneqq x\) is corrupted by replacing tokens with \([MASK]\), producing increasingly noisy sequences \(x_1,\dots,x_T\), and then reconstructed by a reverse denoising process:
$$
p_{\theta}(x)=\sum_{x_{1:T}\sim q}p(x_T)\prod_{t=1}^Tp_{\theta}(x_{t-1}|x_t).
$$
The operational difference is that Dream predicts masked positions from full bidirectional context rather than from left context alone.

The forward process is an absorbing-state masked diffusion process. Each token is independently retained or replaced by \([MASK]\) according to a noise schedule \(\alpha_t\), with the paper highlighting the continuous-time schedule
$$
\alpha_t = 1-t,
$$
for \(t \in [0,1]\). As \(t \to 0\), the sequence is nearly clean; as \(t \to 1\), progressively more tokens are masked until the noisiest endpoint approaches a fully masked sequence [2508.15487].

Training is cast as denoising only the masked positions through a weighted cross-entropy objective:
$$
L(\theta) = -\mathbb{E}_{x_0 \sim q(x),\,t\sim \mathcal{U}(0,1),\, x_t\sim q(x_t|x_0)} \, w(t) \sum_{n=1}^N \mathbf{1}_{[\mathrm{x}_t^n=\mathrm{MASK}]} \log p_{\theta}(x_0^n|x_t),
$$
with
$$
w(t) = \frac{1}{t}
$$
under the schedule \(\alpha_t = 1-t\). This emphasizes denoising examples closer to the clean-data regime.

A central methodological addition is **CART: Context-Adaptive noise Rescheduling at Token-level**. Rather than assigning a single effective noise level to the whole sequence, CART introduces token-specific weighting:
$$
L(\theta) = -\mathbb{E}_{x_0 \sim q(x),\,t\sim \mathcal{U}(0,1),\, x_t\sim q(x_t|x_0)} \sum_{n=1}^N \mathbf{1}_{[\mathrm{x}_t^n=\mathrm{MASK}]} \, w(t, x_t, n)\log p_{\theta}(x_0^n|x_t),
$$
where
$$
w(t, x_t, n)=\frac{1}{2}\sum_{i=1}^{N}\mathbf{1}_{[\mathrm{x}_t^i\neq \mathrm{MASK}]}\operatorname{Geo}(p,|n-i|-1).
$$
The stated motivation is that masked tokens with richer visible context are effectively less noisy than masked tokens in sparse or distant contexts, so token-level rescheduling better matches the actual denoising difficulty [2508.15487].

## 3. Model construction and training recipe

Dream-7B is not presented as a bespoke transformer architecture. It is built on the standard **Transformer** architecture and uses the **same model configuration as Qwen2.5-7B** [2508.15487]. Its most important construction choice is initialization from the AR checkpoint **Qwen2.5-7B**, rather than diffusion pretraining from scratch. The paper describes this as preserving the AR model’s learned “shift” structure: a hidden state \(h_i\) in the pretrained causal LM is aligned to predict position \(i+1\), and Dream retains this relation during diffusion training via the **Shift Operation** inherited from DiffuLLaMA. The stated purpose is to maximize compatibility with AR pretrained weights while adapting them to bidirectional denoising.

The pretraining corpus totals **580 billion tokens** across general text, mathematics, and code, drawn from **Dolma v1.7**, **OpenCoder**, and **DCLM-Baseline** [2508.15487]. The paper emphasizes that these are open datasets. It also states that strong diffusion performance is achieved with only **0.6T training tokens**, a point used in direct comparison with LLaDA 8B.

The released model family contains at least two public variants. **Dream-Base** is the pretrained diffusion model. **Dream-Instruct** is obtained by supervised fine-tuning for instruction following. For instruction tuning, the paper constructs a **1.8M instruction-response pair** dataset from **Tulu 3** and **SmolLM 2**, applies noise only to the response portion while keeping the prompt clean, and fine-tunes Dream-Instruct for **3 epochs** [2508.15487]. The work explicitly describes this as an early exploration of post-training for diffusion LLMs rather than a full RLHF- or RLAIF-style alignment pipeline.

The ablation most strongly emphasized concerns AR initialization. On a smaller Dream-1B experiment adapted from LLaMA3.2-1B, AR initialization substantially lowers training loss relative to training from scratch throughout the first **200B tokens**. The paper also stresses that learning rate is critical: if it is too high, useful left-to-right knowledge from the AR checkpoint is rapidly lost; if it is too low, the model fails to adapt effectively to the diffusion objective [2508.15487].

## 4. Inference mechanics and generation behavior

Inference in Dream-7B follows the standard discrete diffusion picture. Generation starts from a highly corrupted, often fully masked sequence \(x_T\), then iteratively applies the learned reverse process down to \(x_0\) [2508.15487]. Because the model predicts masked positions conditioned on the full current sequence, it can update many locations in parallel at each denoising step. This stands in contrast to AR generation, which is tightly coupled to a one-token-per-step left-to-right decoding order.

The paper treats the number of denoising timesteps as an **inference-time knob**. Fewer denoising steps make generation faster but usually reduce quality; more steps improve quality at additional cost. This is presented as a practical advantage rather than merely an implementation detail, because it creates a tunable speed-quality trade-off unavailable in fixed-form AR decoding [2508.15487].

Dream-7B’s diffusion formulation also yields **arbitrary-order generation**. The paper highlights three natural regimes. The first is ordinary completion, where a prefix is supplied and the remainder is masked. The second is infilling, where left and right context are given with a masked gap between them. The third is configurable decoding order, where decoding hyperparameters can make generation behave more like structured left-to-right decoding or more parallel and non-monotonic. The paper does not introduce a specialized infilling objective for the base model; rather, it states that these behaviors emerge naturally from the diffusion formulation.

These generation properties are tied to a broader claim about planning and holistic sequence reasoning. Because Dream revises a partially formed sequence globally instead of committing to token choices irreversibly, the paper attributes to it stronger planning abilities, more flexible inference, and improved performance on constraint-heavy tasks such as Countdown, Sudoku, and Trip Planning [2508.15487]. A plausible implication is that Dream’s inductive bias is especially suited to tasks in which globally consistent structure is more important than local next-token fluency.

## 5. Empirical profile relative to diffusion and autoregressive baselines

The main benchmark narrative is twofold. First, Dream-7B strongly outperforms prior open diffusion LMs, especially **LLaDA 8B**, despite using only **0.6T** training tokens versus **2.3T** for LLaDA [2508.15487]. Second, it becomes competitive with its AR donor model **Qwen2.5 7B** on many standard evaluations while exceeding it on several reasoning, coding, and planning tasks.

Relative to **LLaDA 8B**, the reported gains are large across general, mathematical, coding, and planning tasks. Dream vs LLaDA scores are **69.5 vs 65.9 on MMLU**, **57.9 vs 47.4 on BBH**, **59.8 vs 47.5 on ARC-C**, **77.2 vs 70.9 on GSM8K**, **39.6 vs 30.7 on MATH**, **57.9 vs 32.9 on HumanEval**, and **81.0 vs 46.0 on Sudoku** [2508.15487]. These comparisons are used to support the effectiveness of AR initialization together with CART.

Against **Qwen2.5 7B**, the model is closer than earlier diffusion systems while not uniformly superior. Dream scores **69.5 vs 71.9 on MMLU**, **57.9 vs 63.9 on BBH**, **73.3 vs 79.0 on HellaSwag**, **77.2 vs 78.9 on GSM8K**, **39.6 vs 41.1 on MATH**, and **57.9 vs 56.7 on HumanEval** [2508.15487]. The reported interpretation is not that Dream dominates AR baselines on standard broad benchmarks, but that diffusion LMs can now be competitive while offering distinct inference behavior and structured-reasoning strengths.

Those strengths are clearest on planning and constraint-satisfaction tasks. On **Countdown**, Dream scores **16.0** versus **6.2** for Qwen2.5 7B and **13.2** for LLaDA 8B. On **Sudoku**, it scores **81.0** versus **21.0** for Qwen2.5 and **46.0** for LLaDA. On **Trip Planning**, it reaches **17.8** versus **3.6** for Qwen2.5 and **16.4** for LLaDA [2508.15487]. The paper attributes this to diffusion’s capacity for global iterative refinement under multiple constraints.

Dream-Instruct improves over prior diffusion instruction models, but remains behind mature AR post-training pipelines on many evaluations. The reported Dream-Instruct scores are **67.0 on MMLU**, **43.3 on MMLU-Pro**, **81.0 on GSM8K**, **39.2 on MATH**, **33.0 on GPQA**, **55.5 on HumanEval**, **58.8 on MBPP**, and **62.5 on IFEval** [2508.15487]. The paper interprets this as evidence that diffusion post-training is viable but still underdeveloped relative to AR pipelines that employ stronger RL-based alignment.

A further empirical claim concerns test-time compute. On the Countdown task, the quality-speed figure indicates that in the range of about **5–20 denoising steps**, Dream can occupy a region in which it is superior to Qwen2.5 7B in both speed and quality for that task [2508.15487]. This is used to argue that denoising-step control is not merely overhead, but an additional axis of test-time scaling.

## 6. Descendants, specializations, and multimodal extensions

Dream-7B quickly became the basis for specialized descendants. The most direct code-focused extension is **Dream-Coder 7B**, which the paper explicitly positions as a code-specialized descendant of Dream 7B rather than a mere alias for the base model [2509.01142]. Dream-Coder 7B is built from the **Qwen2.5-Coder 7B** checkpoint, adapted into a discrete diffusion model with a continuous-time weighted cross-entropy objective and Context-Adaptive Token-Level Noise Rescheduling. Its pretraining corpus totals **322B tokens**, drawn from **OpenCoder**, **Stack-Edu**, **Dolmino**, and **DCLM-Baseline**. The instruction-tuned variant introduces supervised fine-tuning on **5 million** examples from **Ling-Coder-SFT** together with reinforcement learning using **GRPO** over a **17k prompt** unit-test-verified dataset curated from **KodCode-V1**, **DeepCoder-Preview**, and **Guru-RL-92k** [2509.01142].

Dream-Coder 7B is presented as exhibiting emergent **any-order generation** behavior adapted to programming tasks. The paper reports **sketch-first scaffolding** on LiveCodeBench-style algorithmic problems, **left-to-right completion** on tasks such as HumanEval and MBPP, and **interleaved reasoning generation** on CRUXEval [2509.01142]. The headline instruction result is **21.4% pass@1 on LiveCodeBench (2410–2505)**, alongside **82.9 on HumanEval**, **79.6 on MBPP**, **73.1 on EvalPlus**, **37.1 on BigCodeBench Full**, **17.6 on BigCodeBench Hard**, **64.6 on CRUXEval Input-CoT**, and **63.1 on CRUXEval Output-CoT** [2509.01142]. These results materially exceed the general Dream-7B-Instruct baseline on coding-specific evaluations.

The multimodal extension line is described in "Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone" [2512.22615]. That paper explicitly states that Dream-VL is built on the diffusion language model **Dream-7B**, and Table 1 names the concrete backbone as **Dream-v0-Instruct-7B**. Dream-VL combines **Qwen2ViT** as the vision tower with the Dream-v0-Instruct-7B language backbone, trains with the **same discrete diffusion loss as Dream 7B**, and reaches a full-model size of **8.3B** parameters in later multimodal training stages [2512.22615].

Dream-VLA continues pretraining from Dream-VL on **970k robot manipulation trajectories** from the **Open-X Embodiment dataset** and is evaluated on downstream robotics benchmarks [2512.22615]. The reported results are **97.2% average success rate on LIBERO**, **71.4% overall average on SimplerEnv-Bridge**, and **60.5% overall average on SimplerEnv-Fractal**. The paper also attributes strong planning behavior and action chunking to the bidirectional diffusion backbone, noting that Dream-VL can work well with **1 diffusion step** for multiple low-level actions and that predicting **12 actions at once** yields **27× speedup** in one low-level action setting [2512.22615]. These extensions position Dream-7B not merely as a text-only model, but as a transferable diffusion backbone for code generation, vision-language reasoning, and vision-language-action control.

## 7. Limitations, open questions, and common misconceptions

The primary limitation acknowledged in the Dream-7B paper is that the model is **not uniformly better than strong AR models on conventional benchmarks** [2508.15487]. Qwen2.5 remains stronger on several standard evaluations, especially after post-training, and Dream-Instruct uses only supervised fine-tuning rather than more elaborate RL-based alignment. The paper also identifies **advanced post-training** and **longer context** as open directions. Inference latency is another trade-off: although Dream offers a denoising-step quality-speed control, raw latency still depends heavily on the number of steps chosen.

A common misconception is to interpret any paper using the token **DREAM** or **Dream** as documenting the same model family. That is incorrect. The 2024 biomedical DREAM paper is a modular autonomous research system with modules such as **dataInterpreter**, **questionRaiser**, **codeMaker**, **dockerMaker**, **resultJudger**, and **deepQuestioner**, and it does **not** introduce a model called Dream-7B [2407.13637]. The 2026 retrieval DREAM paper introduces a retriever-training method that injects query-document similarity scores into selected attention heads of a frozen LLM and explicitly reports **DREAM-8B**, but it does **not** report any model called Dream-7B [2606.24667]. By contrast, the multimodal Dream-VL paper provides direct evidence that a Dream-7B checkpoint family exists and that **Dream-v0-Instruct-7B** is used as the backbone for released multimodal descendants [2512.22615].

Another misconception is that Dream-7B should be understood only as a code model. The most accurate interpretation is the reverse: **Dream-Coder 7B** is the code-specialized descendant, whereas **Dream 7B** is the general diffusion LLM from which that specialization departs [2509.01142]. Similarly, Dream-VL and Dream-VLA are not alternate names for the base model, but multimodal systems built upon it [2512.22615].

Taken together, these papers situate Dream-7B as a general 7B-class discrete diffusion LLM whose main research significance lies in demonstrating that diffusion language modeling can be competitive with modern AR baselines while supporting bidirectional denoising, arbitrary-order generation, natural infilling, and test-time denoising-step control. The later code and multimodal descendants suggest that these properties transfer beyond generic text modeling into code generation, visual planning, and robot control, although the evidence also indicates that diffusion post-training and broad benchmark parity with the strongest AR systems remain active research problems [2508.15487].

Source: https://www.emergentmind.com/topics/dream-7b-c6df8262-070c-4319-a9c2-0fd3e6eb7050