Dream-7B: Diffusion LLM

• Dream-7B is a diffusion large language model with 7B parameters that uses iterative masked denoising for bidirectional text generation.
• It leverages autoregressive checkpoint initialization and introduces Context-Adaptive noise Rescheduling at Token-level (CART) to enhance denoising effectiveness.
• Dream-7B serves as the backbone for specialized variants like Dream-Coder and Dream-VL, extending its applications to coding and multimodal tasks.

Searching arXiv for the relevant Dream-7B papers and closely related work. Dream-7B is a 7-billion-parameter diffusion LLM (DLLM) introduced as an open alternative to standard autoregressive (AR) LLMs. Its defining departure from AR modeling is that it generates by iteratively denoising a masked sequence in parallel, rather than emitting tokens strictly left-to-right by next-token prediction. The model is presented as a general-purpose base LLM, with public variants Dream-Base and Dream-Instruct, and is subsequently used as the backbone family for code and multimodal descendants such as Dream-Coder 7B, Dream-VL, and Dream-VLA (Ye et al., 21 Aug 2025). The name should not be conflated with two unrelated systems that also use the acronym DREAM: the biomedical autonomous research framework in "Autonomous self-evolving research on biomedical data: the DREAM paradigm" and the dense retrieval method in "DREAM: Dense Retrieval Embeddings via Autoregressive Modeling" (Deng et al., 2024, Tang et al., 23 Jun 2026).

1. Identity, scope, and disambiguation

Dream-7B denotes the diffusion LLM family introduced in "Dream 7B: Diffusion LLMs" (Ye et al., 21 Aug 2025). In that work, the model is described as a strong open diffusion LLM designed to narrow the performance gap between discrete diffusion LLMs and modern 7B-class AR baselines such as Qwen2.5-7B, while preserving diffusion-specific capabilities including arbitrary-order generation, infilling, and a tunable quality-speed trade-off.

A recurring source of ambiguity is that several unrelated papers use the string DREAM. The biomedical paper from 2024 uses DREAM as shorthand for Data-driven self-Evolving Autonomous systeM, a modular autonomous biomedical research system rather than a 7B checkpoint (Deng et al., 2024). The 2026 retrieval paper uses DREAM as Dense Retrieval Embeddings via Autoregressive Modeling, a method for training dense retrievers via frozen-LLM supervision rather than a generative LLM (Tang et al., 23 Jun 2026). In the context of generative modeling, Dream-7B refers specifically to the diffusion LLM line.

Name	arXiv id	Scope
Dream 7B	(Ye et al., 21 Aug 2025)	Diffusion LLM
DREAM	(Deng et al., 2024)	Autonomous biomedical research system
DREAM	(Tang et al., 23 Jun 2026)	Dense retrieval embedding method

This naming distinction matters because later multimodal and coding papers explicitly build on the Dream-7B backbone family, whereas the biomedical and retrieval papers are separate research programs with different objectives, architectures, and evaluation regimes (Xie et al., 1 Sep 2025, Ye et al., 27 Dec 2025).

2. Diffusion formulation and training objective

Dream-7B is framed against the conventional AR factorization. For a token sequence $x \coloneqq (x^1,\dots,x^N)$, the paper writes the AR model as

$$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$

which enforces progressive left-context prediction (Ye et al., 21 Aug 2025). Dream instead adopts a masked discrete diffusion formulation in which a clean sequence $x_0 \coloneqq x$ is corrupted by replacing tokens with $[MASK]$, producing increasingly noisy sequences $x_1,\dots,x_T$, and then reconstructed by a reverse denoising process:

$$p_{\theta}(x)=\sum_{x_{1:T}\sim q}p(x_T)\prod_{t=1}^Tp_{\theta}(x_{t-1}|x_t).$$

The operational difference is that Dream predicts masked positions from full bidirectional context rather than from left context alone.

The forward process is an absorbing-state masked diffusion process. Each token is independently retained or replaced by $[MASK]$ according to a noise schedule $\alpha_t$, with the paper highlighting the continuous-time schedule

$\alpha_t = 1-t,$

for $t \in [0,1]$. As $$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$0, the sequence is nearly clean; as $$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$1, progressively more tokens are masked until the noisiest endpoint approaches a fully masked sequence (Ye et al., 21 Aug 2025).

Training is cast as denoising only the masked positions through a weighted cross-entropy objective:

$$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$2

with

$$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$3

under the schedule $$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$4. This emphasizes denoising examples closer to the clean-data regime.

A central methodological addition is CART: Context-Adaptive noise Rescheduling at Token-level. Rather than assigning a single effective noise level to the whole sequence, CART introduces token-specific weighting:

$$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$5

where

$$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$6

The stated motivation is that masked tokens with richer visible context are effectively less noisy than masked tokens in sparse or distant contexts, so token-level rescheduling better matches the actual denoising difficulty (Ye et al., 21 Aug 2025).

3. Model construction and training recipe

Dream-7B is not presented as a bespoke transformer architecture. It is built on the standard Transformer architecture and uses the same model configuration as Qwen2.5-7B (Ye et al., 21 Aug 2025). Its most important construction choice is initialization from the AR checkpoint Qwen2.5-7B, rather than diffusion pretraining from scratch. The paper describes this as preserving the AR model’s learned “shift” structure: a hidden state $$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$7 in the pretrained causal LM is aligned to predict position $$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$8, and Dream retains this relation during diffusion training via the Shift Operation inherited from DiffuLLaMA. The stated purpose is to maximize compatibility with AR pretrained weights while adapting them to bidirectional denoising.

The pretraining corpus totals 580 billion tokens across general text, mathematics, and code, drawn from Dolma v1.7, OpenCoder, and DCLM-Baseline (Ye et al., 21 Aug 2025). The paper emphasizes that these are open datasets. It also states that strong diffusion performance is achieved with only 0.6T training tokens, a point used in direct comparison with LLaDA 8B.

The released model family contains at least two public variants. Dream-Base is the pretrained diffusion model. Dream-Instruct is obtained by supervised fine-tuning for instruction following. For instruction tuning, the paper constructs a 1.8M instruction-response pair dataset from Tulu 3 and SmolLM 2, applies noise only to the response portion while keeping the prompt clean, and fine-tunes Dream-Instruct for 3 epochs (Ye et al., 21 Aug 2025). The work explicitly describes this as an early exploration of post-training for diffusion LLMs rather than a full RLHF- or RLAIF-style alignment pipeline.

The ablation most strongly emphasized concerns AR initialization. On a smaller Dream-1B experiment adapted from LLaMA3.2-1B, AR initialization substantially lowers training loss relative to training from scratch throughout the first 200B tokens. The paper also stresses that learning rate is critical: if it is too high, useful left-to-right knowledge from the AR checkpoint is rapidly lost; if it is too low, the model fails to adapt effectively to the diffusion objective (Ye et al., 21 Aug 2025).

4. Inference mechanics and generation behavior

Inference in Dream-7B follows the standard discrete diffusion picture. Generation starts from a highly corrupted, often fully masked sequence $$p_{\theta}(x)=p_{\theta}(x^1)\prod_{n=2}^{N}p_{\theta}(x^n \mid x^{1:n-1}),$$9, then iteratively applies the learned reverse process down to $x_0 \coloneqq x$0 (Ye et al., 21 Aug 2025). Because the model predicts masked positions conditioned on the full current sequence, it can update many locations in parallel at each denoising step. This stands in contrast to AR generation, which is tightly coupled to a one-token-per-step left-to-right decoding order.

The paper treats the number of denoising timesteps as an inference-time knob. Fewer denoising steps make generation faster but usually reduce quality; more steps improve quality at additional cost. This is presented as a practical advantage rather than merely an implementation detail, because it creates a tunable speed-quality trade-off unavailable in fixed-form AR decoding (Ye et al., 21 Aug 2025).

Dream-7B’s diffusion formulation also yields arbitrary-order generation. The paper highlights three natural regimes. The first is ordinary completion, where a prefix is supplied and the remainder is masked. The second is infilling, where left and right context are given with a masked gap between them. The third is configurable decoding order, where decoding hyperparameters can make generation behave more like structured left-to-right decoding or more parallel and non-monotonic. The paper does not introduce a specialized infilling objective for the base model; rather, it states that these behaviors emerge naturally from the diffusion formulation.

These generation properties are tied to a broader claim about planning and holistic sequence reasoning. Because Dream revises a partially formed sequence globally instead of committing to token choices irreversibly, the paper attributes to it stronger planning abilities, more flexible inference, and improved performance on constraint-heavy tasks such as Countdown, Sudoku, and Trip Planning (Ye et al., 21 Aug 2025). A plausible implication is that Dream’s inductive bias is especially suited to tasks in which globally consistent structure is more important than local next-token fluency.

5. Empirical profile relative to diffusion and autoregressive baselines

The main benchmark narrative is twofold. First, Dream-7B strongly outperforms prior open diffusion LMs, especially LLaDA 8B, despite using only 0.6T training tokens versus 2.3T for LLaDA (Ye et al., 21 Aug 2025). Second, it becomes competitive with its AR donor model Qwen2.5 7B on many standard evaluations while exceeding it on several reasoning, coding, and planning tasks.

Relative to LLaDA 8B, the reported gains are large across general, mathematical, coding, and planning tasks. Dream vs LLaDA scores are 69.5 vs 65.9 on MMLU, 57.9 vs 47.4 on BBH, 59.8 vs 47.5 on ARC-C, 77.2 vs 70.9 on GSM8K, 39.6 vs 30.7 on MATH, 57.9 vs 32.9 on HumanEval, and 81.0 vs 46.0 on Sudoku (Ye et al., 21 Aug 2025). These comparisons are used to support the effectiveness of AR initialization together with CART.

Against Qwen2.5 7B, the model is closer than earlier diffusion systems while not uniformly superior. Dream scores 69.5 vs 71.9 on MMLU, 57.9 vs 63.9 on BBH, 73.3 vs 79.0 on HellaSwag, 77.2 vs 78.9 on GSM8K, 39.6 vs 41.1 on MATH, and 57.9 vs 56.7 on HumanEval (Ye et al., 21 Aug 2025). The reported interpretation is not that Dream dominates AR baselines on standard broad benchmarks, but that diffusion LMs can now be competitive while offering distinct inference behavior and structured-reasoning strengths.

Those strengths are clearest on planning and constraint-satisfaction tasks. On Countdown, Dream scores 16.0 versus 6.2 for Qwen2.5 7B and 13.2 for LLaDA 8B. On Sudoku, it scores 81.0 versus 21.0 for Qwen2.5 and 46.0 for LLaDA. On Trip Planning, it reaches 17.8 versus 3.6 for Qwen2.5 and 16.4 for LLaDA (Ye et al., 21 Aug 2025). The paper attributes this to diffusion’s capacity for global iterative refinement under multiple constraints.

Dream-Instruct improves over prior diffusion instruction models, but remains behind mature AR post-training pipelines on many evaluations. The reported Dream-Instruct scores are 67.0 on MMLU, 43.3 on MMLU-Pro, 81.0 on GSM8K, 39.2 on MATH, 33.0 on GPQA, 55.5 on HumanEval, 58.8 on MBPP, and 62.5 on IFEval (Ye et al., 21 Aug 2025). The paper interprets this as evidence that diffusion post-training is viable but still underdeveloped relative to AR pipelines that employ stronger RL-based alignment.

A further empirical claim concerns test-time compute. On the Countdown task, the quality-speed figure indicates that in the range of about 5–20 denoising steps, Dream can occupy a region in which it is superior to Qwen2.5 7B in both speed and quality for that task (Ye et al., 21 Aug 2025). This is used to argue that denoising-step control is not merely overhead, but an additional axis of test-time scaling.

6. Descendants, specializations, and multimodal extensions

Dream-7B quickly became the basis for specialized descendants. The most direct code-focused extension is Dream-Coder 7B, which the paper explicitly positions as a code-specialized descendant of Dream 7B rather than a mere alias for the base model (Xie et al., 1 Sep 2025). Dream-Coder 7B is built from the Qwen2.5-Coder 7B checkpoint, adapted into a discrete diffusion model with a continuous-time weighted cross-entropy objective and Context-Adaptive Token-Level Noise Rescheduling. Its pretraining corpus totals 322B tokens, drawn from OpenCoder, Stack-Edu, Dolmino, and DCLM-Baseline. The instruction-tuned variant introduces supervised fine-tuning on 5 million examples from Ling-Coder-SFT together with reinforcement learning using GRPO over a 17k prompt unit-test-verified dataset curated from KodCode-V1, DeepCoder-Preview, and Guru-RL-92k (Xie et al., 1 Sep 2025).

Dream-Coder 7B is presented as exhibiting emergent any-order generation behavior adapted to programming tasks. The paper reports sketch-first scaffolding on LiveCodeBench-style algorithmic problems, left-to-right completion on tasks such as HumanEval and MBPP, and interleaved reasoning generation on CRUXEval (Xie et al., 1 Sep 2025). The headline instruction result is 21.4% pass@1 on LiveCodeBench (2410–2505), alongside 82.9 on HumanEval, 79.6 on MBPP, 73.1 on EvalPlus, 37.1 on BigCodeBench Full, 17.6 on BigCodeBench Hard, 64.6 on CRUXEval Input-CoT, and 63.1 on CRUXEval Output-CoT (Xie et al., 1 Sep 2025). These results materially exceed the general Dream-7B-Instruct baseline on coding-specific evaluations.

The multimodal extension line is described in "Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion LLM Backbone" (Ye et al., 27 Dec 2025). That paper explicitly states that Dream-VL is built on the diffusion LLM Dream-7B, and Table 1 names the concrete backbone as Dream-v0-Instruct-7B. Dream-VL combines Qwen2ViT as the vision tower with the Dream-v0-Instruct-7B language backbone, trains with the same discrete diffusion loss as Dream 7B, and reaches a full-model size of 8.3B parameters in later multimodal training stages (Ye et al., 27 Dec 2025).

Dream-VLA continues pretraining from Dream-VL on 970k robot manipulation trajectories from the Open-X Embodiment dataset and is evaluated on downstream robotics benchmarks (Ye et al., 27 Dec 2025). The reported results are 97.2% average success rate on LIBERO, 71.4% overall average on SimplerEnv-Bridge, and 60.5% overall average on SimplerEnv-Fractal. The paper also attributes strong planning behavior and action chunking to the bidirectional diffusion backbone, noting that Dream-VL can work well with 1 diffusion step for multiple low-level actions and that predicting 12 actions at once yields 27× speedup in one low-level action setting (Ye et al., 27 Dec 2025). These extensions position Dream-7B not merely as a text-only model, but as a transferable diffusion backbone for code generation, vision-language reasoning, and vision-language-action control.

7. Limitations, open questions, and common misconceptions

The primary limitation acknowledged in the Dream-7B paper is that the model is not uniformly better than strong AR models on conventional benchmarks (Ye et al., 21 Aug 2025). Qwen2.5 remains stronger on several standard evaluations, especially after post-training, and Dream-Instruct uses only supervised fine-tuning rather than more elaborate RL-based alignment. The paper also identifies advanced post-training and longer context as open directions. Inference latency is another trade-off: although Dream offers a denoising-step quality-speed control, raw latency still depends heavily on the number of steps chosen.

A common misconception is to interpret any paper using the token DREAM or Dream as documenting the same model family. That is incorrect. The 2024 biomedical DREAM paper is a modular autonomous research system with modules such as dataInterpreter, questionRaiser, codeMaker, dockerMaker, resultJudger, and deepQuestioner, and it does not introduce a model called Dream-7B (Deng et al., 2024). The 2026 retrieval DREAM paper introduces a retriever-training method that injects query-document similarity scores into selected attention heads of a frozen LLM and explicitly reports DREAM-8B, but it does not report any model called Dream-7B (Tang et al., 23 Jun 2026). By contrast, the multimodal Dream-VL paper provides direct evidence that a Dream-7B checkpoint family exists and that Dream-v0-Instruct-7B is used as the backbone for released multimodal descendants (Ye et al., 27 Dec 2025).

Another misconception is that Dream-7B should be understood only as a code model. The most accurate interpretation is the reverse: Dream-Coder 7B is the code-specialized descendant, whereas Dream 7B is the general diffusion LLM from which that specialization departs (Xie et al., 1 Sep 2025). Similarly, Dream-VL and Dream-VLA are not alternate names for the base model, but multimodal systems built upon it (Ye et al., 27 Dec 2025).

Taken together, these papers situate Dream-7B as a general 7B-class discrete diffusion LLM whose main research significance lies in demonstrating that diffusion language modeling can be competitive with modern AR baselines while supporting bidirectional denoising, arbitrary-order generation, natural infilling, and test-time denoising-step control. The later code and multimodal descendants suggest that these properties transfer beyond generic text modeling into code generation, visual planning, and robot control, although the evidence also indicates that diffusion post-training and broad benchmark parity with the strongest AR systems remain active research problems (Ye et al., 21 Aug 2025).