Dream 7B: Open Diffusion LLM for Text, Code & Vision

• Dream 7B is a 7-billion-parameter diffusion language model that iteratively refines token sequences and supports applications in text, code, and multimodal tasks.
• Its diffusion-based approach leverages continuous-time parameterization and context-adaptive token noise rescheduling (CART) for flexible, parallel sequence refinement.
• By employing AR-to-diffusion adaptation, Dream 7B efficiently transfers pretrained knowledge and achieves competitive performance on planning, code generation, and cross-modal reasoning.

Dream 7B is a 7-billion-parameter open diffusion LLM (DLLM) that redefines language modeling, code generation, and multimodal (vision-language, vision-language-action) paradigms by adopting discrete diffusion-based sequence refinement in place of the conventional autoregressive (AR) next-token generation framework. The technical lineage and deployment of Dream 7B span the foundational text generation model itself (Ye et al., 21 Aug 2025), its code-specialized variant Dream-Coder 7B (Xie et al., 1 Sep 2025), and its role as the backbone for vision-language and robotics architectures Dream-VL-7B and Dream-VLA-7B (Ye et al., 27 Dec 2025). This entry surveys the modeling framework, training protocols, decoding properties, task-specific instantiations, evaluation, and methodological implications of Dream 7B and its ecosystem.

1. Foundational Modeling Paradigm

Dream 7B is a discrete diffusion LLM: rather than sequence modeling via the left-to-right probability factorization

$$p_\theta(x) = p_\theta(x^1)\prod_{n=2}^N p_\theta(x^n \mid x^{1:n-1}),$$

it generates text through iterative denoising of corrupted (partially masked) token sequences, leveraging bidirectional context and enabling non-sequential, parallel refinement. The generative process is expressed as

$$p_\theta(x) = \sum_{x_{1:T}\sim q} p(x_T) \prod_{t=1}^T p_\theta(x_{t-1} \mid x_t),$$

with $x_0$ the clean target sequence, $x_t$ its masked/noised versions at diffusion timestep $t$, and $q(x_{1:T}\mid x_0)$ the forward noising process substituting tokens with an absorbing [MASK] symbol. Unlike finite-step discrete diffusion (as in prior text diffusion efforts), Dream 7B adopts a continuous-time parameterization $t\in[0,1]$, supporting arbitrary noise levels and resulting in both more flexible training and improved inference control.

The primary diffusion loss, for each noised sequence $x_t$ at level $t$, is a weighted cross-entropy computed only over masked positions:

$$L(\theta) = -\mathbb{E}_{x_0,t,x_t} \, w(t) \sum_{n=1}^N 1_{[x_t^n=\mathrm{MASK}]} \log p_\theta(x_0^n|x_t),$$

with the weighting function $$p_\theta(x) = \sum_{x_{1:T}\sim q} p(x_T) \prod_{t=1}^T p_\theta(x_{t-1} \mid x_t),$$0 for the linear masking schedule $$p_\theta(x) = \sum_{x_{1:T}\sim q} p(x_T) \prod_{t=1}^T p_\theta(x_{t-1} \mid x_t),$$1, favoring denoising decisions at lower noise closer to $$p_\theta(x) = \sum_{x_{1:T}\sim q} p(x_T) \prod_{t=1}^T p_\theta(x_{t-1} \mid x_t),$$2.

Dream 7B further introduces context-adaptive token-level noise rescheduling (CART), where $$p_\theta(x) = \sum_{x_{1:T}\sim q} p(x_T) \prod_{t=1}^T p_\theta(x_{t-1} \mid x_t),$$3 is replaced by $$p_\theta(x) = \sum_{x_{1:T}\sim q} p(x_T) \prod_{t=1}^T p_\theta(x_{t-1} \mid x_t),$$4, a geometric-distance weighted function reflecting each masked token’s effective context, allowing the loss to be sensitive to local prediction difficulty.

2. Model Architecture, Initialization, and Training Procedures

Dream 7B adopts a full transformer architecture and parameter configuration matching Qwen2.5-7B. Crucially, it is not trained from scratch: parameter initialization is performed via AR-to-diffusion adaptation. The core adaptation is the shift operation, wherein the AR model’s next-token prediction alignment (hidden $$p_\theta(x) = \sum_{x_{1:T}\sim q} p(x_T) \prod_{t=1}^T p_\theta(x_{t-1} \mid x_t),$$5 predicts $$p_\theta(x) = \sum_{x_{1:T}\sim q} p(x_T) \prod_{t=1}^T p_\theta(x_{t-1} \mid x_t),$$6) is preserved in the denoising model, avoiding representational misalignment and enabling the efficient reuse of existing pretrained AR weights.

Pretraining covers 580B tokens (text, math, code), with fully open datasets: Dolma v1.7 (text), OpenCoder (code), and DCLM-Baseline (curated math/text). For code specialization, Dream-Coder 7B (Xie et al., 1 Sep 2025) is initialized from Qwen2.5-Coder-7B, then retrofitted to the diffusion objective and further pre-trained on 322B tokens (OpenCoder, Stack-Edu, Dolmino, DCLM-Baseline).

Supervised fine-tuning (SFT, for Dream-Instruct and Dream-Coder-7B-Instruct) is performed with open instruction datasets (e.g., Tulu 3, SmolLM 2 for Dream-7B; Ling-Coder-SFT for Dream-Coder-7B), applying diffusion noise only to the response region and preserving the instruction prompt. Post-SFT, Dream-Coder-7B-Instruct is further improved via RL with verifiable, unit-test-based rewards and a specially stabilized PPO/GRPO recipe tailored for diffusion sequence models.

3. Decoding, Inference, and Emergent Generation Properties

Generation in Dream 7B proceeds by iterative refinement: starting with a masked (noisy) sequence, the model repeatedly denoises using global (bidirectional) context, updating masked tokens in multiple steps until a clean sequence is produced. Unlike strictly AR LLMs, Dream supports:

• Arbitrary-order decoding and in-filling (not restricted to left-to-right).
• Parallel refinement: multiple masked positions updated per step.
• Quality-speed tradeoffs: user can adjust the number of diffusion steps, achieving faster but less refined outputs or slower, higher-quality generations.
• Adaptive, task-dependent emergent generation order. Dream-Coder 7B displays sketch-first (scaffolding-first) synthesis for complex code, standard left-to-right for completion, and interleaved reasoning for logic-intensive code understanding (Xie et al., 1 Sep 2025).

Inference can start in pure completion, span infilling, or arbitrary masking configurations. Diffusion decoding is more expensive than AR generation due to multiple refinement steps, but partial acceleration is achieved via techniques such as Fast-dLLM.

4. Benchmarks and Empirical Results

Dream 7B and its variants are evaluated across textual, mathematical, and coding tasks, with key results:

Model/Variant	Params	MMLU	GSM8K	HumanEval	MBPP	LiveCodeBench (pass@1)	Planning (Sudoku, Countdown)
Dream 7B (Base) (Ye et al., 21 Aug 2025)	7B	69.5	77.2	57.9	56.2	—	81.0 / 16.0
Dream-Instruct (Ye et al., 21 Aug 2025)	7B	67.0	81.0	55.5	58.8	—	—
Dream-Coder-7B (Xie et al., 1 Sep 2025)	7B	65.6	71.1	66.5	75.9	—	—
Dream-Coder-7B-Instruct (Xie et al., 1 Sep 2025)	7B	—	—	82.9	79.6	21.4	—

On general reasoning benchmarks (MMLU, GSM8K), math (MATH, GPQA), and code generation (HumanEval, MBPP), Dream 7B is competitive with state-of-the-art AR LLMs of similar scale (Qwen2.5 7B), often substantially outperforming previous open diffusion models (LLaDA 8B, DiffuLLaMA). It is especially strong on planning tasks such as Sudoku and Countdown, where it surpasses even much larger AR models on specific settings.

Dream-Coder-7B-Instruct attains 21.4% pass@1 on LiveCodeBench, matching proprietary models such as Claude 3 Haiku (21.2) and Mercury Coder Small (22.9) and outperforming other open-weight code LMs (OpenCoder-8B-Instruct: 10.6; Qwen2.5-Coder-7B-Instruct: 19.3). It is robust across coding, reasoning, and code understanding tasks.

In the multimodal setting (Ye et al., 27 Dec 2025), Dream 7B (as 'Dream-v0-Instruct-7B') underlies Dream-VL-7B and Dream-VLA-7B:

• Dream-VL-7B achieves 52.2 MMMU, 83.0 MMBench, and 63.1 MathVista, rivaling leading AR VLMs on several metrics.
• Dream-VLA-7B establishes SOTA results on LIBERO (97.2% average success), with native support for parallel action chunking and fast finetuning convergence, outperforming prior AR-based VLA models.

5. Task-Specific Architectural Extensions

Code Generation: Dream-Coder 7B

Dream-Coder 7B (Xie et al., 1 Sep 2025), derived from Dream 7B, adapts Qwen2.5-Coder-7B to the diffusion framework with shift operation, continuous-time weighted masked denoising, and context-adaptive rescheduling. It supports:

• Emergent sketch-first, any-order, and interleaved reasoning generation.
• Post-training stabilization via random truncation and padding penalties.
• RL with verifiable (unit test) rewards, employing diffusion-stable GRPO with group-normalized advantages and coupled informative substitution, as well as accelerated diffusion rollout.

Vision-Language and Robotics: Dream-VL-7B and Dream-VLA-7B

In the multimodal sphere (Ye et al., 27 Dec 2025), Dream-VL-7B and Dream-VLA-7B integrate

• A Qwen2ViT vision encoder and Dream-v0-Instruct-7B diffusion LLM backbone (total model: 8.3B parameters).
• A multimodal connector for vision-text fusion via concatenation and projector tuning.
• Bidirectional attention and masked diffusion denoising for robust cross-modal representation and plan/action generation.
• For Dream-VLA-7B, downstream continuous pretraining on open robotics datasets (Open-X Embodiment), and support for both discrete diffusion and continuous action flow matching objectives, enabling parallel robot action generation and native chunking.

6. Methodological Implications, Strengths, and Limitations

• Diffusion LLMs like Dream 7B demonstrate that competitive linguistic, reasoning, planning, and code generation abilities can be achieved without the sequential AR constraint.
• Bidirectional, iterative denoising offers advantages for planning, constraint satisfaction, flexible infilling, and code synthesis, as evidenced by strong results on planning and complex code tasks.
• The shift operation enables effective transfer of AR-pretrained knowledge into diffusion LMs, preserving linguistic structure and accelerating convergence.
• The context-adaptive noise rescheduling (CART) mechanism enhances learning by dynamically weighting the denoising difficulty of each token, improving both pretraining efficiency and generative quality.

Nevertheless, diffusion decoding incurs higher inference cost vs. one-pass AR models, requiring multiple denoising steps—partially mitigated via acceleration techniques. Dream-Instruct and Dream-Coder-7B-Instruct, despite strong performance, do not yet close the gap to top RLHF-tuned AR LLMs on all benchmarks. Implementation details such as exact hyperparameters, context lengths, or padding-penalty formulas are not fully specified in published text.

7. Released Artifacts and Practical Utility

Dream 7B and its ecosystem are open source, with released checkpoints for:

• Dream-Base and Dream-Instruct (Ye et al., 21 Aug 2025)
• Dream-Coder-7B and Dream-Coder-7B-Instruct (Xie et al., 1 Sep 2025)
• Dream-VL-7B and Dream-VLA-7B, plus code and training recipes (Ye et al., 27 Dec 2025)

These models collectively enable research into diffusion-based language, code, vision-language, and vision-language-action modeling at 7B–8B scale, supporting flexible generation and strong performance under open training data constraints. Dream 7B serves as an open diffusion backbone for advancing text, code, and multimodal research, with special strengths in planning, infilling, and complex sequence refinement outside the AR paradigm.