Sumi: Open Uniform Diffusion Language Model from Scratch

Abstract: Diffusion models have become a promising alternative to autoregressive models. Among these, uniform diffusion LLMs (UDLMs) permit any token to be updated at any step, in principle enabling more flexible generation. However, no UDLM has yet been pretrained from scratch at both large parameter scale and large token budget. Both autoregressive modeling and masked diffusion modeling already have capable models at scale that the community can study and build on; uniform diffusion has none. A scratch-pretrained UDLM at scale would provide a clean reference point for studying scaling behavior, generation dynamics, controllability, and trade-offs against established autoregressive and masked diffusion models. To this end, we introduce Sumi ("ink" in Japanese), a fully open 7B uniform diffusion LLM pretrained from scratch on 1.5T tokens. Sumi performs competitively with autoregressive models trained at comparable token budgets on knowledge, reasoning, and coding benchmarks, while under-performing on commonsense benchmarks, where our education-heavy data mixture is a likely contributor. We release our model weights, checkpoints, and full training recipe, including a complete specification of the data mixture over publicly available corpora. We hope this release enables the community to study native uniform diffusion at scale and catalyzes work on its as-yet poorly understood aspects.

• The paper introduces Sumi, the first large-scale native UDLM that allows token updates at any generation step and shows competitive performance.
• It employs a 36-layer bidirectional transformer with grouped-query attention and a GIDD objective for stable training.
• Evaluation reveals strong results in knowledge and coding tasks but a gap in commonsense benchmarks, highlighting the impact of data composition.

Sumi: Open Uniform Diffusion LLM from Scratch

Introduction and Motivation

The paper presents Sumi, a fully open 7B-parameter Uniform Diffusion LLM (UDLM) pretrained from scratch on 1.5T tokens, marking the first large-scale native UDLM. Unlike Masked Diffusion LLMs (MDLMs), where once masked tokens are filled they cannot be revised, UDLMs allow updates to any token at any generation step. This flexibility is theorized to enable enhanced generation dynamics, intrinsic self-correction, and greater controllability compared to autoregressive (AR) and MDLM paradigms. Prior large-scale uniform diffusion models, such as DiffusionGemma, adapted pretrained AR models rather than training natively at scale. Sumi addresses this gap, providing a clean reference for comparative studies across model families.

Model Architecture and Training Procedure

Sumi employs a bidirectional, time-agnostic 36-layer Transformer with grouped-query attention, SwiGLU MLPs, RMSNorm, untied I/O embeddings, and an off-by-one softmax to mitigate attention sinks. The pretrained base model is trained using the Generalized Interpolating Discrete Diffusion (GIDD) objective, SNR-reparameterized, under a pure uniform noise schedule with log-SNR restricted to $[-9, 9]$. Optimization uses AdamW with gradient clipping and an auxiliary z-loss for stability.

Training is staged: pre-training on 1.3T tokens (sequence length 1184), followed by two mid-training phases on 250B tokens (lengths 1184, 4864) from domain-specific mixtures weighted toward code, math, and reasoning. All training is performed on 288 H100 GPUs, totaling 43,308 GPU-hours.

Figure 1: Training data composition for the pre-training phase (1.3T tokens), highlighting the educational and code-centric mixture.

Data Mixture and Corpus Selection

Pre-training uses a curated subset of LLM-jp-corpus-v4, focusing on educational scoring and code content. Key subsets include StarCoder for code and Python from swallow_code_v2. The mid-training mixture is drawn from LLM-jp-corpus-midtraining-v2 alongside filtered MegaMath-Web-Pro-Max content, emphasizing math and reasoning skill acquisition. All data is publicly available, enabling independent reconstruction of the training recipe.

Evaluation and Benchmarking

Evaluations utilize the lm-evaluation-harness, extended for diffusion-based scoring. Thirteen benchmarks span four categories: general knowledge (MMLU, RACE, TruthfulQA), reasoning/math (GSM8K, ARC-Easy/Challenge, GPQA, BBH), coding (HumanEval, MBPP), and commonsense (WinoGrande, PIQA, HellaSwag). Sumi's performance is compared under a matched protocol to Falcon-7B, Llama 2-7B, and OLMo-7B baselines.

On general knowledge and coding tasks, Sumi achieves the highest scores among open models evaluated, attributable to its education- and code-heavy data mixture. On reasoning/math, it is competitive with Llama 2-7B and ahead of Falcon-7B. For commonsense benchmarks, Sumi underperforms, displaying a notable gap likely due to data composition but with an effect larger than data alone can explain.

Generation Dynamics and Inference-Time Analyses

Detailed inference-time probes illuminate several aspects of Sumi's native uniform diffusion behavior:

• Generation Fluency and Canvas Length Sensitivity: Sumi generates fluently within a task-dependent canvas-length band. For all evaluated tasks, a canvas length of 2048 falls within the fluent range. Outside this band, especially at shorter lengths for GSM8K, fluency degrades sharply as measured by Falcon-7B perplexity.

  Figure 2: Generation fluency across canvas lengths, demonstrating task-dependent degradation outside the trained range.

• Commit Order Under Confidence Sampling: Adaptive (confidence-based) sampling imposes a structured commitment order on token updates, despite the model's order-agnostic objective. This self-organized structure is absent under ancestral sampling. The imposed order enables limited parallel decoding, as tokens determined with high confidence are committed earlier.

  Figure 3: Per-position commit order in the answer window, contrasting adaptive with ancestral sampling—the former induces visible structure.

• Parallel Decoding: Sumi supports modest parallel token commitment. Coding tasks maintain accuracy with up to four tokens committed per denoising step; arithmetic reasoning (GSM8K) degrades immediately beyond single-token steps, indicating sensitivity to commitment order.

  Figure 4: Accuracy for varying tokens committed per denoising step ($k$) across tasks, showing preserved coding accuracy up to $k=4$.

• Revision Budget and Self-Correction: Despite the theoretical ability for self-correction, over-denoising yields negligible net token change and almost never alters extracted answers. Edits are mainly round trips ($A \to B \to A$), not directed improvements. This indicates the lack of native self-correction under generic revision settings.

Implications and Future Directions

Sumi establishes a first large-scale reference for native UDLMs, facilitating rigorous comparison of scaling behavior, controllability, and generation characteristics. The exhibited task-dependent fluency, self-organized commitment under confidence sampling, and parallel decoding capacity raise new research questions on generation dynamics distinct from AR and MDLM paradigms. The absence of robust self-correction invites exploration into targeted revision strategies and error-aware denoising setups. The education- and code-heavy data bias sharply impacts commonsense benchmarks, underscoring the influence of training recipes on emergent abilities.

Practically, release of full weights, checkpoints, and detailed training provenance enables transparent reproducibility and downstream experimentation. Theoretically, Sumi provides a basis for dissecting uniform diffusion’s unique traits as scaling advances, informing future paradigm shifts in generative modeling.

Conclusion

Sumi represents the first fully open, large-scale UDLM pretrained from scratch, performing competitively on knowledge, reasoning, and coding, but with clear gaps in commonsense. Its training recipe, rigorous benchmarking, and inference-time probes supply essential empirical grounding for further study. Future work will examine instruction tuning, error-targeted revision, and matched protocol comparisons to distinguish intrinsic properties of uniform diffusion from those entangled with data or architecture. As UDLMs scale and diversify, Sumi stands as a foundational artifact for both practical and theoretical advancement in LLM research.