Identify Dream model characteristics causing diminished RL policy performance

Determine which characteristics of the Dream-7B-Instruct diffusion language model contribute to the diminished performance of reinforcement learning-trained unmasking policies for masked diffusion sampling.

Background

The paper proposes training lightweight transformer-based unmasking policies via reinforcement learning to control which tokens masked diffusion LLMs unmask at each step. On LLaDA-8B-Instruct, these RL-trained policies match or surpass heuristic samplers (e.g., Fast-dLLM) under several settings. However, when policies are trained on Dream-7B-Instruct, the authors observe a small performance gap compared to results on LLaDA, indicating model-dependent differences that affect RL policy effectiveness.

The authors explicitly note that understanding the source of this performance gap is important. They hypothesize that characteristics such as Dream’s initialization from an autoregressive model may play a role, but do not resolve which specific factors are responsible. This motivates a targeted investigation into Dream’s model properties and training history that impact RL-based sampling policies.