Stable injection of residual context into diffusion LLM input embeddings

Develop a method to inject soft-token residual context into the input embeddings of diffusion large language models that preserves the discrete masking scheme and avoids unstable recursive dependencies during training.

Background

Soft tokens can represent mixtures of vocabulary embeddings and thereby carry fine-grained contextual information prior to commitment. However, directly adding such soft tokens to the input embeddings of diffusion LLMs can distort the masked denoising setup and create unstable recursive dependencies.

The authors state this challenge explicitly as an open question and then propose RCD, which constructs residual vectors using the model’s embedding codebooks and injects them with entropy-based weighting to stabilize training and inference.