Unclear effectiveness of ARM-to-MDM adaptation for building strong foundation models

Determine whether fine-tuning autoregressive language models into masked diffusion models, as in the adaptation approach of Gong et al. (2024), can produce a foundation language model whose comprehensive evaluation performance is comparable to that of strong large language models.

Background

The paper surveys prior efforts that adapt autoregressive models to diffusion-style formulations for text. While some improvements have been reported on select metrics, a broad, comprehensive evaluation comparable to strong LLMs has not been established.

The authors note that despite observed gains, it remains unresolved whether adaptation via fine-tuning from autoregressive models can yield a competitive foundation model under wide-ranging benchmarks.