Autoregressive language models can learn to infill text by simply moving a span of text from the middle of a document to its end.
Training models with a large fraction of transformed data doesn't harm the original left-to-right generative capability, and is useful, simple, and efficient.