Training a fully native VLM entirely from scratch

Develop a training procedure to train a fully native vision-language model such as NEO entirely from scratch without initialization from any pretrained large language model.

Background

NEO is introduced as a native, unified vision-LLM that integrates image and text processing within a single architecture, leveraging a pre-Buffer and post-LLM design during training to align pixels and words efficiently.

In the limitations, the authors state that due to constraints in available text corpora and computational resources, they could not train a fully native model purely from scratch without starting from an existing LLM. Overcoming this limitation would validate truly de novo multimodal training in their framework and further reduce reliance on language-initialized models.