Multimodal Latent Language Modeling with Next-Token Diffusion (2412.08635v1)

Abstract: Multimodal generative models require a unified approach to handle both discrete data (e.g., text and code) and continuous data (e.g., image, audio, video). In this work, we propose Latent LLMing (LatentLM), which seamlessly integrates continuous and discrete data using causal Transformers. Specifically, we employ a variational autoencoder (VAE) to represent continuous data as latent vectors and introduce next-token diffusion for autoregressive generation of these vectors. Additionally, we develop $\sigma$-VAE to address the challenges of variance collapse, which is crucial for autoregressive modeling. Extensive experiments demonstrate the effectiveness of LatentLM across various modalities. In image generation, LatentLM surpasses Diffusion Transformers in both performance and scalability. When integrated into multimodal LLMs, LatentLM provides a general-purpose interface that unifies multimodal generation and understanding. Experimental results show that LatentLM achieves favorable performance compared to Transfusion and vector quantized models in the setting of scaling up training tokens. In text-to-speech synthesis, LatentLM outperforms the state-of-the-art VALL-E 2 model in speaker similarity and robustness, while requiring 10x fewer decoding steps. The results establish LatentLM as a highly effective and scalable approach to advance large multimodal models.

Insights on Multimodal Latent LLMing with Next-Token Diffusion

Multimodal generative models have continually posed a significant challenge in artificial intelligence due to the inherent need to effectively handle both discrete data—such as text and code—and continuous data, like images, audio, and videos. This paper presents a novel approach described as Latent LLMing (LatentLM), which introduces an effective solution harnessing the next-token diffusion mechanisms and sets a new precedent in the way continuous data is processed alongside discrete data by multimodal LLMs.

Latent LLMing Paradigm

Traditional approaches have struggled with efficiently integrating both types of data due to their distinct natures. Most existing models either adopt a piecemeal solution requiring independent processing modules or suffer from information loss due to tokenization bottlenecks associated with vector quantization. LatentLM innovatively addresses these challenges by employing a unified framework using variational autoencoders (VAE) to encode continuous data into latent space and adopting next-token diffusion for autoregressive generation. This transition is pivotal, allowing continuous data representations as latent vectors, which are smoothly integrated into the causal Transformer that already processes discrete tokens via next-token prediction.

Technical Contributions and Innovations

One core innovation is the introduction of σ-VAE, which resolves the prevalent issue of variance collapse seen in traditional VAEs when applied to autoregressive tasks. The latent LLM here ensures robust variance across latent dimensions, resulting in improved generative capabilities and the model's adaptability to exposure biases encountered during autoregressive training.

The paper reports substantial improvements in scalability and performance metrics across multiple modalities. LatentLM demonstrates superior performance over existing methods like Diffusion Transformers and vector-quantized models. Notably, in multimodal LLMing, LatentLM made significant strides, surpassing competing models including Transfusion and VALL-E 2, both in terms of scalability when training tokens are increased and in specific applications like text-to-speech synthesis.

Empirical Evaluations

The paper provides comprehensive experimental validation across several tasks:

• Image Generation: On datasets like ImageNet, LatentLM matches or exceeds the performance of state-of-the-art systems like DiT and U-ViT, with favorable scaling properties. Notably, it achieves improved FID scores, indicating enhanced image realism at high resolutions.
• Multimodal LLMs: When applied to tasks involving interleaved data and tasks like text-to-image generation, LatentLM outperformed existing methodologies in perplexity and FID scores. This advantage is pivotal for applications involving language understanding intertwined with vision data.
• Text-to-Speech Synthesis: Representing continuous data through σ-VAE, LatentLM achieves high speaker similarity and robustness in zero-shot speech synthesis, outperforming leading models like VALL-E 2 while requiring fewer inference steps.

Implications and Future Directions

The practical improvements introduced by LatentLM have significant implications for future developments in AI. Multimodal models that seamlessly integrate various data types are crucial for advancing AI applications in areas such as autonomous systems, interactive dialogue systems, and robust decision-making frameworks.

Future research directions could explore further extending the framework's applicability to more complex tasks such as video generation and world modeling. The ability to handle temporal dynamics through autoregressive strategies can lead to growth in domains like interactive simulations and dynamic environment modeling. Moreover, the potential applications in embodied AI and robotics could herald a new era of intelligent agents capable of sophisticated interactions in diverse contexts.

In summation, the paper presents a substantive step forward in multimodal modeling, setting the stage for expanded research and development across AI disciplines. LatentLM’s integration of next-token diffusion for unified data modeling provides a powerful template for building the next generation of artificial intelligence systems.