Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models (2404.12387v1)

Abstract: We introduce Reka Core, Flash, and Edge, a series of powerful multimodal LLMs trained from scratch by Reka. Reka models are able to process and reason with text, images, video, and audio inputs. This technical report discusses details of training some of these models and provides comprehensive evaluation results. We show that Reka Edge and Reka Flash are not only state-of-the-art but also outperform many much larger models, delivering outsized values for their respective compute class. Meanwhile, our most capable and largest model, Reka Core, approaches the best frontier models on both automatic evaluations and blind human evaluations. On image question answering benchmarks (e.g. MMMU, VQAv2), Core performs competitively to GPT4-V. Meanwhile, on multimodal chat, Core ranks as the second most preferred model under a blind third-party human evaluation setup, outperforming other models such as Claude 3 Opus. On text benchmarks, Core not only performs competitively to other frontier models on a set of well-established benchmarks (e.g. MMLU, GSM8K) but also outperforms GPT4-0613 on human evaluation. On video question answering (Perception-Test), Core outperforms Gemini Ultra. Models are shipped in production at http://chat.reka.ai . A showcase of non cherry picked qualitative examples can also be found at http://showcase.reka.ai .

• The paper presents novel modular transformer architectures using SwiGLU, Grouped Query Attention, and rotary embeddings to enhance multimodal processing.
• The models achieve state-of-the-art results on language benchmarks, visual question answering, and specialized tasks such as medical reasoning.
• The research leverages efficient distributed training on Nvidia H100 GPUs to scale performance across diverse datasets and long-context inputs.

Exploring the Capabilities of Reka Models: Core, Flash, and Edge in Multimodal Language Tasks

Introduction

Reka introduces its series of models: Reka Core, Flash, and Edge, demonstrating significant advancements in handling multimodal inputs such as text, images, video, and audio. These models vary in size and computational needs, with Reka Edge and Flash achieving state-of-the-art results within their compute classes and Reka Core proving robust in comparisons with other frontier models on comprehensive benchmarks across modalities.

Model Overview and Training Details

Reka's architecture is built on a modular encoder-decoder transformer framework optimizing for multimodal inputs. Details include:

• Training Data: A rich dataset comprised of both public and proprietary sources, focusing on a diverse set of languages and modal data including images and video.
• Infrastructure: Primarily utilizing Nvidia H100 GPUs, Reka models used distributed training strategies with peak resources reaching substantial computing power.
• Model Architecture: Reka employs techniques such as SwiGLU, Grouped Query Attention, Rotary positional embeddings, and RMSNorm, with a backbone similar to the PaLM architecture yet tailored for multimodal inputs. Context lengths vary, with Reka Flash and Core supporting extensive contexts up to 128K tokens for enhanced performance on long-context tasks.

Performance and Evaluations

Language and Multimodal Benchmarking

The evaluation of Reka's models includes:

• Language Tasks: Reka models showcase competitive performance across standard benchmarks like MMLU and GSM8K, with particularly strong showings in code generation tested through HumanEval.
• Multimodal Performance: Reka Core excels in visual question answering, ranking competitively against models such as GPT-4V and surpassing Claude 3 in multimodal chat setups based on third-party blind human evaluations.
• Specialized Domain Applications: In detailed model comparisons, Reka models exhibit strong performances in specific domains such as medical reasoning, showing capabilities on par with or exceeding specialized models like Med-PaLM-2 and GPT-4 on various medical benchmarks.

Detailed Comparative Analysis

Reka consistently positions itself as a strong contender in the landscape of AI models:

• Reka Edge stands out within its compute class, outperforming comparable models in tasks varying from standard language understanding to complex reasoning and multilingual understanding.
• Reka Flash, despite its middling size (21B parameters), often matches or surpasses much larger models across a variety of benchmarks, highlighting its efficiency and the effectiveness of its training and architectural choices.

Future Outlook and Theoretical Implications

The ongoing development and enhancements of Reka models suggest future improvements and potential new applications, especially in domains requiring robust multimodal understanding and interaction. Theoretical explorations could further explore the efficiency of training protocols, the scalability of multimodal models, and the transferability of learned representations across diverse data types and tasks.

Conclusion

Reka's suite of models demonstrates substantial advancements in the field of AI, particularly in handling and reasoning across multiple modal domains. By leveraging innovative architectural features and a diverse training corpus, Reka manages to set new standards within specific compute classes and shows promise in competing at the frontier level of multimodal LLMs. The ongoing enhancements and expansions of Reka models are expected to further solidify their standing in the AI research community and expand their applicability across a broader spectrum of real-world applications.