PaliGemma 2: A Family of Versatile VLMs for Transfer (2412.03555v1)

Abstract: PaliGemma 2 is an upgrade of the PaliGemma open Vision-LLM (VLM) based on the Gemma 2 family of LLMs. We combine the SigLIP-So400m vision encoder that was also used by PaliGemma with the whole range of Gemma 2 models, from the 2B one all the way up to the 27B model. We train these models at three resolutions (224px, 448px, and 896px) in multiple stages to equip them with broad knowledge for transfer via fine-tuning. The resulting family of base models covering different model sizes and resolutions allows us to investigate factors impacting transfer performance (such as learning rate) and to analyze the interplay between the type of task, model size, and resolution. We further increase the number and breadth of transfer tasks beyond the scope of PaliGemma including different OCR-related tasks such as table structure recognition, molecular structure recognition, music score recognition, as well as long fine-grained captioning and radiography report generation, on which PaliGemma 2 obtains state-of-the-art results.

• The paper introduces PaliGemma 2, a scalable family of vision-language models that significantly improves transfer learning through multi-stage fine-tuning.
• It integrates SigLIP-So400m encoders with Gemma 2 language models, enabling flexible architectures across varied input resolutions.
• Extensive evaluations demonstrate state-of-the-art performance in tasks like OCR, medical imaging, and spatial reasoning, emphasizing robustness and versatility.

Overview of PaliGemma 2: A Family of Versatile Vision-LLMs for Transfer

The paper presents PaliGemma 2, a series of Vision-LLMs (VLMs) developed to enhance transfer learning through fine-tuning, building upon the foundational work of the PaliGemma model. Notably, PaliGemma 2 offers scalability in model sizes and input resolutions, showcasing versatility across a broad range of vision-language tasks through different configurations of model components.

Model Architecture and Training

PaliGemma 2 integrates the SigLIP-So400m vision encoder with the Gemma 2 series of LLMs, stretching from 2B to 27B parameters. This integration facilitates multi-scale learning with resolutions set at 224px², 448px², and 896px². The model's design accommodates scalability allowing users to fine-tune the models based on specific computational budgets and task requirements. The training framework involves a defined three-stage process, aligning with the architecture and methods of the preceding PaliGemma model. This process leverages a mixture of multimodal tasks to induce broad knowledge within base models, emphasizing fine-tuning for performance optimization. Such a systematic approach facilitates a controlled analysis of various factors influencing transfer performance, such as model size, resolution, and learning rate.

Performance and Evaluation

PaliGemma 2's architecture includes comprehensive testing across a spectrum of tasks beyond those previously considered in PaliGemma. The enhanced suite of tasks comprises optical character recognition (OCR), music score recognition, molecular structure identification, spatial reasoning, and radiographic report generation. These expansions reflect PaliGemma 2's superior transfer learning capabilities. The model demonstrates state-of-the-art results in many of these tasks, particularly in OCR and medical imaging, underscoring its practical applicability and robustness. The flexible interface and strong performance across varied applications suggest substantial advancements in the model's multimodal understanding and its suitability for domain-specific tasks.

Model Scaling and Transfer Dynamics

The paper thoroughly investigates the impacts of model size and input resolution on transfer task performance. Findings reveal that resolution improvements dramatically benefit certain tasks like document understanding, characterized by intricate visual details. Conversely, LLM scaling demonstrates more significant gains in tasks requiring complex reasoning and multilingual capabilities. These insights contribute to a nuanced understanding of the interaction between computational resources and task requirements, further informing the development of efficient and effective VLMs.

Implications and Future Directions

PaliGemma 2 sets a precedent for versatility and performance in VLM applications, making it a valuable asset for researchers and practitioners working within varied domains of vision-language integration. The release of PaliGemma 2 as open-weight models bolsters its accessibility, facilitating further exploration into fine-tuning dynamics and model scaling. The paper's insights notably prompt further inquiry into optimizing VLM deployments in resource-constrained environments, with potential developments in refining inference processes for efficiency in CPU-bound applications.

Conclusion

This work extends the frontier of VLM scalability and performance, offering a comprehensive framework that balances model complexity with transfer task proficiency. By addressing a diverse set of tasks and demonstrating detailed analyses of model performance dynamics, PaliGemma 2 reinforces its role as a pivotal tool in vision-LLM research and implementation. The model's adaptability and performance affirm the importance of structured transfer learning in advancing the capabilities of VLMs across multiple application domains.