Olympus: A Universal Task Router for Computer Vision Tasks

Abstract: We introduce Olympus, a new approach that transforms Multimodal LLMs (MLLMs) into a unified framework capable of handling a wide array of computer vision tasks. Utilizing a controller MLLM, Olympus delegates over 20 specialized tasks across images, videos, and 3D objects to dedicated modules. This instruction-based routing enables complex workflows through chained actions without the need for training heavy generative models. Olympus easily integrates with existing MLLMs, expanding their capabilities with comparable performance. Experimental results demonstrate that Olympus achieves an average routing accuracy of 94.75% across 20 tasks and precision of 91.82% in chained action scenarios, showcasing its effectiveness as a universal task router that can solve a diverse range of computer vision tasks. Project page: http://yuanze-lin.me/Olympus_page/

• The paper introduces Olympus, a framework that leverages multimodal LLMs as controllers to delegate diverse vision tasks using instruction-based routing.
• It achieves an impressive average routing accuracy of 94.75% across 20 tasks, with 91.82% precision in complex chain-of-action scenarios.
• Olympus features scalable task delegation with innovative routing tokens and specialized datasets (OlympusInstruct and OlympusBench) to enhance adaptability.

Overview of "Olympus: A Universal Task Router for Computer Vision Tasks"

The paper introduces Olympus, an advanced framework that strategically employs Multimodal LLMs (MLLMs) as task routers across a spectrum of computer vision tasks, encompassing images, videos, and emerging 3D content. Olympus addresses a significant challenge in the field: the integration and execution of diverse vision-language tasks without the need for expansive generative model training. This is achieved by leveraging a controller MLLM, which delegates tasks to specialized modules through an innovative instruction-based routing system.

Key Contributions

1. Task Delegation through MLLMs: Olympus utilizes MLLMs not only for understanding contextual tasks but also for interacting with external models for specialized tasks. This dual-functionality optimizes both the comprehension of vision-language tasks and the external execution of vision-specific functions.
2. High Routing Accuracy: The paper reports a notable routing accuracy average of 94.75% across 20 defined tasks, with a precision of 91.82% in complex chain-of-action scenarios, highlighting the efficacy and sophistication of the Olympus framework in task management.
3. Scalable Framework and Instruction Dataset Development: Olympus is designed to scale efficiently with the increasing diversity and complexity of vision tasks. The introduction of OlympusInstruct and OlympusBench—comprehensive datasets tailored for training and evaluation across multiple vision tasks—underpins this scalability.
4. Innovation in Task-Specific Routing Tokens: The development of dedicated routing tokens enables Olympus to effectively map user instructions to the appropriate task models, thus streamlining the task execution process and enhancing interoperability within the framework.

Implications and Future Perspectives

Olympus offers significant implications for both theoretical and practical advancements in AI. From a theoretical standpoint, it demonstrates a scalable approach to multitask integration using modular architecture, suggesting pathways for future research into more adaptable and efficient MLLM applications. Practically, it provides a model for future implementations that require versatile and precise task routing across multimodal domains. One area for further exploration could be the refinement of task delegation strategies to incorporate increasingly sophisticated specialist models as they become available, thus maintaining Olympus's cutting-edge performance.

Conclusion

The introduction of Olympus marks a meaningful stride towards solving the integration challenges posed by current all-in-one models, which often sacrifice task-specific performance for versatility. By transforming MLLMs into universal task routers, the Olympus framework not only achieves high accuracy and precision across diverse domains but also sets a foundation for future work aiming to create robust and scalable AI frameworks capable of handling the intricacies of complex vision-language tasks.

This work underscores the potential for modular approaches to redefine the landscape of AI applications in computer vision, paving the way for increasingly sophisticated and responsive systems. As the field evolves, further attention to the dynamic interplay between model scalability, efficiency, and performance will be essential to maintaining and expanding the capacities of applied AI frameworks like Olympus.