Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought (2505.23766v1)

Abstract: Recent advances in multimodal LLMs (MLLMs) have demonstrated remarkable capabilities in vision-language tasks, yet they often struggle with vision-centric scenarios where precise visual focus is needed for accurate reasoning. In this paper, we introduce Argus to address these limitations with a new visual attention grounding mechanism. Our approach employs object-centric grounding as visual chain-of-thought signals, enabling more effective goal-conditioned visual attention during multimodal reasoning tasks. Evaluations on diverse benchmarks demonstrate that Argus excels in both multimodal reasoning tasks and referring object grounding tasks. Extensive analysis further validates various design choices of Argus, and reveals the effectiveness of explicit language-guided visual region-of-interest engagement in MLLMs, highlighting the importance of advancing multimodal intelligence from a visual-centric perspective. Project page: https://yunzeman.github.io/argus/

Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought

The paper "Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought" presents a novel approach to enhancing multimodal LLMs (MLLMs) for vision-language tasks, specifically addressing the limitations in vision-centric scenarios. The authors introduce Argus, which incorporates a grounding-driven visual attention mechanism to facilitate a more effective visual reasoning process.

Technical Overview

Argus leverages a mixture-of-vision-experts (MoVE), combining the capabilities of three visual foundation models—CLIP, ConvNeXt, and EVA-02—to acquire comprehensive visual embeddings. These embeddings are used to re-engage visual attention through explicit region-of-interest (RoI) sampling, effectively acting as visual chain-of-thought signals that align closely with the cognitive principles of goal-directed and stimulus-driven attention mechanisms.

Key Contributions

1. Vision-Centric Attention Mechanism: Argus introduces a new module for goal-directed visual tokenization, using language-conditioned instructions to locate relevant RoIs within images. This explicit top-down approach facilitates more accurate multimodal reasoning by ensuring the model focuses on pertinent visual information, supported by bounding box predictions as intermediate thought processes.
2. Superior Multimodal Reasoning: The framework is validated across multiple challenging benchmarks, demonstrating improvements in multimodal reasoning tasks (such as Visual Question Answering) and grounding tasks. It outperforms comparable models in vision-centric benchmarks like V-Star and CV-Bench. The Argus model structure efficiently utilizes visual CoT signals by both re-encoding and re-sampling RoIs, providing flexibility between computational efficiency and detail preservation.
3. Evaluation and Results: Argus achieves leading performance among publicly available models with similar parameter sizes and training scales, highlighting its effectiveness in combining goal-specific reasoning with precise visual engagement. The results indicate that explicit visual attention engagement mechanisms and grounded CoT signals offer substantial performance improvements over implicit self-attention methods.

Implications and Future Directions

The paper suggests significant potential in integrating explicit, visual CoT signals within the design space of MLLMs, paving the way for vision-centric multimodal intelligence solutions. Practically, this approach benefits applications requiring higher degrees of visual attention, such as autonomous vehicles, medical imaging interpretation, and interactive AI systems like personal assistants.

Future research could explore expansion into diverse domains and tasks, integrating broader datasets to increase robustness. The paper also hints at applying similar mechanisms to other multimodal architectures, exploring the scalability and application of grounded attentional mechanisms in emergent AI fields. Additionally, research may delve into optimizing computational efficiency while retaining the benefits of detailed visual processing.