InsightSee: Multi-Agent VLM Reasoning

• InsightSee is a multi-agent VLM framework that decomposes complex visual queries for robust spatial reasoning.
• It applies coordinated description, adversarial reasoning, and decision fusion to enhance interpretability and accuracy.
• Empirical results show InsightSee outperforms baselines, achieving 74.47% average accuracy in spatial tasks.

InsightSee is a multi-agent framework designed to enhance the interpretive capabilities of vision-LLMs (VLMs) in complex visual understanding scenarios. The paradigm systematically decomposes high-level visual reasoning into coordinated subprocesses involving description, adversarial reasoning, and decision fusion. By leveraging a four-agent architecture and prompting strategies applied to frozen, large-scale VLMs (notably GPT-4V), InsightSee advances state-of-the-art performance in spatial vision-language tasks, particularly where occlusion and ambiguity challenge conventional models (Zhang et al., 2024).

1. System Architecture and Agent Roles

InsightSee operationalizes multi-agent vision-language reasoning through a modular pipeline. The architecture encapsulates four agent entities, each instantiated by GPT-4V, orchestrating sequential and adversarial processing stages:

• Description Agent performs chain-of-thought decompositions over the input image $I$ and a natural-language query $q$, producing structured textual observations $D = \{d_{\mathrm{global}}, d_{\mathrm{detail}}\}$. The agent explicitly separates global scene context from fine-grained regional attributes (e.g., shape, color, size).
• Reasoning Agents (A₁, A₂) independently generate interpretation hypotheses $h_1$, $h_2$ from $(D, I, q)$. They engage in an adversarial loop for $T$ rounds: each agent critiques the other's previous hypothesis, refining its own via iterative message passing and challenge-response updates.
• Decision Agent synthesizes the hypotheses $\{h_1, h_2\}$, performing a majority-vote procedure or explicit conflict resolution (with an auxiliary neutral analysis $v$), outputting the final answer $\hat{y}$.

The system processes each image-query pair as follows:

Input: I, q D ← DescriptionAgent(I, q) h1, h2 ← Reason1.init(D, I, q), Reason2.init(D, I, q) for t in 1…T do h1_new ← Reason1.update(h1, h2) h2_new ← Reason2.update(h2, h1) h1, h2 ← h1_new, h2_new end ŷ ← DecisionAgent.vote(h1, h2) return ŷ

No custom feature extraction modules or fine-tuning procedures are implemented in the base system; all agent-level processing leverages chain-of-thought prompting on a frozen pre-trained VLM.

2. Mathematical Formulation

InsightSee abstracts the multi-agent reasoning process as compositions over a base VLM function $E_{\mathrm{vlm}}(\cdot)$:

• Description Generation: $$D = f_{\mathrm{desc}}(I, q) \coloneqq E_{\mathrm{vlm}}([\mathrm{Prompt}_{\mathrm{desc}}]; I, q)$$
• Reasoning (per agent $i \in \{1,2\}$):
  • Initial hypothesis: $h_i^{(0)} = f_{\mathrm{reason}}^0(D, I, q)$
  • For $t = 1\dots T$:

  $$h_i^{(t)} = f_{\mathrm{reason}}^t(h_i^{(t-1)}, h_j^{(t-1)})$$, where $j \neq i$

• Final Decision: $\hat{y} = \mathrm{Vote}(\{h_1^{(T)}, h_2^{(T)}\})$

No formal equations for feature extraction, neural scoring, or agent-specific loss definitions are provided in the original data.

3. Training Protocol and Agent Initialization

InsightSee forgoes fine-tuning, joint optimization, or agent-specific training. Each agent is instantiated via zero-shot or few-shot prompting atop GPT-4V. The pipeline is driven solely by natural-language instructions formulated to elicit structured reasoning and adversarial critique. There are no custom objectives (for example, cross-entropy, contrastive loss) or supervised learning procedures applied (Zhang et al., 2024). This suggests scalability across base VLMs, provided robust prompt engineering.

4. Experimental Setup and Benchmark Comparison

Empirical evaluation is conducted on a modified SEED-Bench dataset, emphasizing spatial reasoning in nine dimensions:

1. Scene Understanding (SU)
2. Instance Identity (IIden)
3. Instance Attributes (IA)
4. Instance Location (IL)
5. Instance Counting (ICount)
6. Spatial Relation (SR)
7. Instance Interaction (IInter)
8. Visual Reasoning (ViR)
9. Text Recognition (TR)

Temporal and action-oriented tasks are excluded to focus on spatial interpretability. The protocol involves 1,000 randomly sampled questions, each run three times; answers are correct if the model matches ground truth in at least two of three executions.

Major baselines for comparison include InstructBLIP-Vicuna, InstructBLIP, Qwen-VL, LLaVA-1.5, ShareGPT-4V-13B, InternVL-Chat-V1.2-Plus, GPT-4V.

Model	Avg. Accuracy	# Dimensions Led
InstructBLIP-Vicuna	52.14%	0
InstructBLIP	49.26%	0
Qwen-VL	59.60%	0
LLaVA-1.5	66.89%	0
ShareGPT-4V-13B	67.40%	0
InternVL-Chat-V1.2-Plus	72.44%	3
GPT-4V	67.53%	0
InsightSee-GPT4V	74.47%	6

InsightSee achieves 74.47% average accuracy, leading in 6 out of 9 reasoning dimensions. Notably, Text Recognition (TR) performance matches that of GPT-4V, reflecting a prioritization of spatial reasoning over OCR capabilities.

5. Qualitative Analysis and Ablation-like Observations

Formal ablation studies (e.g., agent removal, loss-based analysis) are absent. However, empirical observations highlight several phenomena:

• On tasks involving hidden or context-dependent elements (e.g., occluded purses), InsightSee’s decomposition and adversarial critique mechanisms enable superior disambiguation versus single-agent VLM baselines.
• The separation of descriptive and reasoning responsibilities allows the framework to capture both context and detail, with adversarial rounds amplifying hypothesis refinement.
• The lack of improvement in the TR dimension suggests prompts and architectures designed for spatial reasoning are suboptimal for text-heavy vision tasks. A plausible implication is the need for agent specialization.

6. Design Significance and Prospective Extensions

InsightSee’s multi-agent pipeline enables systematic decomposition of complex visual queries into sequential and adversarial reasoning subroutines, strengthening interpretability and robustness on occluded or ambiguous tasks. The architecture preserves zero-shot simplicity via prompt-based deployments atop pre-trained large-scale VLMs.

Potential future directions identified include:

• Integration of specialized OCR or text-recognition modules to elevate text-heavy performance metrics.
• Formal training or fine-tuning of inter-agent prompt embeddings to reduce manual prompt engineering dependence.
• Extension to $N > 2$ reasoning agents or introduction of role-specialized agents (e.g., commonsense modules).
• Implementation of confidence-weighted voting schemes, leveraging agent self-reported uncertainty for soft decision fusion.
• Systematic ablation on agent count, number of reasoning rounds $T$, and prompt formulations to refine empirical understanding of architecture-performance tradeoffs.

These directions indicate modularity and extensibility as salient properties of the InsightSee framework (Zhang et al., 2024).