InsightSee: Multi-agent Visual Reasoning

• InsightSee is a multi-agent architectural augmentation that enhances visual question answering and scene understanding by decomposing tasks into specialized agent stages.
• It leverages a two-stage description and adversarial reasoning to resolve occlusions and ambiguities in complex visual scenarios.
• Experiments demonstrate that InsightSee improves accuracy across multiple spatial dimensions compared to leading single-agent VLMs without additional model fine-tuning.

InsightSee is a multi-agent architectural augmentation applied as a plug-and-play wrapper around vision-LLMs (VLMs), designed to advance visual question answering and scene understanding in complex visual scenarios. By orchestrating specialized agents—a description agent, adversarial reasoning agents, and a decision agent—InsightSee systematically decomposes visual-question tasks, yielding measurable improvements over state-of-the-art single-agent VLMs without requiring any additional model fine-tuning (Zhang et al., 2024).

1. Framework Overview and Motivation

InsightSee addresses persistent challenges in visual question answering, particularly the recognition of occluded or ambiguously presented elements in complex scenes. While large-scale VLMs such as GPT-4V exhibit strong multimodal capabilities, their performance often degrades on tasks requiring nuanced, multi-stage reasoning or resolution of ambiguity. InsightSee reimagines the visual understanding process as a staged, multi-agent deliberation. This approach enhances interpretability and accuracy by exploiting redundancy, critique, and consensus among agents, as opposed to the single-pass, monolithic inference of standard models (Zhang et al., 2024).

2. System Architecture and Agent Specialization

The core InsightSee pipeline decomposes visual-question answering ($Q$ about image $I$) into four sequential stages:

1. Description Agent ($f_{\text{desc}}$): Constructs a two-part scene representation $D = \{d_{\mathrm{global}}, d_{\mathrm{detail}}\}$ using chain-of-thought prompting. The "global" component describes main objects and scene context. The "detail" component focuses on regions possibly relevant to $Q$, explicitly noting attributes such as shape, color, occlusion, and coordinates.
2. Reasoning Agents ($f_{\text{reason}_1}$, $f_{\text{reason}_2}$): Independently ingest $(D, I, Q)$ to produce initial hypotheses $r_1^0$, $r_2^0$. These agents then iterate through an adversarial exchange: at each round $t$, each agent critiques and refines its reasoning based on the peer’s last hypothesis, converging to final outputs $r_1^T$, $r_2^T$ after $T$ rounds or upon consensus.
3. Decision Agent ($f_{\text{dec}}$): Aggregates $(D, r_1^T, r_2^T)$, arbitrates disagreements via implicit majority vote, and emits the final prediction $\hat{y}$.

The process is realized through specialized prompts to GPT-4V, orchestrating the flow without altering the underlying VLM parameters. Message-passing between reasoning agents is structurally equivalent to multi-head cross-attention, aligning latent reasoning representations across multiple deliberative rounds (Zhang et al., 2024).

3. Formal Pipeline and Information Flow

Let $I$ denote the input image, $Q$ the associated question, and $D=f_{\text{desc}}(I, Q)$. Initialize hypotheses $r_1^0 = f_{\text{reason}_1\_\text{init}}(D, I, Q)$ and $r_2^0 = f_{\text{reason}_2\_\text{init}}(D, I, Q)$. The adversarial update at each round $t=0, ..., T-1$ proceeds as: $$r_1^{t+1} = f_{\text{reason}_1}(D, I, Q, r_1^t, r_2^t)$$

$$r_2^{t+1} = f_{\text{reason}_2}(D, I, Q, r_2^t, r_1^t)$$

The final answer is given by: $\hat{y} = f_{\text{dec}}(D, r_1^T, r_2^T)$ or, in compact notation,

$$\hat{y} = f_{\text{dec}}\Big(f_{\text{desc}}(I, Q), f_{\text{reason}_1}^T(I, Q), f_{\text{reason}_2}^T(I, Q)\Big)$$

No gradient-based training or regularization is introduced. All operations are instantiated via prompt engineering and majority-vote decoding over multiple (typically three) independent inference runs to account for GPT-4V’s output non-determinism (Zhang et al., 2024).

4. Experimental Protocols and Evaluation

Experiments utilize a modified SEED-Bench dataset, focusing on nine spatial dimensions: scene understanding, instance identity, instance attributes, instance location, counting, spatial relations, interactions, visual reasoning, and text recognition. The evaluation metric is task-average accuracy, first computed per dimension then aggregated to avoid bias from question-count imbalance.

InsightSee is benchmarked against seven leading single-agent VLMs (InstructBLIP-Vicuna, InstructBLIP, Qwen-VL, LLaVA-1.5, ShareGPT4V-13B, InternVL-Chat-V1.2-Plus, and GPT-4V).

Model	Avg. Accuracy	SOTA in # Dimensions
InstructBLIP-Vicuna	52.14%	-
InstructBLIP	49.26%	-
Qwen-VL	59.60%	-
LLaVA-1.5	66.89%	-
ShareGPT4V-13B	67.40%	-
InternVL-Chat-V1.2-Plus	72.44%	-
GPT-4V	67.53%	-
InsightSee-GPT-4V	74.47%	6/9

InsightSee surpasses all baselines in overall mean accuracy, and achieves the highest score in six out of nine dimensions, including marked gains in scene understanding (+1.9%), instance attributes (+9.1%), instance counting (+9.2%), and visual reasoning (+9.7%) over the next best approach (Zhang et al., 2024).

5. Contributions of Multi-agent Design

Several elements underlie InsightSee’s performance improvements:

• Two-stage Description: The separation of global scene summarization from detailed region-of-interest ensures richer context and finer grounding in scene elements.
• Adversarial Reasoning: Iterative debate between reasoning agents functions analogously to an internal consistency check, compelling each agent to identify and rectify weaknesses in peer-generated arguments.
• Detail-Context Synergy: The explicit integration of both coarse and fine-grained features enables the system to better disambiguate partially occluded objects or overlapping scene elements.
• Structured Conflict Resolution: The decision agent’s majority-vote arbitration mitigates the risk of outlier or misaligned reasoning chains dominating the output.

This framework operates entirely at the prompt orchestration level, allowing it to function as a drop-in augmentation without training overhead (Zhang et al., 2024).

6. Limitations and Future Directions

InsightSee, while advancing visual understanding on complex spatial tasks, inherits the deterministic and prompt-sensitivity limitations of underlying VLMs such as GPT-4V. No explicit mechanism for gradient-based optimization or end-to-end differentiability is provided. The majority-vote approach to inference imposes computational overhead. A plausible implication is that future work may explore tighter coupling between agent outputs, more sophisticated consensus models, or selective invocation of multiple agent reasoning to balance efficiency and performance. The plug-and-play nature of the framework, coupled with architecture-agnosticism, suggests extensibility to other VLMs and possibly broader multimodal reasoning contexts (Zhang et al., 2024).