InsightSee Framework: Multi-Agent Visual Inference

• InsightSee is a multi-agent inference framework that decomposes visual interpretation into specialized agents for handling occlusion and ambiguity.
• Its adversarial reasoning agents iteratively refine hypotheses through debate, significantly enhancing spatial reasoning and hypothesis diversification.
• The framework achieves state-of-the-art accuracy on diverse spatial dimensions while maintaining interpretable and robust decision outputs.

InsightSee is a multi-agent inference framework designed to enhance the visual interpretation abilities of general-purpose vision-LLMs (VLMs), with a particular focus on resolving the challenges posed by occluded, cluttered, and ambiguous visual environments. Instantiated on top of off-the-shelf models such as GPT-4V, InsightSee decomposes the visual understanding process into discrete, specialized agents: a description agent, two adversarial reasoning agents, and a decision agent. Through iterative debate and consolidation mechanisms, the framework refines scene understanding, yielding both interpretable and state-of-the-art performance across diverse spatial reasoning tasks (Zhang et al., 2024).

1. Definition, Motivation, and Objectives

InsightSee is formulated as a modular, agent-based, zero-shot inference scheme. The core objective is threefold: (a) to explicitly decompose visual interpretation into semantically grounded sub-tasks, (b) to enable iterative adversarial hypothesis refinement concerning ambiguous or hidden elements via reasoning-agent debate, and (c) to aggregate agent outputs into a consolidated, accurate decision suitable for downstream tasks.

This approach responds to the empirical limitations of monolithic VLMs, which, despite their strong scene-level encoding, struggle with fine-grained discernment—especially when objects are occluded or the context is ambiguous. By structuring the analysis pipeline into collaborative yet distinct agents, InsightSee seeks improved hypothesis diversification and robustness in answer consolidation.

2. Agent-Based Architectural Design

The InsightSee architecture comprises four principal agents, each encapsulating a distinct functionality and interacting via defined interfaces. The workflow is as follows:

1. Input Provisioning: The system receives an image $I \in \mathcal{X}$ and a textual query $Q \in \mathcal{T}$.
2. Description Agent ($A_0$): Produces a hierarchical, global-to-detail natural-language description $D = A_0(I, Q)$.
3. Reasoning Agents ($A_1$, $A_2$): Initialized with $(I, Q, D)$, each agent forms an initial answer hypothesis ($R_1^0$, $R_2^0$). They engage in $T$ rounds of adversarial debate, with each iteration incorporating the opponent's last hypothesis.
4. Decision Agent ($A_3$): Aggregates the final outputs ($R_1^T$, $R_2^T$) as well as optional summary information from $D$, emitting a consolidated answer $Y$ via majority rule or explicit conflict resolution.

A table summarizing agent roles is provided below.

Agent	Role	Input(s)
Description ($A_0$)	Scene + detail summarization	$I, Q$
Reasoning ($A_1$, $A_2$)	Hypothesis generation / critique	$I, Q, D$, opponent prev.
Decision ($A_3$)	Output consolidation	$R_1^T, R_2^T, D$

This architecture is strictly modular and non-trainable—each agent corresponds to a prompted instance of a pretrained GPT-4V model; no additional parameters are learned during deployment.

3. Mathematical Formalism and Algorithmic Workflow

The InsightSee process is governed by an iterative, agent-interactive protocol underpinned by the following sequence:

1. Description: $D = A_0(I, Q)$.
2. Initial Reasoning:
   • $R_1^0 = A_1(I, Q, D)$
   • $R_2^0 = A_2(I, Q, D)$
3. Adversarial Reasoning (for $t = 1, ..., T$):
   • $R_1^t = A_1(I, Q, D, R_2^{t-1})$
   • $R_2^t = A_2(I, Q, D, R_1^{t-1})$
   • Convergence condition: $R_1^t = R_2^t$ or $t = T_{\text{max}}$

Each adversarial step can be viewed as maximizing an implicit agent-specific scoring function:

$$R_i^t = \arg\max_{r} f_i(r \mid I, Q, D, R_j^{t-1})$$

where $f_i$ is implemented via GPT-4V’s chain-of-thought prompt design.

1. Decision: $Y = A_3(R_1^T, R_2^T, D)$. If both reasoning agents agree, $A_3$ returns this value. If not, $A_3$ invokes a conflict-resolution prompt, weighing the content and confidence of each proposition, possibly supplemented by $D$.

4. Core Agent Pseudocode

The mechanics of each agent can be schematically represented as follows (where “prompt” specifies template-driven prompt engineering):

Description Agent A₀:

def DescriptionAgent(I, Q): prompt = "First, give a global overview of the scene. Then, focus on key regions: describe shape, color, size, context…" response_D = GPT4V(I, Q, prompt) return response_D

Reasoning Agent Aᵢ ($i \in \{1,2\}$):

def ReasoningAgent_i(I, Q, D, prev_opponent=None): if prev_opponent is None: prompt = "Given description D, analyze attributes and context to hypothesize answer to Q." else: prompt = "Given description D and opponent’s last view prev_opponent, critique and refine your hypothesis." response_R = GPT4V(I, Q, D, prompt) return response_R

Adversarial Loop:

R1 = ReasoningAgent_1(I, Q, D) R2 = ReasoningAgent_2(I, Q, D) for t in range(1, T_max+1): R1_new = ReasoningAgent_1(I, Q, D, R2) R2_new = ReasoningAgent_2(I, Q, D, R1) if R1_new == R2_new: R1, R2 = R1_new, R2_new break R1, R2 = R1_new, R2_new

Decision Agent A₃:

def DecisionAgent(R1, R2, D): if R1 == R2: return R1 else: prompt = "Two reasoning agents disagree: R1 vs. R2. Based on D and both arguments, which is more plausible?" return GPT4V(D, R1, R2, prompt)

5. Inference Paradigm, Datasets, and Evaluation Protocols

InsightSee operates strictly in zero-shot or few-shot inference mode; no gradient-based fine-tuning or parameter update occurs post-deployment. Instead, functional optimization is performed exclusively at the prompt engineering stage, selecting and refining chain-of-thought and adversarial prompts for each agent.

Datasets: The experimental regime employs a spatial subset of SEED-Bench, evaluating across nine dimensions: scene understanding (SU), instance identity (IIden), instance attributes (IA), instance location (IL), instance counting (ICount), spatial relation (SR), interaction (IInter), visual reasoning (ViR), and text recognition (TR). Each dimension is assessed via 1,000 multiple-choice questions repeated across three trials, with the per-item correct answer defined by majority consensus ($\geq 2$ of 3 runs matching ground truth).

Baselines: Comparative analysis is performed against InstructBLIP-Vicuna, InstructBLIP, Qwen-VL, LLaVA-1.5, ShareGPT4V-13B, InternVL-Chat-V1.2-Plus, and base GPT-4V.

Metric: Accuracy is tabulated per dimension and as an overall average, accounting for possible disparity in sample counts.

Model	SU	IIden	IA	IL	ICount	SR	IInter	ViR	TR	Avg
GPT-4V	77.5	73.9	70.6	61.8	56.8	56.9	74.2	78.5	57.6	67.5
InternVL-Chat V1.2-Plus	80.2	80.0	77.8	71.3	72.3	63.3	77.3	79.8	50.0	72.4
InsightSee-GPT4V	82.1	80.0	79.3	70.7	68.6	63.6	80.6	87.7	57.6	74.5

InsightSee achieves the highest average accuracy (74.47%) and leads on 6 out of 9 spatial dimensions. Notably, gains are pronounced for instance attributes (+8.7%), instance counting (+11.8%), and visual reasoning (+9.2%) relative to the GPT-4V baseline (Zhang et al., 2024).

6. Strengths, Limitations, and Prospective Directions

Strengths

• Retains GPT-4V’s pretrained capabilities, including robust zero-shot generality and detailed vision–language embedding.
• Structures inference into interpretable, modular sub-tasks, enhancing transparency and division of labor.
• Adversarial reasoning effectively highlights and resolves conflicting cues, improving hypothesis discrimination.
• The decision agent cleanly encapsulates the multi-agent debate, yielding a singular, interpretable output.

Limitations

• No specialized fine-tuning for OCR/text recognition; thus, InsightSee’s performance on text-centric tasks matches GPT-4V baseline (~57.6%).
• Inference latency and computational cost are nontrivial: system requires 3–5 times the inference budget of direct GPT-4V querying, due to the multiplicity of agent invocations.
• Convergence criterion for adversarial reasoning is heuristic (fixed $T_{max}$ or exact match) and does not guarantee globally optimal synthesis in all cases.

Future Work

• Integration of dedicated OCR or scene-text modules to enhance text-heavy visual tasks.
• Introduction of lightweight, trainable policies governing the termination of adversarial rounds and dynamic confidence weighting.
• Extension to temporal or embodied scenarios (e.g., video understanding, robotics) through the addition of further agent types or persistent memory structures.

A plausible implication is that, by instantiating collaborative and adversarial modularity atop large VLMs, InsightSee achieves enhanced robustness and fine-grained reasoning—particularly under complex spatial environments—while preserving the generic strengths of foundational models (Zhang et al., 2024).