- The paper introduces AMIGO, a diagnostic benchmark designed to evaluate long-horizon, evidence-based interactions in vision-language models.
- It employs controlled attribute-based multi-image galleries to scrutinize protocol compliance, candidate identification accuracy, and interaction efficiency.
- Empirical analysis reveals key trade-offs in model behavior, highlighting challenges like premature guessing and protocol violations under confusable distractor conditions.
Agentic Multi-Image Grounding Oracle (AMIGO): Benchmarking Long-Horizon Vision-Language Interaction
Problem Motivation and Benchmark Design
AMIGO introduces a demanding and diagnostic benchmark for evaluating agentic behaviors in vision-LLMs (VLMs), with a focus on multi-image, multi-turn, hidden-target identification. Existing benchmarks for VLMs are predominantly limited to single-image and single-turn paradigms, lacking the capacity to scrutinize agentic attributes such as long-horizon planning, active information gathering, constraint tracking, or robustness under protocol noise. In contrast, AMIGO operationalizes a setting where a model must recover a hidden target image from a gallery of visually similar candidates through a sequence of attribute-focused binary questions under strict interaction protocols, with explicit penalties for protocol violations.
The core task, "Guess My Preferred Dress," leverages galleries of highly similar dresses, emphasizing fine-grained attribute discrimination (e.g., construction details, seam placement, drape) while deliberately blocking trivial cues such as color, pattern, or length. Each turn requires the model to formulate a single binary question; answers are Yes/No/Unsure, with "Skip" signaling protocol non-compliance. Importantly, the benchmark allows injection of controlled oracle imperfections to stress-test models' verification and consistency behaviors.
Dataset and Pipeline Construction
The AMIGO benchmark is underpinned by a large-scale, curated dataset of 4,880 unique dress images from Target's online catalog. Construction of galleries with calibrated difficulty leverages attribute-based similarity retrieval, controlled by a tunable threshold. Distractors are selected to maximize fine-grained attribute similarity with the target. The attribute labeling pipeline employs an ensemble of open-source VLMs (Qwen3-VL-235B-FP8, Intern-S1, and GLM-4.5V) for per-image annotation. Attribute normalization leverages multiple LLMs to ensure lexical coherence, while quality is further improved via template paraphrasing, multi-resolution augmentation, and targeted manual audits.
Difficulty is modulated along two axes: (1) attribute similarity threshold (higher yields more confusable distractors), and (2) gallery size (range: 6 to 40+). Multiple similarity thresholds are sampled, producing galleries that rival human-level discriminative capacity with respect to visual subtlety.
Protocol and Evaluation Methodology
The protocol enforces a constrained interaction action space, penalizing questions about forbidden attributes, redundancies (e.g., attribute enumeration between turns), index references, and premature guessing. Enforcement is automated with a LLM-based violation detector, fully decoupled from the benchmarked VLM, ensuring consistent protocol adjudication.
AMIGO introduces rigorous evaluation metrics:
- Identification accuracy: Distinguishes verified success (correct answer with feasible candidate set of size one immediately before guessing) from "random-guess correct" (model guesses before sufficient evidence is accrued).
- Interaction efficiency: Measures number of dialogue turns to solution, partitioned by episode outcome.
- Protocol compliance: Quantified via Skip (violation) rate and premature output rate.
- Robustness to noise: Tested via injected oracle inconsistencies; verified accuracy is monitored for evidence of conservative or corrective querying.
A state-tracking verification module explicitly maintains the evolving candidate set, auditing evidence consistency and identifying contradiction episodes.
Empirical Results and Failure Analysis
Multiple recent open-source VLMs are evaluated, notably Qwen3-VL-235B-Instruct-FP8, Qwen3.5-397B-A17B-FP8, and Step3-VL-10B. Key findings:
- Verified accuracy varies with gallery difficulty. Qwen3.5-397B-A17B-FP8 attains the highest verified accuracy at lower similarity thresholds; Step3-VL-10B outperforms at the highest threshold (T=0.8) despite its modest parameter count. This suggests an architectural or capacity-related difference in handling strongly confusable distractors vs. larger gallery sizes.
- Random-guess accuracy: Qwen3-VL-235B-Instruct-FP8 shows elevated random-guess correct rates, indicating a proclivity to guess before evidence sufficiency.
- Interaction efficiency: Step3-VL-10B exhibits longer episode length distributions (when verified correct), consistent with a more conservative, evidence-accumulating strategy. In contrast, Qwen3-VL-235B-Instruct-FP8 is biased toward early commitment.
- Protocol compliance: Qwen3.5-397B-A17B-FP8, while excelling at accuracy, frequently incurs Skip penalties due to protocol violations, especially for attribute re-enumeration and querying forbidden topics. Step3-VL-10B is most prone to premature querying, particularly in larger galleries.
- Failure cases: Detailed error analysis reveals several prominent agentic failure modes:
- Protocol myopia (e.g., repeated enumeration of restricted features, querying before receiving the upload-completion signal),
- Constraint drift resulting in logically vacuous or redundant questioning, and
- Insufficient cross-turn evidence integration, producing dialogue stalls and budget exhaustion without valid final guesses.
Implications for Vision-Language Agent Research
AMIGO establishes a new axis of multimodal evaluation targeting long-horizon, agentic behaviors, which are increasingly relevant for consequential real-world vision-language deployments. By isolating protocol adherence, evidence-driven elimination, interaction efficiency, and robustness to feedback noise, AMIGO exposes capabilities and limitations that are invisible in one-shot multimodal QA or static multi-turn dialogue benchmarks.
Practically, agentic models tuned on AMIGO-style diagnostic traces can be guided to avoid premature commitments, strategize for data-efficient active information seeking, and robustly recover from feedback inconsistencies. The milestone of deterministic, protocol-saturating identification under fine-grained ambiguity remains unsolved for existing VLMs, motivating research on belief state modeling, trajectory-level reinforcement learning, and inductive biases aligned with conservative hypothesis elimination.
The diagnostic nature and the temporally extended structure of AMIGO episodes render them directly applicable for offline RL from multimodal feedback, hierarchical policy optimization, preference learning, and protocol-robust reward design.
Conclusion
AMIGO marks a substantive step toward realistic agentic evaluation for vision-LLMs, offering a controlled, interpretable, and scalable benchmark for multi-image, long-horizon, evidence-driven interaction. The benchmark framework systematically elucidates agentic failure modes, efficiency-protocol trade-offs, and the impact of model capacity on both compliance and verifiable identification. Future work is warranted on adaptive strategy optimization, robust state tracking, and transfer to other hidden-target identification domains beyond fashion, catalyzing progress toward trustworthy, interpretable, and high-reliability vision-language agents.