MIRAGE: A Micro-Interaction Relational Architecture for Grounded Exploration in Multi-Figure Artworks

Abstract: Appreciating multi-figure paintings requires understanding how characters relate through subtle cues like gaze alignment, gesture, and spatial arrangement. We present MIRAGE, an evidence-centric framework designed to scaffold the exploration of these "micro-interactions" in multi-figure artworks. While such cues are essential for deep narrative appreciation, they are often distributed across complex scenes and difficult for viewers to systematically identify. Existing vision-LLMs (VLMs) frequently fail to provide reliable assistance, offering ungrounded interpretations that lack traceable visual evidence. MIRAGE addresses this by constructing a structured intermediate representation capturing identities, pose cues, and gaze hypotheses. However, the challenge extends beyond extracting these cues to coordinating them during interpretation. Without an explicit mechanism to organize and reconcile relational evidence, models often collapse multiple interaction hypotheses into a single unstable or weakly grounded narrative, even when low-level signals are available. This representation allows users to verify how high-level interpretations are anchored in low-level visual facts. By separating spatial grounding from narrative generation, MIRAGE enables users to inspect and reason about figure-to-figure relationships through a verifiable evidence layer. We evaluate MIRAGE against painting-only VLM baselines using a blind assessment protocol. Results show that MIRAGE significantly improves identity consistency, reduces relational hallucinations, and increases the coverage of subtle interactions. These findings suggest that structured grounding can serve as a critical interaction control layer, providing the necessary scaffolding for a more reliable, transparent, and human-led understanding of complex visual narratives.

• The paper introduces a decoupled, evidence-centric architecture that stabilizes identities and quantifies relational cues for enhanced artwork interpretation.
• It employs YOLOv8-face based character detection and geometric constraints, achieving marked improvements in identity consistency (92%) and relational fidelity (94%) over baselines.
• The system’s transparent, structured outputs facilitate human–AI collaboration and set new benchmarks for explainable art analysis.

MIRAGE: Structured Grounding for Micro-Interaction Reasoning in Multi-Figure Artworks

Motivation and Problem Context

Interpreting multi-figure artworks inherently demands not only recognizing individual agents but also discerning the subtle relational cues—gaze alignment, gesture, posture, spatial arrangement—that underwrite complex narrative and social interpretations. Conventional vision-LLMs (VLMs) possess some facility in holistic description but consistently fail in structured reasoning over micro-interactions: they hallucinate, collapse identities, and conflate relations in densely populated or compositionally ambiguous scenes. MIRAGE ("A Micro-Interaction Relational Architecture for Grounded Exploration in Multi-Figure Artworks" (2604.23788)) directly addresses these deficits by introducing a decoupled, evidence-centric architecture. The key design features are the stabilization of individual identities, the construction of quantifiable pairwise relations, and the preservation of transparency through intermediate representations, thus scaffolding higher-level interpretation upon verifiable low-level visual cues.

System Architecture: Decoupled Grounding and Interpretation

MIRAGE's architecture decouples spatial grounding from semantic narrative generation. The pipeline begins with face-first character detection using YOLOv8-face, prioritizing stable identity anchors over potentially unreliable full-body detections typical in painted scenes. Body association is then achieved via geometric constraints and confidence priority, ensuring character regions are properly indexed even under substantial occlusion or stylization. Object anchoring is pursued synergistically: objects of interactional salience (e.g., tombs, inscriptions, personal artifacts) are detected and filtered, not through category generalization, but via semantic centrality determined with VLM support.

Geometric and semantic enrichment follows. Character-centric information is formalized into profiles containing static (ID, region) and dynamic (pose, gaze, action cues) attributes. Notably, pose and gaze estimation are conditionally engaged, only where supporting evidence exists to obviate propagation of unreliable geometry.

Figure 1: MIRAGE's adaptive pose and gaze estimation; when body evidence is incomplete, inference is disabled, preventing spurious geometric interpretation.

Pairwise relational grounding then exhaustively constructs relation records between characters. Metrics such as inter-character distance, bounding-box IoU, and directionality vectors are numerically encoded and, crucially, rendered accessible—enabling explicit inspection, not just black-box inference.

Figure 2: MIRAGE's explicit relational grounding process, making pairwise structure transparent via spatial, gaze, and proximity cues.

These heterogeneous signals are consolidated into a unified Markdown "grounding document"—the architectural interface between visual extraction and VLM-driven interpretation. The choice of Markdown leverages empirical findings on LLM robustness to structured macro-formatting, reducing spurious formatting-induced performance degradation (2604.23788).

Notably, the geometry–VLM integration preserves ambiguity rather than collapsing modality conflicts. Divergences between pose estimators and VLM interpretation (e.g., standing vs. kneeling) are annotated, not forcibly resolved, and preserved for downstream abductive reasoning.

Empirical Evaluation: Evidence-Based Superiority

Quantitative evaluation targets hallucination, identity drift, grounding adherence, and relational fidelity. On a controlled dataset spanning canonical, lesser-known, and synthetic works, MIRAGE achieves significant absolute improvement in all assessed dimensions over a strong GPT-5.4 baseline:

Metric	Baseline (GPT-5.4)	MIRAGE
Identity	0.72	0.92
Interaction	0.81	0.94
Directionality	0.83	0.92
Grounding	0.71	0.88

These results empirically demonstrate marked increases in identity consistency, relational coverage, and grounding fidelity.

Qualitative analysis reveals MIRAGE's ability to preserve local micro-interactions, discriminate overlapping identities, and robustly represent object-mediated and indeterminate relations—capabilities not found in end-to-end VLMs.

Figure 3: Comparative output—MIRAGE maintains identity consistency, directional awareness, and evidence-linked rationale in complex scenes, outperforming baseline VLMs.

Figure 4: Grounding in "Et in Arcadia Ego": MIRAGE visualizes identities, gaze, touch, and shared object anchors, allowing explicit tracing of relational structure.

Figure 5: In "The Birth of Venus", MIRAGE articulates overlapping agents and emergent global interaction topology centered on the principal figure.

Interpretability and Human-AI Collaboration

A user study revealed that structured grounding not only improved interpretive confidence, micro-interaction discovery, and evidence verification but fostered collaborative sensemaking. The system's transparency—surfacing ambiguity, exposing conflicts, and anchoring outputs in inspectable evidence—advanced interpretive agency. Cognitive workload remained low, with frustration minimal, confirming that the additional structural information is beneficial rather than burdensome. Critically, users shifted from narrative-heavy impressionism to structured, evidence-driven reasoning, explicitly leveraging geometric and relational cues.

Theoretical and Practical Implications

MIRAGE's explicit separation between perceptual grounding and narrative generation aligns with the current trajectory in multimodal reasoning research advocating for intermediate structured representations. Unlike opaque, end-to-end VLMs susceptible to hallucination and drift, MIRAGE makes its evidentiary substrate transparent and queryable, which is essential for high-stakes analysis in subjective, culturally significant domains such as art history.

From a practical perspective, MIRAGE opens new avenues for computational museology: by grounding interaction evidence, it enables explainable assistance, collaborative exploration, and robust verification not possible in existing VLM-driven art interpretation pipelines. Additionally, the architecture is amenable to edge-deployment, as the grounding stage is lightweight and can be precomputed, facilitating in-gallery or mobile applications.

On a theoretical level, the paradigm suggests a generalizable framework for structured reasoning in other domains characterized by ambiguous, relationally complex, or highly stylized visual input: social scene analysis, behavioral video annotation, and even mixed-reality human–AI interaction settings.

Limitations and Future Directions

Key technical ceilings persist: the foundational detection and pose estimation leverage models trained primarily on naturalistic data, resulting in inevitable domain gap issues for stylized or non-Western artworks. 2D geometric reasoning is limited when narrative implication is not a simple function of spatial proximity. The reliance on probabilistic VLMs for semantic disambiguation cannot fully suppress persistent hallucination risk.

Future extensions should incorporate domain-adaptive fine-tuning, monocular depth estimation, and neuro-symbolic reasoning enhancements. Furthermore, the methodological groundwork MIRAGE establishes invites exploration into multi-perspective and character-driven narrative generation rigorously grounded in evidence, and the systematic study of interpretive bias that might arise from strong structural scaffolding.

Conclusion

MIRAGE presents a rigorously structured, evidence-centric architecture for micro-interaction interpretation in multi-character artworks, demonstrably outperforming image-only VLM prompting in both numeric and qualitative dimensions. The architecture's explicit intermediate representation, ambiguity preservation, and transparent pipeline not only elevate model reliability but reframe the interpretive process as a genuinely collaborative exercise between human intuition and algorithmic reasoning. These contributions signal a significant methodological advance in explainable AI for art and open the possibility for broad dissemination in educational and museological contexts. The generality of MIRAGE’s paradigm further holds implications for any field requiring high-fidelity relational grounding under visual complexity.