Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents

Abstract: Vision-LLMs (VLMs) frequently misread values, hallucinate details, and confuse overlapping elements in charts. Current approaches rely solely on pixel interpretation, creating a Pixel-Only Bottleneck: agents treat interactive charts as static images, losing access to the structured specification that encodes exact values. We introduce Introspective and Interactive Visual Grounding (IVG), a framework that combines (1) spec-grounded introspection, which queries the underlying specification for deterministic evidence, with (2) view-grounded interaction, which manipulates the view to resolve visual ambiguity. To enable evaluation without VLM bias, we present iPlotBench, a benchmark of 500 interactive Plotly figures with 6,706 binary questions and ground-truth specifications. Experiments show that introspection improves data reconstruction fidelity, while the combination with interaction achieves the highest QA accuracy (0.81), with +6.7 % gains on overlapping geometries. We further demonstrate IVG in deployed agents that explore data autonomously and collaborate with human users in real time.

• The paper introduces the IVG framework that integrates spec-grounded introspection with view-grounded interaction for precise chart reasoning.
• It overcomes the pixel-only bottleneck by enabling deterministic verification of chart values and geometries via structured specification access.
• Experimental results show improved QA accuracy and reduced tool inefficiencies, demonstrating enhanced performance especially in ambiguous and overlapping chart scenarios.

Introspective and Interactive Visual Grounding: Eliminating the Pixel-Only Bottleneck for Chart Reasoning Agents

Motivation and Problem Formulation

The rapid proliferation of VLM-enabled agents in data analysis has exposed persistent perceptual failure modes: value misreading, hallucination, and confusion in the presence of overlapping or dense chart elements. The central diagnosis is the "Pixel-Only Bottleneck," where agents reason exclusively via pixel data, discarding the structured chart specification that encodes precise semantics. This bottleneck fundamentally limits the determinism and auditability of agents' visual reasoning, especially in analytic tasks where minimal ambiguity is essential. The absence of structured manipulation—zooming, toggling, selection—further limits agent agency compared to human analysts.

IVG Framework: Spec-Grounded Introspection and View-Grounded Interaction

The IVG (Introspective and Interactive Visual Grounding) framework is introduced to overcome these two bottlenecks by empowering agents with structured access to both the chart specification and the interaction state. IVG comprises two orthogonal capabilities:

1. Spec-Grounded Introspection: Direct querying of the complete chart specification (e.g., Plotly JSON), enabling agents to deterministically verify values, geometries, and mappings without reliance on visual estimation. This removes noise and hallucinations endemic to pixel-level approaches.
2. View-Grounded Interaction: Programmatic manipulation of chart views (zoom, pan, legend toggling, region selection) to derive focal context. By generating explicit interaction histories, agents can localize attention, resolve visual ambiguities (e.g., occlusion), and filter the specification to relevant substructures.

This dual mechanism transforms agents from passive pixel consumers to active explorers of their visualization outputs.

Figure 1: Overview of IVG—contrasting VLM-based pixel reasoning with agents that manipulate and introspect the underlying visualization state for deterministic evidence.

The agent workflow (Figure 2) tightly interleaves these mechanisms: agents alternate between interaction to constrain the problem space and introspection to extract deterministic answers from the specification. This iteration continues until a grounded, verifiable answer is assembled.

Figure 2: Workflow of IVG on a concrete example, illustrating iterative alternation between interaction and introspection to gather evidence.

iPlotBench: Deterministic Benchmark for Interactive Chart Grounding

To quantify the effectiveness of IVG and decouple perceptual failures from reasoning errors, the authors introduce iPlotBench, an interactive Plotly-based benchmark. iPlotBench features 500 procedurally generated figures across line, dot-line, bar, and pie charts, each associated with 6,706 binary questions based on 15 precise templates (aggregation, comparison, topology). Critically, each figure is paired with its ground-truth specification, enabling structural, white-box evaluation.

The evaluation protocol comprises two sequential tasks:

• Task 1: Chart recreation from static reference, evaluated using Semantic Structural Similarity metrics that utilize trace type, data fidelity (via Chamfer distance under Hungarian matching), text role similarity, and style similarity.
• Task 2: Binary visual question answering conducted on the agent-created chart, enabling isolation of reasoning over agent-generated figures.

Experimental Results and Empirical Insights

Systematic ablations reveal the relative and complementary strengths of introspection and interaction, evaluated with foundation models such as Claude Haiku 4.5 and a range of Qwen-VL models:

• Spec-Grounded Introspection yields strong gains in semantic data reconstruction, boosting $S_{Data}$ from 0.88 to 0.90. This modality excels for bar and pie charts, where visual ambiguity is minimal and specification access suffices.
• View-Grounded Interaction enhances question answering for visually ambiguous scenarios, with QA accuracy improving by +6.7% on line/dot-line charts exhibiting overlapping traces.
• The combination ("Full") achieves maximum overall QA accuracy—0.81, establishing the strongest empirical finding of the paper. This synergy is pronounced for topology questions (e.g., intersection detection), where focal context for narrowing the search space is invaluable.

These findings were robust to parameter choices in matching criteria ($\lambda$ sensitivity), and per-table delta analyses confirmed that only sufficiently large models (≥20B parameters) reliably capitalize on tool orchestration.

Analysis of Agent Behavior

Interaction tool usage analysis demonstrates that agents learn to allocate cognition efficiently: when both modalities are available, unnecessary interactions are minimized (e.g., −71% tool use in Full vs. +Inter on easy charts). For complex, ambiguous geometries, agents exploit both mechanisms, mimicking the process of human analytical focus followed by precise measurement. The trade-off is non-trivial: excessive exploration may be detrimental if introspection alone already localizes the relevant structure.

IVG-Enabled Workflows

IVG’s utility extends beyond controlled Q/A:

• Real-Time Collaboration: IVG captures user interactions—such as pointing, brushing, or toggling—and uses them as machine-interpretable focal context for the agent to ground responses, facilitating natural, ambiguity-free analytical dialogs.

  Figure 3: IVG bridges ambiguous user gestures and precise agent reasoning by capturing focal context.

• Autonomous Exploration: Agents autonomously navigate datasets, repeatedly hypothesizing, verifying, and gathering evidence through IVG cycles; reports are generated with explicit grounding in manipulation history.

  Figure 4: Autonomous exploration—agents continuously validate, revise, and focus their claims using structured interaction and introspection.

• ML Solution Search: IVG enables evidence-grounded search and comparison in ML workflows, where agent decisions (e.g., branch pruning, improvement selection) are based on interaction-located metric retrieval—resolving issues with occlusion and ambiguity in crowded performance plots.

  Figure 5: In ML solution search, IVG allows the agent to distinguish and compare overlapping runs precisely by zooming and querying the exact metric values at focal points.

User Study

A within-subject study (N=12) confirms IVG’s practical benefit: 11 out of 12 practitioners preferred IVG-enabled agents over vision-only baselines for exploration tasks, citing superiority in value grounding and evidence citation. The agents are rated especially highly (4.9/5) on identifying precisely the region the user interacted with, and (4.5/5) on citing numerically exact values supporting their claims.

Implications and Future Directions

The IVG toolkit provides a reproducible model for deterministic, evidence-grounded visual reasoning. It generalizes to any charting backend supporting decoupled rendering specifications and structured interactions, such as Vega-Lite and ECharts, with engineering adaptation. The results indicate that for analytic charts, fully deterministic, auditable pipelines are now feasible, in contrast to prior pixel-level or VLM-judged systems—this directly influences both scientific reproducibility and operational decision making.

Theoretically, IVG demonstrates the necessity of multi-level cognitive control (structured introspection, attention localization via interaction) for agents operating in structured visual settings. Practically, this approach sets design requirements for tool-augmented agent frameworks to support deterministic analytics and debuggable reasoning. It also suggests a delineation in agent deployment: pixel-only vision is insufficient for tasks requiring high precision, but the integration of MCP-style tool protocols with sufficient model scale enables tangible improvements.

Moving forward, future systems will likely extend IVG-style protocols toward more open-ended reasoning benchmarks, generalized to richer analytical tasks (multi-stage queries, temporal reasoning) and alternative visualization ecosystems. The public release of iPlotBench also provides a strong testbed for such developments.

Conclusion

The IVG framework effectively removes the pixel-only bottleneck in agent-based chart analysis by instituting deterministic, structured access to both the chart specification and interaction history. Empirical and user-centric evaluations indicate that this dual-access mechanism produces robust, auditable, and context-sensitive analytic agents. IVG redefines how autonomous agents approach data-centric tasks by grounding every analytic step in explicit, inspectable evidence streams, setting a new standard for the development, evaluation, and deployment of chart reasoning agents.