VACP: Visual Analytics Context Protocol

Abstract: The rise of AI agents introduces a fundamental shift in Visual Analytics (VA), in which agents act as a new user group. Current agentic approaches - based on computer vision and raw DOM access - fail to perform VA tasks accurately and efficiently. This paper introduces the Visual Analytics Context Protocol (VACP), a framework designed to make VA applications "agent-ready" that extends generic protocols by explicitly exposing application state, available interactions, and mechanisms for direct execution. To support our context protocol, we contribute a formal specification of AI agent requirements and knowledge representations in VA interfaces. We instantiate VACP as a library compatible with major visualization grammars and web frameworks, enabling augmentation of existing systems and the development of new ones. Our evaluation across representative VA tasks demonstrates that VACP-enabled agents achieve higher success rates in interface interpretation and execution compared to current agentic approaches, while reducing token consumption and latency. VACP closes the gap between human-centric VA interfaces and machine perceivability, ensuring agents can reliably act as collaborative users in VA systems.

• The paper presents VACP, a formal protocol that bridges semantic gaps in visual analytics by exposing a hierarchical state and formal interaction intents.
• It employs a multi-layer context representation, enabling agents to traverse pixel, DOM, declarative, and semantic layers for robust and low-latency task execution.
• The protocol significantly reduces token use and improves task completion rates, while supporting automated testing and mixed human-agent collaboration despite retrofitting challenges.

VACP: A Semantically Grounded Context Protocol for Agent-Compatible Visual Analytics

Motivation

Contemporary Visual Analytics (VA) systems are grounded in a paradigm that privileges human visual and cognitive perception, optimizing for rendered pixels and direct manipulation through mouse and keyboard input. The shift toward agentic AI—autonomous agents capable of operating and reasoning directly over user interfaces—fundamentally disrupts these assumptions. Current agentic approaches, primarily leveraging computer vision on rendered pixels or parsing raw DOM, fail to meet the precision, semantics, and interaction requirements of non-human actors in VA. The "interface mismatch" between pixel- and DOM-centric access and logical analytic operations imposes a severe bottleneck, precluding both reliable task completion and robust, low-latency actuation by AI agents.

Core Contributions

The Visual Analytics Context Protocol (VACP) is proposed as a formal, application-agnostic protocol designed to close the semantic and operational gap between complex VA environments and AI agents. The protocol is instantiated as an open-source TypeScript framework targeting modern visualization stacks and is accompanied by comprehensive evaluation across a standardized task benchmark. VACP's core differentiators include:

• Explicit, Semantically Enriched State Representation: VACP exposes the VA application's semantic state as a hierarchical, temporally-stable graph, capturing both structural and functional information about interface entities, data mappings, and coordinated views.
• Formalized Interaction Space: The protocol defines, registers, and surfaces all available interaction intents (e.g., filtering, brushing, selection), their parameter schemas, and contextual constraints, allowing agents to introspect, reason, and act without reliance on fragile UI-level event simulation.
• Direct Execution Gateway: VACP provides an API that mediates agent intent, performing multi-stage validation and delivering structured feedback, thereby supporting agent autonomy and enabling robust recovery from invalid or obsolete interactions.
• Optimized, Token-Efficient Data Access: To circumvent LLM context window constraints, VACP exposes data and schema representations on demand, allowing selective access and incremental state diffs, avoiding state bloat and unnecessary serialization.
• Generalizable Design and Integration: VACP is designed for seamless adaptation to both imperative and declarative visualization paradigms, including Vega-Lite and vgplot-driven systems, and supports retrofitting of legacy VA interfaces.

  Figure 1: Overview of VACP’s knowledge representation layers, showing agent accessibility and protocol functions across pixel, DOM, declarative, and semantic abstraction.

Technical Architecture

Multi-Level Context Representation

VACP formalizes VA interface understanding into four distinct layers:

• L1—Pixel-based: The rendered visual output, accessible via screenshots.
• L2—DOM-based: The logical webpage tree, often insufficient for dense, highly interactive SVG/canvas-based VA.
• L3—Declarative Grammar: Encodings specified in formats like Vega-Lite or vgplot, essential for mapping data attributes to interface elements and extracting interaction affordances.
• L4—Semantic/Intent Layer: Application logic, developer rationale, and intended semantics annotated within the protocol.

VACP enables agents to traverse and reason across these layers, fusing visual, structural, and semantic primitives.

Figure 2: Example VACP app-state representation as a capability graph, encoding semantic, structural, and interaction-accessible relationships.

Formal Interaction Modeling

Interactions are abstracted as first-class semantic entities (rather than UI-level events), with explicit parameterization, context sensitivity, and dynamic availability. Interaction endpoints expose not only intent and parameters but also pre- and post-conditions, semantic targets, and provenance metadata for full-cycle analytic execution.

Execution Gateway and Feedback Loop

Agent action via VACP is mediated through a validated, deterministic execution gateway. The protocol enforces state consistency and provides interaction outcomes as structured agent-readable results, including robust error handling for invalid operations. This enables agents to programmatically close the perception-action-feedback loop and maintain analytic provenance.

Figure 3: Evaluation pipeline illustrating task execution workflow, agentic loop, and scenario-based ablation for context and access granularity.

Evaluation

Benchmarks and Methodology

A task benchmark—spanning canonical VA tasks (locate, identify, compare), multiple data and interaction modalities, and both declarative and imperative systems—was curated and expert-validated for baseline solvability and representativeness.

The comparison includes:

• State-of-the-art multimodal models: GPT-5.2, Claude 4.5 Sonnet, Gemini 3 Pro Preview.
• Four scenarios: (S1) UI+DOM access, (S2) UI+DOM+VACP state, (S3) VACP only, (S4) VACP+UI+DOM.

All configurations leverage standardized tool interfaces and a reproducible, agentic loop (ReAct framework with OpenCode instrumentation).

Quantitative Results

Completion Rates: Non-VACP approaches (S1, S2) achieve low success on VA benchmarks—maximum 51%, median 28–31%. VACP-driven agents (S3, S4) consistently reach near-perfect task completion (see Figure 4).

Token Consumption & Latency: VACP agents demonstrate a significant reduction in both tokens (~2000 vs. >30,000 for baseline in GPT-5.2) and time per task. The protocol's details-on-demand architecture and semantic filtering are central to this reduction (see Figure 5).

Layer Strata Utilization: Successful task completion is correlated with elevated semantic-layer interaction; failure modes in traditional pipelines are marked by reliance on brittle pragmatic/syntactic interactions.

Figure 4: Success rates per agent and interaction context. VACP scenarios yield near-perfect completion; traditional agents fail on the majority of tasks.

Figure 5: Median token use and execution time per scenario. VACP reduces both cost and latency compared to UI or DOM-based baselines.

Qualitative Findings

• Verification Loops: Agents with VACP access prefer semantically grounded reasoning, occasionally corroborating via pixel-based checks, paralleling human QA strategies.
• Failure Recovery: VACP exposes granular state and interaction errors, enabling self-correction; baseline agents are susceptible to compound errors due to interface opacity.
• Hybrid Access: Mixed modality (VACP+visual) supports rich sensemaking but comes at the cost of higher token usage with minimal incremental gains in completion rates.

  Figure 6: Stepwise agent workflow on a canonical VA task, illustrating state inspection, interaction actuation via VACP, and semantic-visual verification.

Implications, Limitations, and Future Prospects

Practical Implications

• Agentic Collaboration: VACP establishes a pragmatic substrate for mixed-initiative VA, supporting concurrent human-agent analytic flows.
• Automated Testing: Protocol-level exposure of system state enables systematic benchmarking, regression, and provenance validation at scale.
• Efficiency and Model-Agnosticism: Token and latency reduction enable practical deployment with both frontier and resource-constrained (on-device) LLMs.

Theoretical Extensions

• Model Integration: VACP’s design is compatible with evolving MCP standards and can be instrumented for broader tool-based agent workflows.
• Multimodal Reasoning: Experiments indicate that exclusive semantic access is insufficient for perceptual or attention-based inference; future protocols may fuse semantic and controlled visual abstraction to optimize agent performance under both precision and compression constraints.

Limitations

• Developer Overhead: There is non-negligible complexity in retrofitting or instrumenting bespoke VA systems with semantic mappings.
• Non-Standard or Dynamic UIs: Highly irregular or stateful interfaces may necessitate additional protocol extensions to sustain dynamic semantic consistency.
• Interface Clutter/Omission: Current VACP design intentionally omits certain visual signals (e.g., perceptual grouping, micro-clusters) easily apprehended by humans but difficult to codify semantically.

Conclusion

VACP represents a foundational step in making VA systems natively accessible to AI agents, addressing critical mismatch in abstraction granularity, semantic context, and interaction mediation. The protocol's impact is empirically validated—yielding marked improvements in precision, efficiency, and reliability. The generalization capacity of VACP suggests applicability far beyond canonical VA, with implications for geographic information systems, 3D modeling, and mixed-agent collaborative environments. Standardized, semantic context protocols such as VACP will be indispensable in the emerging agent-driven analytic ecosystem.