DashboardQA: Interactive Dashboard QA Benchmark

• DashboardQA is a benchmark for interactive dashboard reasoning that requires agents to ground GUI controls, plan actions, and synthesize evidence across coordinated views.
• It includes 112 Tableau dashboards and 405 QA pairs across five categories, emphasizing dynamic state tracking and cross-view analytics.
• The benchmark advances research by highlighting challenges in visual grounding, sequential planning, and memory retention in complex, interactive graphical interfaces.

Searching arXiv for DashboardQA and closely related interactive visualization QA benchmarks to ground the article in current papers. DashboardQA is a benchmark and evaluation environment for question answering on interactive dashboards by multimodal, GUI-capable agents. It was introduced to evaluate capabilities that are largely absent from static chart QA benchmarks: grounding dashboard controls, planning action sequences, tracking dashboard state across interactions, and synthesizing evidence across coordinated views. The benchmark comprises 112 interactive dashboards from Tableau Public and 405 question–answer pairs spanning five categories—multiple-choice, factoid, hypothetical, multi-dashboard, and conversational—and is designed around real-world analytical workflows such as filtering, tab navigation, cross-highlighting, and iterative drill-down (Kartha et al., 24 Aug 2025).

1. Problem setting and conceptual scope

DashboardQA was proposed in response to a mismatch between existing visualization QA benchmarks and actual analytical practice. Prior benchmarks such as FigureQA, DVQA, PlotQA, ChartQA, ChartQAPro, CharXiv, and Multi-ChartQA evaluate VLMs on static images; even when they require nontrivial reasoning, they remain non-interactive. DashboardQA instead treats dashboard reasoning as an interactive task in which the agent must operate over dynamic GUI state rather than a single chart snapshot (Kartha et al., 24 Aug 2025).

The benchmark’s core motivation is that dashboards are not merely collections of charts. They expose linked controls and coordinated views whose semantics emerge only through interaction. Real analytical work depends on changing filters and dropdowns, switching tabs, cross-highlighting linked views, inspecting details-on-demand, zooming, and iteratively refining a query. DashboardQA therefore targets the combined problem of GUI grounding, action execution, state tracking, cross-view synthesis, and numerical or logical reasoning.

The task is defined at the level of interactive examples. Each instance consists of an interactive dashboard $d_i$, a question $q_i$, and a ground-truth verifiable answer $a_i$, with the agent required to navigate and interact with the dashboard to generate the answer. This framing makes the benchmark closer to a GUI-agent evaluation problem than to conventional visual question answering. A plausible implication is that DashboardQA occupies an intermediate space between multimodal reasoning benchmarks and embodied desktop automation benchmarks.

2. Dataset construction and benchmark composition

DashboardQA is built from 112 high-quality dashboards selected from Tableau Public. Tableau was chosen because it supports rich interactivity and coordinated views, especially cross-filtering and cross-highlighting. The curation excluded single-view or decorative infographics and prioritized multi-view dashboards with meaningful text-chart integration, diverse topics such as health and renewable energy, and modern layouts (Kartha et al., 24 Aug 2025).

The benchmark contains 405 expert-authored question–answer pairs divided into five categories. Factoid questions target retrieval and localized interpretation. Multiple-choice questions probe comparison and directed selection. Hypothetical questions include trends and counterfactuals. Multi-dashboard questions require reasoning across more than one dashboard. Conversational questions introduce multi-turn, context-dependent interaction.

Question authoring followed a three-phase construction process. First, dashboards were curated. Second, annotators created seed QAs and then used GPT-4o, Gemini, and Claude to expand them with more diverse questions requiring multi-view interactions; humans then refined the results to remove incorrect, ambiguous, or trivial items. Third, a different annotator reviewed the output, with discrepancies resolved and unresolved items discarded. On a sample of 367 QA pairs, inter-annotator agreement was 74.93%, and 24.86% required revision. During annotation, estimation answers within a $< 0.5\%$ error margin from the reference were considered acceptable (Kartha et al., 24 Aug 2025).

The benchmark emphasizes interaction complexity rather than only visual diversity. Questions often require navigation across multiple views and multiple dashboard states before an answer can be extracted. The reported “views required to answer” distributions indicate that multi-step interaction is common, with multi-dashboard questions requiring the deepest transitions. This suggests that DashboardQA measures not only perception and reasoning but also trajectory management over stateful interfaces.

3. Interaction model and environment

Evaluation is conducted in an Ubuntu virtual machine using OSWorld’s desktop automation framework. Dashboards are opened in Chrome at $1920 \times 1080$ full-screen resolution. Agents receive either raw screenshots or screenshots augmented with an accessibility tree derived from ATSPI, which provides structured UI element information such as labels, roles, and bounding boxes (Kartha et al., 24 Aug 2025).

The action vocabulary follows a pyautogui-like interface. Agents can perform mouse moves, clicks, drags, scrolls, keyboard inputs, and hotkeys, and can also emit the meta-actions WAIT, FAIL, and DONE. Each episode is capped at 25 steps. Termination occurs when the agent signals DONE, signals FAIL, or reaches the step limit. The environment logs actions, intermediate reasoning, and final answers.

Although the benchmark paper does not explicitly formalize the environment as an MDP, it is naturally described as an episodic partially observable setting. At each timestep the agent observes the current GUI state through a screenshot, optionally combined with structured accessibility information, takes a GUI action, receives a new dashboard state, and updates its internal plan. This suggests that DashboardQA evaluates a compound competency: perception over mixed visual-structural observations, control over desktop actions, and sequential reasoning under partial observability.

The environment also foregrounds cross-view dependencies. Tableau dashboards support coordinated multi-view interactions natively, so actions in one view may update others through cross-filtering or cross-highlighting. Agents must therefore reason over causally linked visual state changes rather than independent panels.

4. Dashboard characteristics and task demands

DashboardQA contains substantial UI and visualization heterogeneity. Four primary UI control types are reported as prevalent: dropdown menus at 92.31% with count 266 and average 2.38 per dashboard, tabs at 23.08% with count 103 and average 0.92, radio buttons at 7.69% with count 26 and average 0.23, and range sliders at 7.69% with count 9 and average 0.08. These controls define the operational surface over which agents must ground and act (Kartha et al., 24 Aug 2025).

The dashboards span 13 visualization types. Reported examples include line charts at 48.51% with count 86, bar charts at 42.57% with count 79, maps at 20.79% with count 25, area charts at 16.83% with count 31, scatter plots at 10.89% with count 30, and pie charts at 6.93% with count 9. The average is 2.66 visualizations per dashboard. Because questions often require interactions that update multiple views, the benchmark couples chart interpretation with interface manipulation and cross-panel consistency checking.

A concise summary of the reported dashboard composition is useful.

Aspect	Reported values	Notes
Dashboards	112	From Tableau Public
QA pairs	405	Five categories
Avg. visualizations/dashboard	2.66	13 visualization types total
Dropdown prevalence	92.31%	Count 266; avg. 2.38/dashboard
Tabs prevalence	23.08%	Count 103; avg. 0.92/dashboard

The benchmark’s interaction demands arise from this combination of coordinated controls and multi-view layouts. Examples described in the benchmark include selecting multiple items in dropdowns, filtering views, navigating tabs, and synthesizing information across linked panels before computing ratios, trends, or outliers. A common misconception is that dashboard QA is just chart QA with more panels; DashboardQA rejects that interpretation by requiring state-changing actions as part of the answering process.

5. Evaluation protocol and empirical performance

DashboardQA uses an enhanced relaxed-accuracy protocol adapted to heterogeneous answer types. Numerical answers are accepted within a 5% tolerance. Year references require exact match. Textual answers are evaluated with ANLS. Results are reported overall and by category: Factoid, MCQ, Conversational, Hypothetical, and Multi-Dashboard (Kartha et al., 24 Aug 2025).

The systems evaluated include closed-source agents such as GPT-4o, O4-mini, Gemini Pro 2.5, and OpenAI Computer Use Agent; hybrid systems such as JEDI-3B w/GPT-4o and JEDI-7B w/GPT-4o; and open-source systems such as UI-TARS-2B and UI-TARS-1.5-7B. Two observation regimes are compared: Screenshot only, and Screenshot + A11y.

The principal quantitative result is the difficulty of the benchmark. Under Screenshot + A11y, Gemini-Pro-2.5 achieves 38.69% overall accuracy, with 40.03% on Factoid, 46.25% on MCQ, 51.20% on Conversational, 39.27% on Hypothetical, and 15.22% on Multi-Dashboard. GPT-4o reaches 22.94% overall in the same regime, and OpenAI CUA reaches 18.50%, with Multi-Dashboard at 0.00% for CUA. Under Screenshot-only evaluation, the best hybrid system, JEDI-7B w/GPT-4o, achieves 33.09% overall, while OpenAI CUA reaches 22.69%; Gemini-Pro-2.5 drops to 11.86%, and GPT-4o to 11.50% (Kartha et al., 24 Aug 2025).

These results can be organized compactly.

Regime	System	Overall accuracy
Screenshot + A11y	Gemini-Pro-2.5	38.69%
Screenshot + A11y	GPT-4o	22.94%
Screenshot + A11y	OpenAI CUA	18.50%
Screenshot only	JEDI-7B w/GPT-4o	33.09%
Screenshot only	OpenAI CUA	22.69%
Screenshot only	Gemini-Pro-2.5	11.86%

The per-category pattern is also significant. Multi-Dashboard is consistently the hardest category, with the best A11y-enabled score at only 15.22%. Conversational performance is relatively better in some settings, particularly for hybrid screenshot-only pipelines, but remains far from robust. This suggests that structured UI input helps grounding, but long-horizon planning and cross-dashboard reasoning remain major unresolved bottlenecks.

6. Failure modes and analytical significance

DashboardQA includes behavioral analysis beyond final accuracy. The benchmark reports step distributions, the frequency of hitting the 25-step limit, action-type frequencies, and error categories including grounding, planning, reasoning, and state retention. JEDI-7B w/GPT-4o hits the 25-step ceiling in 56 cases under Screenshot-only evaluation, while Gemini-Pro-2.5 with A11y hits the ceiling in 77 cases. Gemini’s action traces are diverse, including click at 2,920, moveTo at 3,796, and hotkey at 876, whereas JEDI traces are shorter and more click-centric, yet still often saturate the step budget (Kartha et al., 24 Aug 2025).

The benchmark identifies three dominant error families. First, there are grounding and planning failures: agents forget completed steps, loop back to previously handled controls, repeat dropdown selections, or revisit views unnecessarily. Second, there are state-retention failures: agents misremember values or rely on stale screenshots captured before the dashboard finishes updating. Third, there are reasoning failures: agents misread multi-line charts or make arithmetic and logical errors in multi-step chains.

Cross-view complexity is the structural cause behind many of these failures. Linked views require the agent to maintain coherence between an action, the resulting dashboard update, and the new evidential configuration across panels. Performance degrades as the number of views, controls, and interactions increases. The benchmark therefore isolates a capability gap that is partly perceptual, partly algorithmic, and partly architectural: current multimodal GUI agents do not yet reliably integrate grounding, planning, memory, and visual-mathematical reasoning under dynamic interface state.

This also clarifies the role of accessibility trees. Screenshot + A11y materially improves performance for some agents, especially Gemini-Pro-2.5, indicating that structured UI information can mitigate grounding failure. However, it does not solve compositional planning and dynamic reasoning. A plausible implication is that future progress will require tighter coupling between interface grounding modules and trajectory-level planners rather than improvements in OCR-like perception alone.

7. Position within dashboard-oriented QA research and future directions

DashboardQA is specifically a benchmark for interactive dashboard reasoning by multimodal agents, not a generic term for all dashboard-based quality assurance systems. This distinction matters because other dashboard-centered QA research addresses different domains and objectives. For example, "Metrics Dashboard: A Hosted Platform for Software Quality Metrics" focuses on longitudinal, team-level software engineering metrics mined from GitHub repositories rather than interactive multimodal QA (Thiruvathukal et al., 2018). "Analytics for Quality Assurance for Item Pools (AQuAP)" uses a dashboard environment to monitor item-bank health in AI-driven assessment systems through metrics such as EBS, AEBS, MCE, and RAF, again addressing operational analytics rather than agent benchmarking (Davier et al., 16 Jun 2026).

Within its own niche, DashboardQA’s main contribution is to expose the gap between static chart QA and real dashboard use. It operationalizes interactivity as part of the evaluation target: dashboard state must be changed before the relevant evidence even becomes visible. That design makes it especially relevant to research on GUI agents, multimodal planning, accessibility-grounded interaction, and analytical workflow automation (Kartha et al., 24 Aug 2025).

The benchmark’s stated limitations define an immediate research agenda. Its current coverage is Tableau-centric and web-based; desktop BI applications and other platforms such as Power BI and Zoho are not included. The interaction space could be extended to richer controls such as brushing, drag-based range selection, hover tooltips, and keyboard-driven filter entry. The dataset is English-only, and future versions could incorporate multilingual dashboards and broader visual conventions. The paper also identifies open directions in evaluation, including grounding precision, plan optimality, interaction success rates per tool, and state-consistency scoring.

In that sense, DashboardQA functions both as a benchmark and as a diagnosis tool. Its empirical results show that even strong systems remain far from reliable at interactive visualization QA, with best overall performance at 38.69% under Screenshot + A11y and severe weakness on multi-dashboard reasoning. This suggests that progress in dashboard-capable agents will depend on integrated advances in visual grounding, action planning, execution monitoring, memory, and compositional reasoning rather than on static-chart QA improvements alone (Kartha et al., 24 Aug 2025).