nvBench-CoT: Reasoning-Centric NL2VIS Benchmark

• nvBench-CoT is a reasoning-centric extension of nvBench, augmenting NL2VIS examples with explicit chain-of-thought supervision to improve transparency and traceability.
• The dataset uses a two-module pipeline with structured database descriptions and a five-stage reasoning schema, achieving an All Acc of 77.16% and outperforming non-CoT models by over 17.8 points.
• DeepVIS integrates nvBench-CoT to offer an interactive interface that exposes intermediate reasoning steps, enabling iterative refinement and effective error diagnosis.

Searching arXiv for the primary and background papers to ground the article with current citations. nvBench-CoT is a curated, reasoning-centric extension of the original nvBench Natural Language to Visualization benchmark that augments each NL2VIS example with explicit Chain-of-Thought supervision. Introduced alongside DeepVIS, it is designed to address the opacity and brittleness of black-box NL2VIS systems by representing the translation from ambiguous natural-language intent to executable visualization specification as a staged, inspectable process. Each item extends the original pairing of table, natural-language query, and Visualization Query Language program with a structured database description, representative value samples, and five textual reasoning stages aligned with grounded VQL clauses, thereby supporting both supervised model training and interactive reasoning inspection at inference time (Shuai et al., 3 Aug 2025).

1. Provenance within NL2VIS research

nvBench-CoT emerges from the NL2VIS line of work built around nvBench, which was introduced as the first large-scale NL2VIS benchmark and synthesized from cross-domain NL→SQL corpora to support end-to-end translation from natural language to visualization specifications. The original benchmark contains 25,750 NL–VIS pairs from 750 tables over 105 domains, and its role is to supply enough paired data for deep learning-based NL2VIS systems (Luo et al., 2021).

Subsequent work using nvBench emphasized direct sequence generation rather than staged reasoning. In particular, transformer-based and code-pretrained models were trained to map natural-language queries to Vega Zero command sequences, and the paper explicitly notes that nvBench as used there did not provide chain-of-thought rationales or intermediate reasoning annotations. That formulation treated NL2VIS as a sequence-to-sequence prediction problem aided by templates and schema hints, not as a transparent reasoning workflow (Wang et al., 2023).

Against that background, nvBench-CoT was created to make intermediate decisions explicit. The motivating problem is that prior NL2VIS systems often function as black boxes: they map from natural language directly to a visualization specification without exposing intermediate decisions. The stated consequence is reduced user trust, difficulty debugging suboptimal outputs, and lost opportunities to learn from model reasoning. nvBench-CoT therefore reframes NL2VIS as a sequence of justified design and data-selection decisions rather than a single opaque translation step (Shuai et al., 3 Aug 2025).

2. Construction pipeline and curation protocol

nvBench-CoT is built by augmenting nvBench with structured reasoning traces using a two-module pipeline. The first module is the Database Description Module, which renders schema information in a human-readable format and adds representative value samples for relevant columns. The schema string includes column names and types, such as facid:number, fname:text, rank:text. Value sampling is included to reduce predicate mismatches in operations such as WHERE, for example when a natural-language query refers to "Associate Professor" but the stored value is "AssocProf.". GPT-4o-mini identifies columns referenced by the natural-language query and schema and selects value examples accordingly. The reported ablation indicates that removing value sampling lowers Data Acc from 80.74% to 72.11% and All Acc from 77.16% to 69.31%, typically because incorrect predicates retrieve no data (Shuai et al., 3 Aug 2025).

The second module is the Reasoning Steps Generation Module. Using the ground-truth VQL and carefully designed prompts, GPT-4o-mini generates structured reasoning text for five stages, S1–S5. The paper states that GPT-4o-mini was selected as a compromise between reasoning quality and cost for large-scale generation. Each stage is aligned with VQL clauses and is required to justify decisions against the natural-language intent and schema.

Prompt constraints are a central part of the curation strategy. After post-tuning with Llama3.1-8B-Instruct, the authors observed illegal specifications such as VISUALIZE HISTOGRAM and invalid expressions such as WEEKDAY(Date) outside VQL syntax. They therefore restricted chart types to BAR, [PIE](https://www.emergentmind.com/topics/planetary-infrared-excess-pie), LINE, and SCATTER, and allowed only schema columns or valid derivations such as COUNT, AVG, MAX, and MIN. The ablation study reports that removing these constraints mainly degrades Chart Acc and Axis Acc and leads to invalid VQL usage.

The dataset also incorporates explicit cleaning and filtering. Rule-based filtering removed 41 problematic samples from nvBench: 9 duplicates, 26 illegal VQLs, and 6 empty VQLs. GPT-4o-mini was then used for inconsistency detection, flagging 1,351 items with query–VQL mismatches, which were removed. Finally, 15% of the augmented data was randomly sampled for manual evaluation of the generated reasoning steps.

3. Five-stage reasoning schema

The core design of nvBench-CoT is a validated five-stage CoT schema for NL2VIS derived from literature review and expert interviews. Each stage corresponds to a specific subset of visualization design and query-construction decisions, and each is paired with grounded VQL elements and prompt constraints to reduce illegal or non-compliant outputs (Shuai et al., 3 Aug 2025).

Stage	Function	VQL alignment
S1	Determine chart type	VISUALIZE
S2	Retrieve relevant data	FROM, SELECT, WHERE
S3	Define data granularity	GROUP BY, BIN BY
S4	Refine data for visualization	ORDER BY, SORT DIRECTION, LIMIT
S5	Generate visualization	Complete VQL

S1 determines chart type by analyzing the VISUALIZE clause and explaining how the chosen mark fulfills the intent. S2 identifies necessary tables, columns, and filters using FROM, SELECT, and WHERE, and justifies each selection and predicate. S3 specifies grouping, aggregation, and temporal binning through GROUP BY and BIN BY, linking granularity decisions to trends or summaries. S4 handles sorting and limiting through ORDER BY, sort direction, and LIMIT, primarily for readability and intent alignment. S5 synthesizes the prior decisions into the final executable VQL.

An example in the dataset illustrates the schema using the query, “Compute the total number of rank across rank as a pie chart.” The reasoning chain selects PIE in S1 because the task concerns parts of a whole, retrieves Rank and COUNT(Rank) in S2, groups by Rank in S3, omits sorting and limiting in S4, and emits the final VQL in S5:

VISUALIZE Pie SELECT Rank, COUNT(Rank) FROM Faculty GROUP BY Rank

This example is representative of the intended use of nvBench-CoT: each stage is not merely a decomposition of syntax, but an explicit textual rationale connecting task intent, schema structure, and executable specification.

4. Data representation and relation to executable specifications

Each nvBench-CoT item contains four principal elements. The first is the natural-language description expressing the user’s intent. The second is the augmented database description, consisting of a schema string with column names and types plus representative value samples for referenced columns. The third is structured CoT reasoning text aligned to S1–S5, where each stage references the natural-language intent, schema, value samples, and corresponding VQL clauses. The fourth is the final visualization specification in executable VQL form. The paper states that the VQL can be converted to Vega-Lite JSON for rendering, and DeepVIS supports export of either Vega-Lite JSON or SVG (Shuai et al., 3 Aug 2025).

This representation differs materially from earlier uses of nvBench. In the original nvBench setting, items paired natural-language queries with visualization specifications synthesized from NL→SQL sources, and the dataset was intended to support direct NL2VIS modeling (Luo et al., 2021). In later sequence-to-sequence work, the emphasis shifted toward direct prediction of Vega Zero strings with structural templates and schema blocks, but without intermediate rationales or CoT prompting (Wang et al., 2023). nvBench-CoT adds a reasoning layer that sits between intent and specification and is explicitly grounded in VQL clauses rather than inferred post hoc.

The dataset also preserves evaluation discipline inherited from nvBench practice. The main paper reports use of the standard nvBench train/dev/test split protocol following Song et al. (RGVisNet) to prevent database leakage across splits. A plausible implication is that the contribution is not only interpretability at annotation time, but compatibility with existing NL2VIS benchmark procedures.

5. Benchmark performance, ablations, and failure modes

The principal quantitative evaluation fine-tunes Llama3.1-8B-Instruct on nvBench-CoT using five-stage CoT supervision and augmented inputs. The resulting system, labeled NL2VIS-CoT, is compared against rule-based and translation models (Seq2Vis, Transformer, ncNet, RGVisNet), general-purpose LLMs with few-shot prompting, and ChartGPT, which also decomposes generation but uses a different step order. Evaluation uses five dimensions defined by comparison between generated VQL and ground truth: Chart Acc, Axis Acc, SQL Acc, Data Acc, and All Acc (Shuai et al., 3 Aug 2025).

On the test set, NL2VIS-CoT achieves Chart Acc 97.52%, Axis Acc 95.17%, SQL Acc 74.63%, Data Acc 80.74%, and All Acc 77.16%. The corresponding end-to-end baseline with the same backbone but without CoT supervision, Llama3.1-8B-SFT, reaches All Acc 59.37%. RGVisNet reaches All Acc 51.70%, and ChartGPT reaches All Acc 73.03%. The paper summarizes the gain as over 17.8 points in All Acc versus the same backbone trained without CoT, while also outperforming prior staged approaches across all metrics.

The ablation study isolates the importance of individual nvBench-CoT components. Without value sampling, All Acc drops from 77.16% to 69.31%. Without prompt constraints, it drops to 75.39% and produces illegal or unsupported function use such as WEEKDAY(). Without CoT, All Acc drops to 49.31%, which the paper presents as evidence that CoT is critical for performance and correctness. The comparison between ChartGPT’s step order and nvBench-CoT’s five-stage order reports All Acc 77.16% versus 73.54%, indicating that the validated stage ordering contributes materially to performance.

The error analysis is organized by reasoning stage. Reported error counts are S1 56, S2 183, S3 202, and S4 253. Frequent issues include 131 aggregation function errors in S2, 191 GROUP BY errors in S3, and 220 ORDER BY errors in S4. Chart-type error rates vary across marks: BAR 1.18%, PIE 2.97%, SCATTER 7.63%, and LINE 9.83%. The paper attributes many of these errors to multi-solution scenarios in which several chart types could satisfy the same query, while the evaluation framework assumes a single valid answer. This suggests that some measured chart-selection failures are partly artifacts of benchmark formulation rather than purely model misunderstanding.

6. Integration with DeepVIS, user-facing role, and prospective extensions

nvBench-CoT is operationalized through DeepVIS, an interactive interface that exposes the five-stage reasoning tree at inference time. In the CoT view, S5 appears as the root containing the final VQL, while S1–S4 appear as child nodes with concise summaries and details-on-demand justifications. Selecting a node updates an Information pane with the full textual rationale and a Table pane with the data after that step; the Chart pane renders the final chart from the generated VQL and supports export as Vega-Lite JSON or SVG (Shuai et al., 3 Aug 2025).

DeepVIS supports two refinement modes. In self-correction, the model reconsiders decisions using hints about potential errors in a flagged step. In manual correction, users specify preferences such as chart type or aggregation, and the system regenerates downstream steps to preserve logical consistency. The interface differentiates changed from unchanged nodes, highlights affected fields, and appends new reasoning for comparison. The use cases reported in the paper illustrate both modes: one case adds ORDER BY DESC to reveal the maximum category more clearly, while another changes “analyze age distribution” to “show average age per major,” switching the chart to BAR, adding AVG(age), and introducing GROUP BY major.

The accompanying user study evaluates ncNet, DeepSeek, ChartGPT, and DeepVIS on 10 randomly sampled nvBench examples with 32 participants, including 20 students and 12 industry professionals aged 20–51, of whom 25 had used LLMs for visualization. DeepVIS receives higher ratings on insights communication, intent reflection, logic comprehension, error identification, refinement efficiency, and workload saving. Negative ratings for refinement efficiency are reported as only 9%, compared with 50% for ncNet, 38% for DeepSeek, and 47% for ChartGPT. Participant feedback emphasizes that transparent reasoning helps identify errors, iterate quickly, and refine natural-language queries with useful hints.

The paper also identifies several limitations. Errors remain concentrated in retrieval and transformation logic, especially aggregation, grouping, and sorting. Automatic annotation noise is possible because CoT steps and value sampling are generated by GPT-4o-mini, although the prompts, constraints, manual QA, and removal of problematic items are intended to reduce this risk. Coverage is limited by VQL-to-Vega-Lite conversion, which abstracts away rendering-specific details such as color scales and mark sizes. Planned extensions include additional CoT fields such as COLOR_BY and MARK_SIZE, multimodal feedback from final charts back into the reasoning process, automatic linting with tools such as VizLinter, and adaptation to richer datasets such as nvBench 2.0 with multiple valid outputs per step. The dataset, prompts, training code, deployment code, videos, and supplemental materials are publicly available through the DeepVIS repository, while license terms are not specified in the paper text and are deferred to the repository (Shuai et al., 3 Aug 2025).