ViseGPT: LLM Code Validation Tool

• ViseGPT is an interactive tool for validating LLM-generated data-wrangling scripts by converting natural language prompts into semantic constraints and test cases.
• It decomposes Python/pandas scripts using AST parsing and visualizes execution results with a tailored Gantt chart to highlight pass/fail statuses.
• Empirical studies show that ViseGPT improves debugging accuracy and reduces code refinement time compared to traditional LLM-driven workflows.

ViseGPT is an interactive “test-and-visualize” companion for LLM-generated data-wrangling scripts. It takes a user’s natural-language description of a wrangling task, automatically derives semantic constraints from that description, turns these constraints into unit-test-like checks, executes them on the LLM’s script, and visualizes which parts of the script and which columns comply or fail, so users can debug and iteratively refine the code. The system is designed to improve the alignment between user prompts and generated pandas-style programs, and its design is informed by a formative study $(N=8)$ and evaluated in a user study $(N=18)$ (Zhu et al., 2 Aug 2025).

1. Definition, scope, and conceptual orientation

ViseGPT is defined as a tool for validating and refining LLM-generated data-wrangling code rather than as a monolithic code-generation model. Its central premise is that natural-language prompts should function as a specification: the prompt is analyzed for semantic requirements, those requirements are converted into constraints, and the resulting constraints are used to verify whether the generated script actually satisfies the intended transformation semantics. In this sense, ViseGPT shifts the emphasis from “Explain this script to me” to “Does this script satisfy my prompt?” (Zhu et al., 2 Aug 2025)

The motivating problem is that LLMs can rapidly generate complex data-wrangling code, but the resulting scripts may hallucinate functions or columns, misinterpret vague instructions, silently ignore parts of the prompt, mishandle edge cases, or make hidden assumptions about data types and formats. Existing tools around data wrangling and LLM-generated code largely help users inspect logic and data flows, but they leave validation to the user. ViseGPT addresses that gap through automatic constraint extraction, test generation, step-wise execution, and visualization-driven debugging (Zhu et al., 2 Aug 2025).

A common misconception is to equate ViseGPT with a vision-language assistant because similar names appear in adjacent GPT-centered systems. The published ViseGPT paper instead defines a data-wrangling validation environment. A plausible implication is that the term sits at an intersection of two lines of research: prompt-aligned program validation on the one hand, and broader GPT-style multimodal tooling on the other (Zhu et al., 2 Aug 2025).

2. System architecture and execution workflow

The architecture is organized around three components. The Test Case Generator takes the user’s prompt and, via an LLM-based analysis model, extracts data constraints and turns them into test cases. The Reliability Validator takes the LLM-generated script and test datasets, splits the script into atomic steps, runs it step-by-step, and matches outputs at each step against the test cases to generate a test report. The Script Refiner uses test reports, user instructions, and data examples as part of a new prompt to the code LLM, which returns an updated script (Zhu et al., 2 Aug 2025).

The end-to-end workflow proceeds in five stages. First, the user writes a natural-language description of the wrangling task, and ViseGPT’s embedded code agent generates a Python/pandas script. Second, a separate analysis component extracts constraints from the prompt. Third, the system automatically splits the script into atomic steps, runs a test dataset through those steps, and checks all relevant constraints for each step and each involved column or variable. Fourth, the results are rendered in a tailored Gantt chart embedded in the chat interface. Fifth, the user inspects failures, optionally edits constraints, reruns tests, and sends test reports back to the LLM for guided regeneration (Zhu et al., 2 Aug 2025).

Script decomposition relies on Python’s AST parser. The backend uses the AST to break the script into atomic steps, extract input and output tables or columns for each step, and ignore non-essential statements that do not affect data, such as print or inserted assert statements. This step-level structure underpins both the validator and the visualization (Zhu et al., 2 Aug 2025).

The prototype is implemented as a web application with a React and TypeScript front end. The backend uses Llama-3.3-70b-Instruct hosted by NVIDIA, Python and pandas for script execution and static predicates, and WebSockets to stream responses and dynamically update test results. The system is described as model-agnostic (Zhu et al., 2 Aug 2025).

3. Constraint extraction, test semantics, and validation logic

ViseGPT’s constraint model formalizes what the user expects of the data. The taxonomy builds on Ferry’s classification of data constraints—Type, Format, Range, Order, and Missing—and extends it with Unique, Forbidden Value, and richer Relation categories. Relation constraints may encode equality, magnitude relations for numeric data, substring or superstring relations for strings, or more complex dependencies expressed in natural language (Zhu et al., 2 Aug 2025).

Constraint extraction begins with prompt analysis by an LLM. The analysis model receives the original natural-language prompt and optionally metadata such as column names, then outputs a structured description of constraints organized by column and category. Those descriptions are post-processed into a unified constraint representation. Basic categories are translated into logical checks or regexes; Relation constraints remain natural-language descriptions when they are too complex for a simple deterministic formalization and are later evaluated through the LLM (Zhu et al., 2 Aug 2025).

Ambiguity is handled through defaults and user-editable constraints. If the prompt omits details such as output format, the system may adopt a reasonable default. The paper’s scenario notes that the system defaulted to a date format regex \d{2}-\d{2}-\d{4} when the user had not explicitly specified YYYY-MM-DD. Users can subsequently edit that constraint in the interface, rerun validation, and reconcile the test suite with actual intended semantics (Zhu et al., 2 Aug 2025).

Each constraint is mapped into one or more test cases. Static checks such as Type, Format, Range, Unique, Forbidden Value, Exception, and Order are evaluated with deterministic predicates in the backend. More complex Format, Forbidden Value, and Relation cases can be evaluated by sending a constraint description and representative data samples to the LLM. Test datasets may be user-provided or system-generated; when no test data are provided, the system synthesizes data intended to cover the prompt and variable requirements, including edge cases (Zhu et al., 2 Aug 2025).

Execution is step-wise. For each script step, the system executes the transformation on the current dataset, records the resulting variables, determines which constraints apply, and evaluates the corresponding test cases. At the $(\text{step}, \text{variable})$ level, a failure occurs if any associated test fails. At the summary level, the interface reports statistics such as the number of failed test cases and the pass rate, defined as $\frac{\#\text{passed tests}}{\#\text{total tests}}$ (Zhu et al., 2 Aug 2025).

4. Visualization design and interactive debugging

The signature interface is a tailored Gantt chart aligned to the script. Rows correspond to script steps in execution order, and rectangles at each row indicate the variables read or written at that step. The chart is designed for linear, sequential scripts, which the paper characterizes as common in many pandas-style data-wrangling pipelines. Nonlinear logic such as branching or loops is explicitly left to future work (Zhu et al., 2 Aug 2025).

Color encodes reliability. A green rectangle indicates that all test cases for a variable at a given step pass; a red rectangle indicates that some tests fail. A summary row at the bottom aggregates per-column statistics, including the number of failed test cases and the pass rate. This provides a rapid overview of where misalignment begins and how it propagates through downstream columns (Zhu et al., 2 Aug 2025).

Interaction is centered on drill-down inspection. Clicking a rectangle opens a detail panel showing the variable name, step ID, the list of test cases for that variable at that step, pass/fail status for each case, explanatory text, and controls for editing constraints, sending the test report to the LLM, and filtering data rows that violate the selected test. A “magnify” control expands the view from a single column to the full table, which is especially useful when reasoning about multi-column transformations (Zhu et al., 2 Aug 2025).

The paper’s representative “Mary” scenario illustrates the debugging model. Mary observes failures beginning in the Amount column, then propagating to Cumulative Sum, Price, Price Comparison, and Normalized Amount. Inspection reveals values such as “~50”, which prevent correct numeric conversion and introduce NaNs. Later, she discovers that a date format test is misaligned with actual processed output and edits the regex from \d{2}-\d{2}-\d{4} to \d{4}-\d{2}-\d{2}. She also identifies that Amount Group is being computed from the original Amount rather than Normalized Amount, uses the full-table view to confirm the issue, and sends the test report back to the LLM for script regeneration. The scenario is intended to demonstrate propagation-aware debugging rather than mere output inspection (Zhu et al., 2 Aug 2025).

5. Empirical evaluation

The formative study involved 8 participants: 4 postgraduate students doing data analysis, 2 computer science undergraduates, and 2 data journalists. Participants brought real chat histories where LLM-generated scripts had been suboptimal. The study identified four recurrent themes: LLMs are helpful but unreliable for complex tasks; current debugging methods are fragmented; validation of large outputs is tedious; and iterative dialogue with LLMs can accumulate hallucinations and new bugs. These observations produced five design implications: automatically generate scripts and test cases from prompts, execute tests and visualize results, support inspection and test customization, feed test reports back to the LLM for modification, and integrate with existing chat-based workflows (Zhu et al., 2 Aug 2025).

The controlled user study involved 18 participants in a within-subjects design. The baseline used the same base LLM and chat interface styling but omitted the visualization and test-driven debugging features. Four tasks were constructed from common real-world failure patterns: length mismatch in string, missing boundary in grouping, inconsistent letter casing in color codes, and non-compliant rounding. Each task included a syntactically correct but functionally wrong pandas script (Zhu et al., 2 Aug 2025).

The most salient quantitative result is that ViseGPT improved task success rates across all four tasks.

Task	ViseGPT success	Baseline success
A: length mismatch in string	100%	78%
B: missing boundary in grouping	100%	33%
C: inconsistent letter casing	56%	22%
D: non-compliant rounding	89%	11%

ViseGPT also generally reduced the number of LLM queries. Mean query counts were 2.00 vs. 3.33 for Task A, 1.78 vs. 4.11 for Task B, 2.33 vs. 3.11 for Task C, and 3.00 vs. 2.78 for Task D, with the Task D anomaly attributed to two baseline participants stopping early because they incorrectly believed the script was already correct. More than 60% of ViseGPT queries included attached test reports, and participants actively used the visualization, with mean Gantt clicks of 7.56, 3.89, 9.67, and 8.89 for Tasks A–D, respectively (Zhu et al., 2 Aug 2025).

Completion time was significantly reduced for Tasks A, B, and D, with mean differences greater than 120 seconds. Task C showed no substantial difference: 625 seconds for ViseGPT versus 645 seconds for the baseline. The paper attributes this to a gap between symptom-level and cause-level reasoning. ViseGPT correctly generated a rank_label ∈ [1,5] constraint and detected that rank_label exceeded 5, but many participants described only the symptom to the LLM rather than the underlying case-normalization error, which led the LLM to apply superficial fixes such as truncating results with head() (Zhu et al., 2 Aug 2025).

Subjective evaluation via the User Experience Questionnaire reported that ViseGPT significantly outperformed the baseline on all six UEQ dimensions. Shapiro–Wilk normality tests yielded $p > 0.05$, and paired $t$-tests at $\alpha = 0.05$ showed significance on every dimension. Interview feedback emphasized that the visualization made debugging more intuitive and that test reports served as actionable context, though some participants noted that the system could be less narrative than free-form chat explanations and that test coverage might still be incomplete (Zhu et al., 2 Aug 2025).

6. Relation to adjacent GPT-based systems and naming ambiguity

Although the published ViseGPT system concerns data-wrangling validation, adjacent research has used “ViseGPT-like” as an informal label for GPT-centered multimodal or tool-using systems. This broader usage helps explain naming ambiguity but does not refer to the same artifact. A common thread is the use of GPT-style LLMs as controllers, planners, or unified sequence models over non-text modalities (Zhu et al., 2023).

One nearby line treats a GPT backbone as a unified multimodal generator. VL-GPT is described as a single decoder-only Transformer initialized from LLaMA-7B, with an image tokenizer, text tokenizer, image detokenizer, and text detokenizer, trained with a unified auto-regressive objective over mixed image-text sequences and evaluated on captioning, visual question answering, text-to-image generation, in-context learning, and instruction tuning (Zhu et al., 2023). A second line treats the LLM as a tool orchestrator rather than a direct multimodal encoder: VisionGPT uses LLaMA-2 as the pivot to break down user requests into action proposals and automatically integrate outputs from foundation models such as YOLO, SAM, Stable Diffusion, CLIP, and OpenCV (Kelly et al., 2024). VIoTGPT extends this controller pattern to Video Internet of Things settings, where Llama-7B and Vicuna-7B are instruction-tuned with ReAct to schedule 11 specialized vision tools over large video knowledge bases (Zhong et al., 2023).

A further extension of the GPT analogy appears in video world modeling, affective computing, metaverse support, and 3D vision roadmaps. Video-GPT treats video as a new language and uses next clip diffusion for world modeling and downstream video tasks (Zhuang et al., 18 May 2025). GPT-4V has also been evaluated on visual affective computing, where it shows strong facial action unit detection but weak micro-expression recognition and poor deception detection (Lu et al., 2024). Other work frames GPT as infrastructure for metaverse-based education, entertainment, personalization, and support while foregrounding privacy, bias, and dependence risks (Zhou, 2023), and asks when a “ChatGPT for Computer Vision” from 2D to 3D might become feasible under appropriate architectural and data conditions (Li et al., 2023).

Security research complicates this broader GPT-for-vision trajectory. “Goal hijacking via visual prompt injection” shows that GPT-4V can be redirected from an intended task to an attacker’s alternative task, with an attack success rate of 15.8%, and that a simple system-prompt defense can reduce that rate to 1.8% without eliminating it entirely (Kimura et al., 2024). A plausible implication is that any future system named or conceptualized in the “ViseGPT-like” space of multimodal assistants will need explicit modality-aware instruction hierarchies and adversarial handling of text embedded in images.

7. Limitations and future directions

ViseGPT’s core limitation is its dependence on LLM-based constraint extraction. If the analysis model misreads the prompt, the resulting test suite can itself be misaligned, as illustrated by the default date-format mismatch in the Mary scenario. The system mitigates this through editable constraints, but validation quality remains partly bounded by the quality of prompt interpretation (Zhu et al., 2 Aug 2025).

Constraint coverage is another limitation. The current taxonomy captures many common issues—Type, Format, Range, Unique, Forbidden Value, Missing, and Relation—but participants explicitly worried that the tests might not cover every relevant semantic failure. This creates a risk of false confidence: a script may pass the present tests without satisfying every latent requirement (Zhu et al., 2 Aug 2025).

The visualization is optimized for linear, pandas-style scripts and can become vertically long for large programs because each AST step is represented as a row. Real-world pipelines may include complex one-liners, joins, notebook state, loops, or branching logic that are not fully supported. The paper also notes language and framework scope: the current implementation targets Python/pandas, and extending the approach to SQL, R, or PySpark would require new parsers and constraint mappings (Zhu et al., 2 Aug 2025).

Another limitation is incomplete data lineage support. Users can inspect step-wise snapshots and filter failing rows, but the system does not yet provide explicit row-level causality tracing across transformations such as sorts. Participants suggested severity distinctions such as warning versus error and richer tracing of how particular “bad rows” move through the pipeline (Zhu et al., 2 Aug 2025).

The stated future directions are correspondingly pragmatic: expand test categories, allow user-defined categories, add severity levels, improve dataflow and lineage visualization, introduce hierarchical abstractions for longer scripts, support more languages and frameworks, integrate with IDEs and notebook environments, and explore dynamic learning of constraints from user interaction. Taken together, these directions position ViseGPT not as a replacement for LLM code generation, but as a reliability layer that turns natural-language prompts into executable validation criteria and uses visualization to narrow the gulf between user intent and generated program behavior (Zhu et al., 2 Aug 2025).