ChatVis: Conversational Visualization

Updated 7 July 2026

ChatVis is a design pattern integrating conversational inputs with visualization, enabling natural language to drive scientific scripts and immersive 3D chat interfaces.
It employs retrieval-augmented prompt generation and iterative correction loops to ensure both code executability and high-fidelity visual outputs.
ChatVis systems span diverse applications—from generating ParaView Python scripts and multilingual charting to conversation analytics and embodied 3D online communication.

ChatVis denotes a family of systems in which conversational interaction and visualization are tightly coupled. In the cited literature, the term names an LLM assistant for generating ParaView Python scripts for scientific visualization, but it is also used more broadly for conversational natural-language-to-visualization systems, visual analytics environments for long ChatGPT sessions, and embodied 3D chat interfaces in which participants are represented as animated avatars in shared virtual space (Peterka et al., 30 Jul 2025, Mallick et al., 2024, Maddigan et al., 2023, Liang et al., 2023, Yan et al., 2022).

1. Terminological scope and representative systems

Published work associated with ChatVis spans at least four distinct, partially overlapping strands: scientific-visualization code generation, multilingual NL2VIS, conversation visualization for LLM sessions, and 3D embodied chatrooms. The literature does not present these as a single unified architecture; rather, the shared theme is that chat, conversational context, or communication structure becomes the primary substrate for visualization or for generating visualizations (Peterka et al., 30 Jul 2025, Maddigan et al., 2023, Liang et al., 2023, Yan et al., 2022).

System	Primary function	Representative paper
ChatVis	Generate ParaView Python scripts from natural language	(Mallick et al., 2024, Peterka et al., 30 Jul 2025)
Chat2VIS	Multilingual NL2VIS via LLM-generated plotting code	(Maddigan et al., 2023)
C5	Visualize and manage long multi-turn ChatGPT sessions	(Liang et al., 2023)
WOC	Webcam-based 3D virtual online chatroom with avatars	(Yan et al., 2022)

A recurrent misconception is that ChatVis refers only to a single ParaView assistant. The broader record is more heterogeneous. Chat2VIS is explicitly described as an early concrete realization of what many people now call “ChatVis”; C5 is positioned in the broader ChatVis landscape as a conversation visualization system specialized for ChatGPT sessions; and WOC is described as essentially a full-stack prototype of what one might call a “ChatVis” system for the metaverse era (Maddigan et al., 2023, Liang et al., 2023, Yan et al., 2022).

2. ChatVis as an LLM assistant for scientific visualization

The most specific use of the name appears in the ParaView-oriented papers. The 2024 system develops an iterative assistant called ChatVis that synthetically generates Python scripts for data analysis and visualization using a LLM. A user specifies operations in natural language; the system attempts to generate a ParaView Python script; then it executes the script, extracts runtime errors, and prompts the LLM to revise the script until it executes correctly. The architecture includes a prompt generator, example code snippets for few-shot prompting, ParaView’s PvPython execution environment, an error extraction tool, and an iterative correction loop (Mallick et al., 2024).

The 2025 system retains the execution-and-repair structure but adds a more elaborate agent pipeline. It performs chain-of-thought prompt simplification, retrieval-augmented prompt generation using a vector database, code generation with GPT-4o, execution with pvpython, error extraction from stdout and stderr, and iterative repair until the script runs successfully or a limit is reached. The vector database is built with Faiss from ParaView Python API documentation, canonical visualization tasks, ParaView regression test scripts, and scientific use case scripts; each chunk is embedded with all-MiniLM-L6-v2 (Peterka et al., 30 Jul 2025).

A central design element is operation extraction. Rather than emitting unconstrained reasoning text, the system converts a free-form prompt into a step-by-step list of ParaView operations, separating different applications of the same command and distinguishing variant forms of operations. This explicit action list is then used as the query substrate for retrieval. The retrieved documentation and example scripts are injected into the final prompt, so the LLM is grounded in actual ParaView API text instead of relying on latent recall alone (Peterka et al., 30 Jul 2025).

The retrieval stage is based on similarity search over embedded documentation and code snippets. For an operation description $q$ with embedding $\mathbf{v}_q$ and a stored document $d_i$ with embedding $\mathbf{v}_i$ , the paper describes the search conceptually via cosine similarity: $\text{sim}(q, d_i)=\cos(\mathbf{v}_q,\mathbf{v}_i)=\frac{\mathbf{v}_q\cdot \mathbf{v}_i}{\lVert \mathbf{v}_q\rVert \,\lVert \mathbf{v}_i\rVert }.$ This grounding mechanism is presented as the main reason the assistant avoids hallucinated ParaView APIs such as invalid attribute names or outdated signatures (Peterka et al., 30 Jul 2025).

The 2024 and 2025 systems also clarify a key distinction between code generation and visualization generation. In the volume-rendering case discussed in the 2024 paper, an unassisted GPT-4 script can run without syntax errors yet fail to set up volume rendering correctly. The literature therefore treats syntax correctness as necessary but not sufficient; visual output must also be evaluated against ground truth (Mallick et al., 2024).

3. Benchmarks, metrics, and empirical performance

The 2024 ChatVis paper evaluates five canonical visualization scenarios in ParaView: isosurfacing, slicing followed by contouring, volume rendering, Delaunay triangulation with clipping, and streamline tracing with tubes and glyphs. In every instance, ChatVis successfully generated the correct script, whereas the unassisted LLMs failed to do so. The comparison includes GPT-4, GPT-3.5-turbo, LLaMA 3 8B, Code LLaMA 7B, and CodeGemma; only unassisted GPT-4 succeeds on the isosurfacing example, and even there its background differs from the ground truth (Mallick et al., 2024).

The 2025 paper introduces a much broader 20-task benchmark with three groups: eight canonical visualizations, ten ParaView regression tests, and two scientific use cases. Each task has a ground-truth reference image, a hand-written reference ParaView Python script, and two natural-language prompts, designated full and quick. Evaluation uses pass@1 for first-script executability and three image metrics—SSIM, PSNR, and LPIPS—computed on successful runs (Peterka et al., 30 Jul 2025).

On that 20-task benchmark, ChatVis achieves pass@1 95, SSIM 0.80, PSNR 40.1 dB, and LPIPS 0.26. The best unassisted baseline, o1pro, records pass@1 50, SSIM 0.70, PSNR 35.7 dB, and LPIPS 0.45. Other reported baselines are o3-mini-high at 35/0.58/30.7/0.55, GPT-4.5-Preview at 40/0.66/32.8/0.43, Claude-3.7-Sonnet at 40/0.68/34.4/0.44, and LLaMA-3.1-405B at 15/0.57/24.7/0.59 (Peterka et al., 30 Jul 2025).

The ablation on retrieval is especially consequential. ChatVis with RAG reports pass@1 95, SSIM 0.80, PSNR 40.1 dB, and LPIPS 0.26, whereas the few-shot-only variant reports pass@1 70, SSIM 0.69, PSNR 34.0 dB, and LPIPS 0.46. The paper further states that with RAG the system typically succeeds in 1 iteration, whereas without RAG 2–5 correction iterations are common and may still fail (Peterka et al., 30 Jul 2025).

Prompt specificity also matters. With RAG enabled, full prompts yield pass@1 95, SSIM 0.80, PSNR 40.1 dB, and LPIPS 0.26, while quick prompts reduce success to pass@1 75 with SSIM 0.78, PSNR 39.6 dB, and LPIPS 0.33. This establishes prompt sensitivity as an empirical property of the system rather than a merely anecdotal concern (Peterka et al., 30 Jul 2025).

The two scientific use cases in the 2025 benchmark are noteworthy because no unassisted LLM produced an executable script for them. On the SOMA climate case, ChatVis reports SSIM 0.99, PSNR 36.2 dB, and LPIPS 0.01; on the CDI nanocrystal reconstruction case, it reports SSIM 0.97, PSNR 46.8 dB, and LPIPS 0.05 (Peterka et al., 30 Jul 2025).

Chat2VIS represents a different branch of the ChatVis idea. It is an NL2VIS system built as a web application in Streamlit. Users select a dataset and describe the desired visualization in free-form natural language; the system constructs a prompt containing a Python docstring that describes the task and dataset, appends Python seed code that imports libraries and loads the dataset into a Pandas DataFrame, sends the prompt to an OpenAI model, receives Python plotting code, executes the code, and renders the visualization. The system supports iterative refinement through follow-up natural-language instructions (Maddigan et al., 2023).

The implementation uses GPT-3 text-davinci-003, Codex code-davinci-002, and ChatGPT gpt-3.5-turbo. For GPT-3 and Codex, the reported API parameters are temperature = 0, max_tokens = 500, and stop = "plt.show()". The paper emphasizes that the system does not use an explicit intermediate visualization grammar such as Vega-Lite; the operative internal representation is Python code, typically Matplotlib-based (Maddigan et al., 2023).

A distinguishing property of Chat2VIS is multilingual interaction without retraining or a translation layer. The paper states that no other NL2VIS system had demonstrated this capability. Its case studies show French instructions for grouping and axis assignment, Croatian instructions for title changes, English instructions for label language and font sizes, and Te Reo Māori instructions for color choices. The multilingual capability is attributed to the general language understanding of the underlying LLMs rather than to task-specific multilingual training (Maddigan et al., 2023).

The system is evaluated on nvBench and nlvUtterance. On a filtered nvBench subset of 3,003 bar-chart examples from 138 databases—812 easy, 1,572 medium, 386 hard, and 233 extra hard—Chat2VIS with Codex yields 1,280 matches, 1,387 mismatches, and 336 errors, corresponding to 43% accuracy. The paper compares this with Seq2Vis at 2%, Transformer at 3%, ncNet at 26%, and RGVisNet at 45% (Maddigan et al., 2023).

On nlvUtterance, the reported accuracies are 50% for Chat2VIS with Codex only, 63% for Chat2VIS with Codex plus GPT-3, and 72% for Chat2VIS with Codex plus GPT-3 plus ChatGPT; NL4DV is reported at 64%. The authors also note that many apparent mismatches are semantically plausible visualizations that differ in chart type or encoding, which makes strict benchmark matching an imperfect proxy for usefulness (Maddigan et al., 2023).

A common misconception is that chat-based visualization systems are restricted to simple 2D charting. Chat2VIS does center on chart generation, but the ParaView ChatVis systems target 3D and 4D scientific visualization pipelines, including volume rendering, streamlines, multiblock datasets, lighting, camera control, and multiview layouts (Maddigan et al., 2023, Peterka et al., 30 Jul 2025).

5. ChatVis as conversation visualization and context management

C5 extends the meaning of ChatVis from “visualization generated from chat” to “visualization of chat itself.” It addresses two problems in long multi-turn ChatGPT sessions: human forgetting and model contextual forgetting. The system is explicitly designed for conversation comprehension and contextual continuity, with three linked views: Global View, Topic View, and Context-associated Q&A View (Liang et al., 2023).

The Global View uses a GitLog diagram metaphor. The x-axis represents conversation order, the y-axis represents topic index, conversation nodes are colored by topic, solid lines connect consecutive nodes, and a green dashed “forgotten line” marks turns likely outside ChatGPT’s context window. A linked Content View provides a magnified region and a word cloud in which size encodes importance via TF–IDF rather than raw frequency. A semantic search box embeds the query and highlights matching nodes in the Brush View (Liang et al., 2023).

Topic modeling and node similarity are embedding-driven. Each node is embedded with a GPT-3.5-based embedding model; GPT-3.5 Turbo is used for summarization, topic extraction, and recursive subtopic generation. Topic View displays nodes as nested circular charts whose outer ring encodes similarity to topics and inner ring encodes similarity to subtopics. The similarity is defined by cosine similarity, for example

$\text{sim}(n,t)=\frac{\mathbf{v}_n\cdot \mathbf{v}_t}{\|\mathbf{v}_n\|\,\|\mathbf{v}_t\|}.$

Edges are drawn when node-node similarity exceeds a threshold, and edge thickness and darkness are proportional to similarity (Liang et al., 2023).

The Global View’s y-axis ordering is optimized to reduce “wiggle” in topic trajectories. The paper formulates the layout as a quadratic assignment problem with binary variables $x_{ij}$ indicating whether topic $i$ is placed at vertical position $j$ , and an objective that minimizes weighted vertical displacement across topic transitions. Gurobi is used to solve the optimization (Liang et al., 2023).

The Context-associated Q&A View closes the loop between visualization and subsequent LLM interaction. Users inspect full Q&A text, type a new question, semantically retrieve related prior nodes, and add selected nodes to a Context View list before sending the question plus chosen context back to ChatGPT. The paper explicitly frames this as a visual, human-in-the-loop retrieval-augmented mechanism over conversation history (Liang et al., 2023).

The reported evaluation combines a case study and a user study. In UT1, the group using C5 achieved objective score mean $\mu = 8$ , $\mathbf{v}_q$ 0, subjective score $\mathbf{v}_q$ 1, $\mathbf{v}_q$ 2, and average completion time 8 minutes, whereas the group using the raw ChatGPT web page achieved objective $\mathbf{v}_q$ 3, $\mathbf{v}_q$ 4, subjective $\mathbf{v}_q$ 5, $\mathbf{v}_q$ 6, and time about 19 minutes. In UT2, answers produced with context supplied via C5 were rated at mean $\mathbf{v}_q$ 7, $\mathbf{v}_q$ 8, compared with $\mathbf{v}_q$ 9, $d_i$ 0 without context (Liang et al., 2023).

6. ChatVis as embodied 3D online chat

WOC broadens ChatVis still further by moving from textual logs and coding assistants to embodied, spatialized communication. It is a webcam-based 3D virtual online chatroom for multi-person interaction in which a single monocular RGB webcam captures the 3D motion of all people in view and drives their individual 3D virtual avatars in real time. The system is deployed on the web, requires no installation, and synchronizes motion and voice for all users through a distributed data flow service (Yan et al., 2022).

The architecture comprises three tightly coupled modules: monocular motion capture, automatic avatar animation, and distributed deployment and visualization. Clients capture webcam video and microphone audio, stream them to a cloud server through WebRTC, receive motion states of all participants, and render avatars in the browser with three.js. The server spawns a dedicated GPU motion-capture process per client, runs the pose estimation pipeline, maintains room membership, aggregates motion states, and sends synchronized motion data back to clients in the same chatroom (Yan et al., 2022).

The motion-capture module is built on ROMP and represents each person with SMPL pose parameters $d_i$ 1, SMPL shape parameters $d_i$ 2, and translation $d_i$ 3. For an image $d_i$ 4, the description states that ROMP conceptually produces

$d_i$ 5

for each detected person. To maintain identity over time, WOC develops a 3D-based tracker inspired by ByteTrack but uses ROMP’s per-person 3D translation $d_i$ 6 instead of 2D bounding boxes. Temporal jitter is reduced with three individual OneEuro filters per tracked person, applied to orientation, pose, and translation, with adaptive cutoff

$d_i$ 7

Slow motions therefore receive stronger smoothing, while fast motions preserve responsiveness (Yan et al., 2022).

Avatar animation is based on SMPL-compatible skeletons. Given a user-uploaded mesh $d_i$ 8, a MeshCNN-based network predicts the 3D coordinates of the standard 24 SMPL joints,

$d_i$ 9

after which an auto-rigging algorithm in Blender computes skinning weights $\mathbf{v}_i$ 0 satisfying $\mathbf{v}_i$ 1 and $\mathbf{v}_i$ 2. Because the inferred skeleton is SMPL-compatible, live SMPL poses from ROMP can directly drive arbitrary user-defined characters. The system also supports Mixamo avatars via a conversion module (Yan et al., 2022).

The reported performance is primarily engineering rather than perceptual. Input frames are $\mathbf{v}_i$ 3. A single 1070Ti achieves more than 30 FPS and supports about two concurrent clients in real time; a single 3090Ti achieves more than 60 FPS and about four clients; an 8×3090 GPU server runs one process per user at about 20 FPS. The pure processing latency on the cloud, ignoring network, is about 50 ms from receiving an image to sending the result (Yan et al., 2022).

From the perspective of terminology, WOC shows that ChatVis can also denote the visualization of communication itself as an embodied, co-located, 3D scene rather than as text logs, 2D thumbnails, or plotting code. The paper explicitly contrasts this with traditional text chat, standard 2D video chat, and wearable-equipment-based VR systems (Yan et al., 2022).

7. Recurrent design patterns, limitations, and research directions

Across these systems, several architectural patterns recur. First, all rely on external grounding rather than unconstrained generation. The ParaView ChatVis assistant grounds code generation in documentation, examples, and execution traces; Chat2VIS grounds plotting code in dataset descriptions and seed scripts; C5 grounds new questions in retrieved conversation nodes; and WOC grounds avatar motion in webcam-based 3D pose capture rather than in hand-authored animation (Peterka et al., 30 Jul 2025, Maddigan et al., 2023, Liang et al., 2023, Yan et al., 2022).

Second, iterative refinement is central. The ParaView systems use generate-execute-correct loops; Chat2VIS supports conversational follow-up requests such as changing chart type, grouping, colors, labels, and font sizes; C5 updates all views in real time after a new question is asked with selected context; and WOC continuously estimates, smooths, and broadcasts motion states (Mallick et al., 2024, Maddigan et al., 2023, Liang et al., 2023, Yan et al., 2022).

Third, evaluation is domain-specific and cannot be reduced to a single scalar. The ParaView papers combine executability with image metrics such as SSIM, PSNR, and LPIPS; Chat2VIS exposes benchmark inconsistencies and ambiguity in chart matching; C5 evaluates both comprehension and answer quality under context augmentation; WOC reports computational performance and latency but no formal user study or perceptual evaluation (Peterka et al., 30 Jul 2025, Maddigan et al., 2023, Liang et al., 2023, Yan et al., 2022).

The limitations are correspondingly heterogeneous. The 2025 ChatVis system is tightly built around ParaView’s Python API, depends on the quality of documentation and examples, shows prompt sensitivity, incurs execution cost from pvpython in the loop, and does not perform deeper static program analysis. C5 currently focuses more on content and topic structure than on interaction patterns, is designed for small to medium-scale conversations, and relies heavily on manual context selection. Chat2VIS faces LLM non-determinism, code errors, ambiguous queries, and the absence of large-scale multilingual evaluation. WOC is constrained by monocular occlusion, pose-estimation errors, limited treatment of fine-grained finger and face expression, and the lack of detailed analysis of network delays or physical interactions between avatars (Peterka et al., 30 Jul 2025, Liang et al., 2023, Maddigan et al., 2023, Yan et al., 2022).

A plausible implication of the literature is that “ChatVis” is best understood not as a single system class but as a design pattern: use conversational input, conversational history, or conversational embodiment as the organizing interface for visualization, while pairing that interface with domain-specific grounding and iterative correction. In one branch this yields verified ParaView scripts; in another it yields multilingual chart refinement; in another it yields visual, human-in-the-loop context management for LLM sessions; and in another it yields spatially embodied online communication through animated avatars (Peterka et al., 30 Jul 2025, Maddigan et al., 2023, Liang et al., 2023, Yan et al., 2022).