VisDocSketcher: Code-to-Diagram System

• VisDocSketcher is a code-to-diagram system that uses static analysis and agentic LLMs to convert Jupyter notebook workflows into visual Mermaid diagrams.
• It employs a multi-agent architecture that decomposes tasks into code analysis, diagram synthesis, syntax repair, and visual enrichment to ensure clear and accurate representations.
• The system’s AutoSketchEval framework evaluates diagram quality by reconstructing code and comparing similarity metrics, ensuring high semantic alignment with the source code.

VisDocSketcher is an agent-based system for automatically generating visual documentation from source code, introduced for Jupyter notebooks in the data science domain. It combines static analysis with agentic LLM systems to extract dataflow and workflow structure, convert that structure into Mermaid flowchart diagrams, and evaluate the resulting diagrams with a code-centric framework called AutoSketchEval. In the reported experiments, it is presented as the first agent-based approach that combines static analysis with LLM agents to identify key elements in the code and produce corresponding visual representations, and as the first exploration of using agentic LLM systems to automatically generate visual documentation (Gomes et al., 15 Sep 2025).

1. Definition, scope, and motivation

VisDocSketcher targets the longstanding gap between developers’ mental models of software systems and the actual structure embodied in code. The system is motivated by the observation that developers rely heavily on visual representations—whiteboard sketches and diagrams—for understanding code, onboarding, and collaboration, whereas visual artifacts are typically manual, quickly outdated, and rarely maintained. Within data science workflows, the problem is particularly acute because Jupyter notebooks interleave code, markdown, and outputs across many cells, producing pipelines that are difficult to apprehend at a glance (Gomes et al., 15 Sep 2025).

The system operates on the premise that visual documentation provides a higher-level understanding of system structure and data flow than textual documentation. Its outputs are not formal reverse-engineering diagrams in a UML sense; rather, they are informal, sketch-style workflow visualizations rendered as Mermaid flowcharts. These diagrams emphasize dataflow and logical stages such as data loading, preprocessing, visualization, modeling, and evaluation, with directed edges representing data or logical dependencies between nodes (Gomes et al., 15 Sep 2025).

A central contribution is that the problem is treated not only as generation but also as evaluation. The source material characterizes visual documentation as both difficult to produce and challenging to evaluate, since different diagrams can represent the same code correctly and human judgment is subjective, costly, and difficult to scale. VisDocSketcher addresses the generation side, while AutoSketchEval supplies a ground-truth-free evaluation strategy based on code reconstruction and code-level similarity metrics (Gomes et al., 15 Sep 2025).

2. System architecture and agent roles

VisDocSketcher is described in two variants: a single-agent system and a multi-agent system. The single-agent version receives the entire Jupyter notebook content and prompts a single LLM agent to generate Mermaid flowchart syntax focusing on major workflow stages. This version has no tool access and performs all understanding and visualization within one prompt (Gomes et al., 15 Sep 2025).

The multi-agent variant is the core design. It is orchestrated with LangGraph and uses multiple GPT-4o-mini agents, each assigned a specialized role. The system architecture is organized around a Supervisor Agent that coordinates the pipeline, checks intermediate artifacts with a file_exists tool, and can re-run or skip stages depending on artifact availability. The remaining agents decompose the task into code understanding, sketch synthesis, syntax repair, and visual enrichment (Gomes et al., 15 Sep 2025).

Agent	Role	Tools or output
Supervisor Agent	Orchestrates the pipeline	file_exists, stage control
Analyser Agent	Code understanding via static analysis	extract_notebook_cells, build_dataflow_graph_ast, extract_variable_dependencies
Sketcher Agent	Converts analysis into Mermaid flowchart code	Mermaid output
Repair Agent	Ensures Mermaid syntax validity	validate_mermaid_from_file
Visuals Agent	Enhances visual expressiveness without changing structure	Enriched Mermaid file

The Analyser Agent is the principal source of code-grounded structure. Its responsibilities include identifying data sources, segmenting the notebook into logical sections, mapping dataflow across cells, and detecting ML components such as model definitions, training routines, and evaluation steps. Its output is an intermediate analysis artifact, such as JSON or text, describing the workflow graph. The Sketcher Agent consumes this artifact and produces an initial Mermaid diagram containing nodes for workflow stages, data artifacts, and models, together with directed edges for dependencies and process steps (Gomes et al., 15 Sep 2025).

The Repair Agent then validates and fixes Mermaid syntax errors, including malformed syntax and invalid characters, so that the diagram can be rendered. Finally, the Visuals Agent augments labels with emojis and icons, applies color schemes based on node types, and makes the diagrams more sketch-like and intuitive while preserving the underlying structure. This staged decomposition suggests a deliberate separation between semantic extraction, structural rendering, syntactic correctness, and stylistic enhancement (Gomes et al., 15 Sep 2025).

3. Static analysis, representation, and generation pipeline

VisDocSketcher converts notebook code into Mermaid flowchart diagrams that can be rendered through the Mermaid engine. Conceptually, each diagram is treated as a directed graph $G = (V, E)$, where $V$ denotes nodes representing notebook stages, data sources, models, and related artifacts, and $E$ denotes edges representing data or logical dependencies (Gomes et al., 15 Sep 2025).

The generation pipeline follows a fixed orchestration pattern. First, the Supervisor receives the notebook path and invokes the Analyser. The Analyser parses notebook cells, builds a dataflow graph through AST-based static analysis, and extracts variable dependencies across cells. It then writes a structured analysis artifact to the filesystem. Second, the Supervisor invokes the Sketcher, which converts the analysis artifact into Mermaid flowchart code. Third, the Repair Agent validates the Mermaid specification and repairs syntax when necessary. Fourth, the Visuals Agent adds aesthetic enrichments such as emojis and color cues. The Supervisor may loop over stages if artifacts are missing or validation fails (Gomes et al., 15 Sep 2025).

The use of static analysis is the main mechanism for ensuring alignment with actual code structure rather than relying purely on generative inference. The description emphasizes AST-based dataflow graphs and variable dependency extraction as a way to tie diagrams to actual code relationships. This makes the generated diagrams reflect actual dataflow in the code, distill long notebooks into high-level pipeline representations, and reduce open-ended hallucination during diagram synthesis (Gomes et al., 15 Sep 2025).

In the single-agent baseline, by contrast, there is no such explicit static-analysis-mediated intermediate artifact. The single LLM is prompted as an “expert software visualization system specialized in Python-based data science workflow” and asked to produce a high-level informal sketch directly in Mermaid syntax. This simpler path serves as a direct comparison point for assessing the benefits of decomposition and tool use (Gomes et al., 15 Sep 2025).

4. AutoSketchEval and code-centric evaluation

AutoSketchEval is the evaluation framework paired with VisDocSketcher. It is described as a novel framework for assessing generated visual documentation using code-level metrics, without requiring ground-truth sketches. Its design follows an autoencoder-inspired analogy in which the Code2Sketch system acts as an encoder, the generated Mermaid diagram acts as a latent representation, and a pre-existing Sketch2Code model acts as a decoder that reconstructs code from the diagram (Gomes et al., 15 Sep 2025).

The evaluation process has four stages. First, VisDocSketcher generates a diagram from a notebook. Second, a Sketch2Code model converts the diagram back into code. Third, the reconstructed code is compared with the original notebook using code similarity metrics. Fourth, these similarity scores are interpreted as a sketch quality score. The source material denotes the original code as $C$, the generated sketch as $D = E(C)$, and the reconstructed code as $\hat{C} = Dec(D)$, with similarity $S(C, \hat{C})$ computed by metrics such as CodeBLEU or CodeBERTScore (Gomes et al., 15 Sep 2025).

Two metric families are used. CodeBLEU is employed both in its full form and in a dataflow-only form. The full metric is described as a weighted combination of n-gram match, weighted n-gram match, syntax match, and dataflow match: $$\text{CodeBLEU} = \alpha \cdot S_{ng} + \beta \cdot S_{wng} + \gamma \cdot S_{syn} + \delta \cdot S_{df}.$$ CodeBLEU-dataflow isolates the dataflow component to emphasize high-level semantic similarity relevant to workflows. CodeBERTScore supplies embedding-based similarity via precision, recall, and F1 computed over CodeBERT embeddings (Gomes et al., 15 Sep 2025).

The framework is validated by comparing aligned notebook-diagram pairs against mismatched pairs. In this binary discrimination setting, all metrics show significant separation with Mann–Whitney U and KS tests at $p < 0.001$, and CodeBLEU and CodeBLEU-dataflow also show significantly different variances under Levene’s test. The reported AUC values are 1.0 for CodeBERTScore F1 and Precision, 0.978 for CodeBLEU-full, and 0.871 for CodeBLEU-dataflow, with Cliff’s Delta at least 0.74 for all metrics. These results are presented as evidence that AutoSketchEval can reliably distinguish high-quality, code-aligned visual documentation from low-quality, non-aligned outputs (Gomes et al., 15 Sep 2025).

This evaluation design is notable because it does not score diagram aesthetics directly. Instead, it treats diagram quality as recoverability of code semantics through a round-trip reconstruction process. A plausible implication is that VisDocSketcher prioritizes code alignment over purely visual appeal, even though the Visuals Agent later enriches the rendered diagrams (Gomes et al., 15 Sep 2025).

5. Experimental results, performance, and trade-offs

The reported experiments use two datasets. The “Visual Code Assistants Artifacts” dataset contains human-authored diagrams of ML workflows paired with their intended Jupyter notebooks and is used both for validating AutoSketchEval and for comparing VisDocSketcher to a template-based baseline. The DistillKaggle dataset contributes a curated subset of 1,000 Kaggle notebooks and is used to study scalability and complexity effects in the absence of ground-truth sketches (Gomes et al., 15 Sep 2025).

Against the template-based baseline, VisDocSketcher improves all reported evaluation metrics. At the 5th percentile, CodeBERTScore-F1 rises from 0.633 to 0.753, Precision from 0.725 to 0.784, CodeBLEU from 0.067 to 0.177, and Dataflow from 0.026 to 0.125; the average across metrics rises from 0.363 to 0.460. On mean performance, F1 rises from 0.690 to 0.836, Precision from 0.774 to 0.863, CodeBLEU from 0.132 to 0.333, and Dataflow from 0.103 to 0.342, with the average across metrics increasing from 0.425 to 0.594. These changes are summarized as a 39.8% mean semantic-accuracy improvement and a 26.7% 5th-percentile improvement over the baseline (Gomes et al., 15 Sep 2025).

The multi-agent and single-agent variants exhibit a clear quality–efficiency trade-off. The multi-agent system outperforms the single-agent system in 59.3% of cases with $p = 0.0335$, but the computational cost is substantially higher. Mean runtime per notebook is 8.27 seconds for the single-agent system and 93.43 seconds for the multi-agent system, making the multi-agent pipeline approximately 11.3 times slower (Gomes et al., 15 Sep 2025).

On the DistillKaggle subset, the single-agent system generates 999 Mermaid files, of which 849 render successfully, corresponding to 84.9% valid diagrams. The multi-agent system produces 996 JSON analysis reports, 990 initial sketches, 863 repaired sketches, and 920 enriched outputs, with 74.4% of the total sketches rendering successfully without errors. The paper explicitly states that the approach can valid visual documentation for 74.4% of the samples. This lower rendering-success rate, despite higher semantic quality, is attributed to the greater number of stages and therefore more possible failure points in the multi-agent pipeline (Gomes et al., 15 Sep 2025).

These findings support a narrow but important distinction: the single-agent path is more efficient and has a higher raw rendering-success rate on the large-scale corpus, whereas the multi-agent path yields diagrams of higher code-aligned quality when successful. This suggests that validity of rendering and validity of semantic alignment are separate axes of performance within the system (Gomes et al., 15 Sep 2025).

6. Complexity effects, reliability, and limitations

The paper studies how notebook complexity affects the quality of generated visual documentation by regressing CodeBLEU-dataflow on notebook characteristics such as lines of code, number of code cells, author performance tier, and system type. The general model achieves $V$0, and the interaction model achieves $V$1. The reported significant coefficients are approximately $V$2 for lines of code, $V$3 for number of code cells, and $V$4 for the highest performance tier, all with the stated significance levels (Gomes et al., 15 Sep 2025).

The interpretation given in the source material is that every additional 100 lines of code correspond to roughly a 0.08 decrease in Dataflow score, and every additional 10 code cells correspond to a modest decline in score. More experienced developers, operationalized as performance tier 5, produce code that is about 15.5 percentage points harder to visualize than lower-tier notebooks. Interaction terms with the multi-agent indicator are not significant, indicating that the negative effect of notebook complexity is similar for both architectures (Gomes et al., 15 Sep 2025).

Several mechanisms are identified as safeguards against hallucination and inconsistency. Static analysis constrains the LLMs by providing explicit structural facts. Intermediate artifacts separate semantic extraction from rendering. The Repair Agent removes Mermaid syntax errors. AutoSketchEval can function as a feedback mechanism by identifying low-quality or misaligned diagrams through poor reconstruction scores. The system therefore does not rely on a single generative step; it layers multiple forms of verification and decomposition (Gomes et al., 15 Sep 2025).

The paper also enumerates important limitations. The experiments are confined to data science Jupyter notebooks, so the results do not directly establish effectiveness for software architecture documentation or large distributed systems. The evaluation metrics capture code-level alignment but not human factors such as layout clarity, aesthetics, or explanatory usefulness. There is no new large-scale user study of how the generated diagrams affect onboarding or code comprehension. Finally, quality degrades with code length and notebook complexity, which is attributed in part to long-context LLM limitations (Gomes et al., 15 Sep 2025).

7. Position in the broader research landscape

VisDocSketcher sits at the intersection of automated documentation, code visualization, multi-agent LLM systems, and visual reasoning. Its immediate novelty is code-to-sketch generation for software workflows, but the supplied research context places it alongside several adjacent notions of “sketching” in machine learning and HCI (Gomes et al., 15 Sep 2025).

In software engineering and documentation, its closest conceptual relative is “Automatic Document Sketching,” which formalizes document sketching as generating draft documents from analogous texts and uses editing-effort metrics such as $V$5 to assess sketch quality (Wu et al., 2021). The relation is methodological rather than representational: document sketching there produces draft text, while VisDocSketcher produces Mermaid-based visual documentation. This suggests that both systems treat “sketching” as an intermediate artifact intended to reduce human effort, though in different modalities (Wu et al., 2021).

In visualization synthesis, “Visualization by Example” presents a different but related paradigm: given raw data and a small visual sketch, it synthesizes both a data-wrangling program and a visualization program such that the full visualization is consistent with the sketch (Wang et al., 2019). A plausible implication is that VisDocSketcher can be read as a code-centric analogue of visualization-by-example: rather than inferring plots from a sketch over data points, it infers workflow diagrams from notebook structure and evaluates them by reconstructing code semantics (Wang et al., 2019).

The supplied literature also uses “VisDocSketcher” more loosely as a conceptual extension in adjacent multimodal settings. In “VideoSketcher,” the term is used to denote a system that would document processes as sequential sketches generated from LLM planning and text-to-video diffusion, with extensions such as brush style conditioning and autoregressive co-drawing (Ren et al., 17 Feb 2026). In “DeepSketcher,” it is used as a blueprint for systems that sketch or annotate visual documents while reasoning in visual embedding space, combining textual chain-of-thought with internal visual edits (Zhang et al., 30 Sep 2025). These uses do not redefine the software-engineering system introduced in (Gomes et al., 15 Sep 2025), but they indicate that the name has begun to function as a broader label for sketch-based visual documentation and reasoning.

Within that broader landscape, the specific contribution of VisDocSketcher remains sharply defined: it is an agentic code-to-diagram system for Jupyter notebooks, grounded in static analysis, implemented through Mermaid flowcharts, and evaluated through code reconstruction and code-level similarity metrics (Gomes et al., 15 Sep 2025).