Visualizationary: Automated Design Feedback

• Visualizationary is an automated LLM-driven system that provides iterative design feedback for visualization designers using perceptual filters and structured guidelines.
• It leverages an ACGT workflow to analyze, clarify, guide, and track design iterations, aiding both novice and expert users.
• The system enables longitudinal design improvements through actionable feedback and empirical revision tracking based on computed visualization metrics.

Visualizationary is an automated, LLM-driven system designed to provide actionable, iterative design feedback for visualization designers. The system leverages a combination of perceptual image filters, structured design guidelines, prompt engineering, and revision tracking to enhance the quality of user-authored visualizations, ranging from novice output to expert practices. Visualizationary exemplifies how off-the-shelf LLMs, such as ChatGPT, can be systematically harnessed for critical, stepwise critique and guidance on visual communication, without requiring designers to write code or navigate complex frameworks (Shin et al., 2024).

1. System Architecture and ACGT Workflow

Visualizationary implements an ACGT (Analyze, Clarify, Guide, Track) pipeline. Designers interact with the system primarily through a web client, uploading a screenshot of their visualization (PNG, JPG). The backend processes the image with a suite of image-based "visualization filters" to extract symbolic perceptual and structural metrics. These metrics, alongside a curated preamble of visualization design guidelines, are composed into prompt templates for the LLM (gpt-3.5-turbo) to interpret and provide feedback.

The ACGT pipeline's stages are:

• Input: A screenshot is uploaded.
• Analyze: "Visualization filters" compute salience, color, accessibility, textual, and representation metrics (§4.4).
• Clarify & Guide: The LLM is conditioned with metrics, a guideline preamble, and templated prompts for producing clarifications (interpretation) and suggestions (design improvement).
• Track: An iteration tracker summarizes how metrics and interpretations evolve between versions, functioning as a revision logbook.
• Output: Results are presented as an interactive, hierarchical report, navigable by metric, issue, and revision (§4).

This workflow supports longitudinal design improvement and systematic feedback, leveraging automation for both perception-driven analysis and text-based critique.

2. Perceptual Filters and Metric Extraction

The system's feedback is grounded in computed metrics derived from five principal categories of visualization filters:

• Salience: Employing virtual eyetracker models to generate a pixel-level saliency map $S(x,y)$. Core metrics include:
  • SalText: Proportion of salience on detected text boxes, with “overly salient to text” flagged if $\mathrm{SalText} > p_{90}$ (90th percentile reference).
  • SalCenter: Proportion of salience within a central region $R_c$ (rectangular, centered, occupying $1/3$ width and height), with excessive central focus similarly thresholded.
  • Low-level stimulus: Analysis of edge zones via RGB gradients to determine “scarcely salient” areas if below the 10th percentile of coverage.
• Textual Characters: Title detection uses vision-LLMs like DePlot, while general text is located using OCR (PyTesseract). Absence or overabundance of text are directly quantified.
• Visual Representation: Recommendations for optimal chart type are informed by data extracted via DePlot and cross-referenced with Mackinlay's perceptual ranking. Chartjunk detection utilizes object detection (YOLOR).
• Color Perception: Dominant color palette extraction by clustering, with color similarity computed as the Euclidean RGB distance $$\Delta E_{ij} = \sqrt{(R_i - R_j)^2 + (G_i - G_j)^2 + (B_i - B_j)^2}$$. Groups with $\Delta E < \epsilon$ are flagged for insufficient contrast.
• Accessibility: The system simulates multiple color-vision deficiencies (deuteranopia, protanopia, tritanopia) and computes entropy loss $H_{orig} - H_{sim}$ (Shannon entropy) to alert when perceptual information is lost past a threshold.

Each metric is reported to the LLM as input for analysis and feedback generation.

3. LLM Prompt Engineering and Feedback Generation

Visualizationary conditions the LLM through a prompt preamble embedding domain-relevant guidance. This includes:

• Visual Encoding Rankings: Mackinlay’s APT hierarchy (position > length > angle > color).
• Perceptual Rules: Tufte, Munzner, Franconeri et al. guidelines emphasizing avoidance of chartjunk and cognitive efficiency.
• Color Guidelines: Recommendations for limited, distinct palettes (Healey, Szafir), including accessibility best practices (Chartability, Angerbauer et al.).
• Accessibility Heuristics: Explicit mention of color‐vision strategies.

Two main prompt templates structure interactions:

• T_acg: Analyze–Clarify–Guide. Prompts specify the metric and expected interpretative pathway (e.g., “Analyze visual salience on text. Provide interpretation in 2 sentences.”).
• T_t: Track. Contrasts current and prior outputs/interpretations (“What changed in visual salience between current and previous versions?”).

LLM responses are constrained (e.g., “2 sentences”), and reserved keywords or labels are mandated to ensure parsing consistency and reduce hallucinations. The result is a targeted, actionable, and structured natural language report.

4. User Study and Evaluation

A longitudinal user study was conducted with 13 visualization designers, partitioned by expertise (6 novices, 4 intermediates, 3 experts). Participants authored novel visualizations with at least five revisions over 3–5 days, utilizing Visualizationary’s feedback to inform their editing workflow. Key study elements included a pre-study briefing, iterative design episodes, and a post-study interview analyzing ACGT module impact.

Quantitative expert assessment scored improvement from version 1 to final on a 1-5 Likert scale; the mean score was 3.69 (σ=0.83), indicating positive improvement across all skill levels (novice: 3.34, intermediate: 3.92, expert: 4.11). Qualitative findings highlighted that:

• Analyze surfaced both obvious and overlooked issues.
• Clarify reports functioned as organizers for the revision process.
• Guide output was effective for simple, direct fixes but occasionally too abstract for novices.
• Track provided actionable feedback for monitoring perceptual changes (e.g., reduced central salience).

Designers preferentially applied feedback that they both understood and expected would yield tangible benefit, particularly adopting easily-implementable changes. Experts valued the system’s ability to consistently surface subtle design flaws (Shin et al., 2024).

5. Practical Implications, Recommendations, and Limitations

Visualizationary’s approach codifies several recommendations for tool builders:

• Modularize filters for scalable image-to-metric pipelines.
• Consistently inject domain knowledge into LLM prompts.
• Use structured templates to regularize LLM interaction and limit output entropy.
• Longitudinal tracking is essential for verifying feedback impact.

Best practices for prompt engineering include strict use of reserved labels, constraint of output length, and highly specific queries.

For designers, LLM-generated feedback should be treated as guidance; translation to concrete design edits requires judgment, especially for abstract or high-level recommendations. The track module enables empirical validation of perceptual effects.

Limitations include:

• LLM hallucinations may cause contradictory or spurious advice; planned mitigations involve self-verification methods.
• Privacy concerns with third-party APIs, motivating future deployment of private/open-source models.
• Novice users may struggle to operationalize abstract feedback, suggesting the need for finer-grained, directly actionable suggestions.

Future directions emphasize expansion of perceptual filters (including Gestalt groupings and data-ink metrics), the adoption of text-to-image models for richer feedback, and adaptive scaffolding that adjusts guidance complexity to user expertise (Shin et al., 2024).

6. Significance within Automated Visualization Critique and Related Systems

Visualizationary demonstrates that domain-constrained, filter-augmented LLMs can provide meaningful, custom-tailored feedback to visualization designers, even using "vanilla" general-purpose LLMs. It bridges the gap between code-free visualization tool adoption and expert critique, functioning as an interactive, extensible design critic. This model contrasts with curated example repositories (e.g., DaVE (Koenen et al., 2024)) and algorithmic evaluators by providing dialogic, contextual guidance and revision tracking. Its design is illustrative for future work on integrating LLMs with perceptual pipelines to automate expert-level feedback in visual analytics and communication contexts.