---
title: 'SlideAudit: Taxonomy & Dataset for Slide Evaluation'
url: https://www.emergentmind.com/topics/slideaudit
type: topic
---

# SlideAudit: Taxonomy & Dataset for Slide Evaluation

SlideAudit is a dataset and taxonomy for automated evaluation of presentation slides. It consists of 2,400 presentation-slide images annotated for concrete design flaws and was developed to support automated evaluation, and potentially remediation, of slide designs using large language models (LLMs). The work combines a balanced corpus of native and synthetically altered slides, a rigorously defined taxonomy of slide design flaws, a trained crowdsourcing protocol for annotation, and benchmark evaluations of multiple multimodal models under different prompting strategies. The central empirical finding is that slide design critique remains difficult for current AI systems, but taxonomy-grounded prompting improves both flaw detection and remediation quality [2508.03630].

## 1. Scope, composition, and construction

SlideAudit was assembled from three native slide collections that furnished 600 original slides: 200 from a government-deck corpus, 200 from a public Google Slides crawl, and 200 from AI-generated Gemini 2.0 outputs [2508.03630]. To introduce controlled design problems, each original slide was automatically altered in three distinct ways: within-object alignments, between-object layouts, and typography attributes. These transformations produced 1,800 synthetic variants, yielding 2,400 slides in total.

The alterations were rule-based and repeatable. The reported examples include shifting text alignment from left to center, resizing and overlapping elements to break grid structure, and varying font sizes inconsistently. Because all three alteration types were applied to every original slide, the resulting flaws were balanced and systematically reflected across the dataset.

This construction is notable because it combines naturalistic slide material with controlled perturbations. A plausible implication is that SlideAudit is designed not only to measure whether a model can recognize poor design in the wild, but also to test whether it can recover specific, intentionally introduced flaw types under consistent conditions. The dataset therefore occupies an intermediate position between purely observational corpora and purely synthetic benchmarks.

## 2. Taxonomy of slide design flaws

The SlideAudit taxonomy groups slide-level defects into five top-level dimensions and 27 fine-grained categories, of which 19 were targeted for evaluation [2508.03630]. The dimensions are Composition & Layout, Typography, Color, Imagery & Visualization, and Animation & Interaction. Animation & Interaction is explicitly out of current evaluation scope.

| Top-level dimension | Fine-grained categories described | Evaluation scope |
|---|---:|---|
| Composition & Layout | 6 | Included |
| Typography | 6 | Included |
| Color | 3 | Included |
| Imagery & Visualization | 4 | Included |
| Animation & Interaction | not specified in the summary list | Out of current evaluation scope |

Within Composition & Layout, the taxonomy includes Poor Visual Hierarchy, Cluttered Layout, Unbalanced Space Distribution or Gapping, Object Alignment Issues, Content Overflow/Cut-off, and Occluded Content. Several categories are distinguished by explicit decision rules. Poor Visual Hierarchy is selected only if neither font size, color contrast, nor spatial grouping makes the headline clearly dominant. Cluttered Layout is distinguished from Excessive Text Volume by scope: the former concerns overall element density, whereas the latter refers only to overly large bodies of text. Occluded Content is distinguished from Content Overflow by causal mechanism: overlap by another object rather than clipping at a slide or box boundary.

Typography includes Poor Text Hierarchy, Illegible Typeface Selection or Usage, Improper Font Sizing, Excessive Text Volume, Improper Text Styling, and Improper Line/Character Spacing. Color includes Insufficient Color Contrast for Readability, Excessive or Inconsistent Color Usage, and Inappropriate or Mismatched Color Combinations. Imagery & Visualization includes Irrelevant Visual Content, Poor Image Quality/Editing, Improper Image Sizing, and Inconsistent Visual Style Usage, the last of which is described as very rare.

The taxonomy is not merely a label set; it is a decision framework. The inclusion of category definitions and disambiguation rules indicates an attempt to reduce ambiguity in inherently subjective design judgments. This suggests that SlideAudit treats slide critique as a structured classification problem rather than an open-ended aesthetic assessment.

## 3. Annotation protocol and consensus formation

The annotation pipeline used crowdsourcing on Prolific with strict qualification and quality control procedures [2508.03630]. Participants first completed a custom Qualtrics tutorial and a 10-slide quiz covering taxonomy definitions and bounding-box drawing. Access to the main task required answering all items correctly.

In the main task, 145 qualified workers each annotated 50 slides, with three annotations collected per slide. For nine spatially localizable categories, such as misaligned elements, annotators drew bounding boxes. For the remaining categories, they selected category labels. An LLM-based filter flagged submissions with no labels or over-labelling, and six participants were manually rejected.

Categorical labels were aggregated by majority vote, defined as at least two of three annotators agreeing. “Strong agreement” was defined as all three agreeing. Fleiss’ $\kappa$ for the three-way annotation was 0.26, characterized as fair agreement and stated to be consistent with related UI critique work. Bounding boxes were retained only when at least two annotators’ boxes overlapped.

These design choices clarify the intended granularity of supervision. SlideAudit provides both categorical and spatially grounded annotations, but only for categories that are localizable in the slide image. The reported agreement level also serves as an important caveat. Slide design evaluation is presented as a task with nontrivial annotator variance, which argues against interpreting the benchmark as a straightforward object-recognition problem.

## 4. Benchmarking AI-based flaw detection

The evaluation compared GPT-4o (gpt-4o-2024-08-06), Claude 3.7 Sonnet, and Gemini 2.0 Flash under two input modes: raw slide image alone, and image plus structured object descriptions containing IDs, types, and coordinates [2508.03630]. Four prompting strategies were tested:

1. Baseline (Zero-Shot), with minimal instruction and no taxonomy.
2. High-Level Category, listing the four top-level dimensions.
3. Full Taxonomy, including all 19 category definitions.
4. Taxonomy + Computational Augmentation, adding saliency maps, computational Gestalt groupings, and color-contrast metrics.

The evaluation metrics were defined as:
$$
\mathrm{Precision}=\frac{TP}{TP+FP}
$$

$$
\mathrm{Recall}=\frac{TP}{TP+FN}
$$

$$
F1=\frac{2\times(\mathrm{Precision}\times\mathrm{Recall})}{\mathrm{Precision}+\mathrm{Recall}}
$$

$$
\mathrm{Macro\text{-}F1}=\frac{1}{C}\sum_{i=1}^{C} F1_i
$$

with equal averaging over $C=19$ categories.

Performance varied substantially by prompting regime. F1 ranged from 0.331, obtained by prompting with 70 generic “CrowdCrit” statements, to 0.655 for GPT-4o with object descriptions and the full taxonomy. Baseline zero-shot F1 was approximately 0.48–0.52 across models. Adding taxonomy definitions increased F1 by +0.073 to +0.179, with the largest gain reported for GPT-4o. By contrast, computational augmentation yielded marginal or negative gains; the reported explanation is that over-focusing on saliency and contrast led to false positives.

A central result is therefore that explicit design knowledge encoded as taxonomy definitions improves model performance more reliably than auxiliary computational signals. This suggests that the bottleneck is not only visual perception but category calibration: the models benefit when the decision boundaries among slide flaw types are specified in detail.

## 5. Remediation and fix-plan generation

SlideAudit also includes a remediation study assessing whether AI can improve slides rather than merely critique them [2508.03630]. From the 200-slide evaluation subset, 50 slides were randomly selected. For each slide, GPT-4o generated two step-by-step fix plans: one from zero-shot diagnostics and one from taxonomy-informed diagnostics.

Three blind expert raters, described as HCI/design master’s students, evaluated each plan along several axes. At the step level, they assessed flaw accuracy and executability. At the holistic level, they answered whether the plan improved the slide and recorded plan preference.

The quantitative outcomes favored taxonomy-guided plans. Of the 50 slides, 82.0% (41/50) were judged improved by at least one plan. Among those improved slides, taxonomy-guided plans were preferred in 87.8% (36/41). Flaw-identification accuracy rose from 28.9% in the baseline condition to 59.0% with taxonomy guidance. False positives decreased from 34.6% to 9.6%, and non-critical detections from 13.8% to 7.2%. The implementable solution rate increased from 27.7% to 63.9%, while unnecessary or generic suggestions fell sharply.

Inter-rater agreement was also reported. Step-level Fleiss’ $\kappa$ was 0.57 for flaw accuracy and 0.52 for executability, while holistic preference $\kappa$ was 0.61. No formal $p$-values were reported.

These results indicate that the taxonomy affects not only classification scores but also downstream actionability. The evidence presented is that better flaw specification produces more executable repair plans and fewer generic interventions. A plausible implication is that taxonomic structure functions as a control interface for remediation, constraining the model toward concrete, slide-specific edits.

## 6. Limitations, ambiguities, and research directions

The work explicitly frames slide design as highly subjective and context-dependent [2508.03630]. That characterization is consistent with both the taxonomy design and the annotation statistics. A common misconception would be to treat low-to-moderate benchmark scores as evidence that slide critique is a trivial vision-language task that current LLMs should already solve; the reported results argue otherwise. Even with structured prompts and object descriptions, the best F1 was 0.655, and zero-shot performance remained around 0.48–0.52.

Several limitations are stated directly. LLMs tend to over-interpret extraneous signals, generating false positives. Spatial grounding remains weak, with mean IoU approximately 0.25. Slide-to-slide consistency, transitions, and deck-wide coherence are outside the current scope, as are interactive elements and animations in the evaluation setup. The category Inconsistent Visual Style Usage is also described as very rare, indicating uneven natural prevalence among flaw types even in a balanced benchmark construction.

The proposed extensions follow from these constraints. The paper suggests developing specialized multimodal models trained on SlideAudit for performance greater than 0.7 F1, personalizing critique by learning user preferences through few-shot examples, scaling annotations via semi-automated workflows or active learning, expanding evaluation to interactive elements, animations, and full-deck assessment, and integrating taxonomy-grounded critique into assistive authoring tools. The stated accessibility motivation is especially specific: such tools could empower blind and low-vision users to create visually effective slides independently.

Taken together, these directions position SlideAudit as infrastructure rather than a complete solution. The benchmark establishes a controlled vocabulary, an annotation regime, and empirical baselines for automated slide evaluation. It also highlights the unresolved tension between the formalization required for machine evaluation and the contextual variability inherent in graphic design judgment.

Source: https://www.emergentmind.com/topics/slideaudit