SlidesGen-Bench Evaluation Framework

Updated 1 July 2026

SlidesGen-Bench is a comprehensive evaluation framework that defines and measures slide generation quality using universal, quantifiable, and reliable metrics.
It operationalizes evaluation through source-grounded probes and audience-conditioned utility weights, ensuring objective and reproducible assessments.
The framework’s dynamic pipeline and domain-specific metrics provide actionable insights, guiding improvements in both factual grounding and content relevance.

SlidesGen-Bench is a comprehensive evaluation framework for assessing the quality of automated slide generation systems. It addresses the challenges of comparing systems that vary in generation paradigm, output modality, and intended use, integrating principles of universality, quantification, and reliability. Distinctively, it operationalizes source-grounded, audience-conditioned, and content-specific assessment, supporting both multi-paradigm and user-sensitive evaluation protocols (Yang et al., 14 Jan 2026, Chen et al., 17 Jun 2026).

1. Design Principles and Motivation

SlidesGen-Bench is motivated by three central principles:

Universality: The framework evaluates any slide generator—whether code-driven, template-based, or image-centric—by treating the slide deck as a rendered visual artifact. This enables cross-system comparability without bias toward the generation backend.
Quantification: All scoring is defined via reproducible, closed-form computational metrics or deterministic rules, eliminating fuzzy or subjective adjudication except where reference questions or rubrics are explicitly required.
Reliability: Metrics are validated against large-scale, human-annotated datasets (e.g., Slides-Align1.5k), ensuring that automated scores correlate strongly with expert and user preferences (Yang et al., 14 Jan 2026).

These principles underpin both the computational tractability and the empirical validity of the benchmark, establishing SlidesGen-Bench as a reference-standard for slide generation research.

2. Dataset Construction and Probe Engineering

The core asset is a diverse corpus of 113 topics, spanning 50 academic papers (across CV, NLP, systems, economics, environmental science) and 63 non-academic documents (policy, business reports, technical tutorials, climate/health/social materials). Each topic is aligned with one of seven presentation scenarios (e.g., academic talk, policy briefing, investor pitch) (Chen et al., 17 Jun 2026).

Probe Generation:

Each source yields an exhaustive bank of evidence-grounded probes $e_j$ $e_{j}$ , defined as 7-tuples:
- $q_j$ : atomic fact question
- $a_j$ : canonical answer
- $Z_j$ : exact source span (text/table/figure/chart)
- $d_j$ : depth level $\in \{1,2,3,4\}$
- $m_j$ : modality (text, table, chart, figure, etc.)
- $g_j$ : semantic domain $\in \{$ context, method, evidence, limitations, implementation, implications $\}$

Probes are generated by multiple LLM passes, merged via deduplication (by source span and paraphrase), and encoded with precise semantic and utility annotations.

Audience Profiling:

Three canonical audience archetypes: Specialist (technical rigor), Learner (intuitive/motivational), Decision Maker (actionable/implicational).
Each probe receives a utility weight $q_j$ 0 for every (audience, scene) pairing, as judged by a utility-assessing LLM.
Probes with $q_j$ 1 (typically $q_j$ 2) comprise the "audience-essential" probe set for the given configuration.

This structure allows controlled, audience-specific benchmarking without entangling difficulty with relevance.

3. Metrics and Quantitative Definitions

SlidesGen-Bench articulates four core metrics, each grounded in explicit formalism:

Metric	Formula/Methodology	Assesses
Audience Coverage ( $q_j$ 3)	$q_j$ 4	Fraction of utility-weighted probe content delivered
Domain-wise Coverage ( $q_j$ 5)	$q_j$ 6 for domain $q_j$ 7	Domain-specific targeting/failures
Efficiency ( $q_j$ 8)	$q_j$ 9; $a_j$ 0	Utility per slide or per minute–attention normalization
Correctness ( $a_j$ 1)	Weighted ratio of source-backed vs. unsupported atomic claims	Groundedness of slide claims
SafeEfficiency ( $a_j$ 2)	$a_j$ 3	Efficiency corrected for factual reliability

Where:

$a_j$ 4, with $a_j$ 5 if $a_j$ 6 is answered on the deck and source-grounded; $a_j$ 7 otherwise.
$a_j$ 8 aggregates total audience-essential utility in $a_j$ 9.
$Z_j$ 0 is the slide count; $Z_j$ 1 estimates total attention/time cost.
Correctness is assessed by LLM-based atomic claim extraction and verification against the source.

Metrics are computed for both aggregate and per-domain breakdowns, illuminating blind spots in system content selection or grounding.

4. Dynamic Evaluation Procedure

SlidesGen-Bench employs a dynamic, reweighting pipeline for metric computation:

Given source $Z_j$ 2, generated deck $Z_j$ 3, and audience/scene configuration, assign weights to all probes from the static bank via the UtilityJudge LLM module.
Select the set $Z_j$ 4 of audience-essential probes ( $Z_j$ 5).
For each $Z_j$ 6, determine $Z_j$ 7 iff the answer is present in $Z_j$ 8 and supported by $Z_j$ 9.
Accumulate $d_j$ 0 and derive all metrics (see above).
Extract all atomic slide claims and compute $d_j$ 1 using LLM-based claim extraction and source-verification.
Return metric vector $d_j$ 2.

Because weight vectors are updated in real time per audience and scenario, the same probe set supports dynamic, context-sensitive evaluation.

5. Experimental Protocols and Results

SlidesGen-Bench has been deployed to evaluate several classes of LLM-based slide generation systems, including DeepPresenter (agent pipeline), SlideTailor (preference- and retrieval-augmented), and NotebookLM (ablation, PDF-image output). Evaluation protocol enforces:

Two experimental setups: audience-agnostic vs. audience-conditioned prompts (scenes fixed, only audience switches).
Strict thresholding ( $d_j$ 3) retains only utility-weight 1.0 (essential) probes per audience.
Bootstrapped CIs for all major results.

Key findings at $d_j$ 4:

System	Condition	Audience	$d_j$ 5 (Audience Coverage)
DeepPresenter	Agnostic	Specialist	0.413
DeepPresenter	Conditioned	Learner	0.714
DeepPresenter	Conditioned	Decision Maker	0.654
SlideTailor	Agnostic	Learner	0.493
SlideTailor	Conditioned	Learner	0.594
NotebookLM	Agnostic	Decision Maker	0.853

DeepPresenter and SlideTailor achieve correctness Corr $d_j$ 6.
Visual richness and broad coverage (e.g., in NotebookLM) do not guarantee factual grounding.
Audience conditioning shifts content allocation toward the most relevant domains for each audience (e.g., implications for Decision Makers).
Efficiency and SafeEfficiency may diverge, penalizing visually compelling but ungrounded decks (Chen et al., 17 Jun 2026).

6. Interpretive Examples and Practical Recommendations

Concrete use cases quantify the audience-adaptivity of the protocol:

For a probe (e.g., “Explain why stressor layers are normalized using log(X+1)”), the deck only earns credit if the fact is visibly present and source-backed under the context of interest and if its probe utility passes the audience/scene threshold.
Slide-level scoring is entirely driven by the intersection of factual relevance, source support, and audience priority.

Recommended practices for future system development:

Model audience-conditioned utility explicitly during content selection (beyond appearance).
Implement source-grounding at the atomic claim level for every piece of extracted or generated slide content.
Optimize for high $d_j$ 7 on domains most valued by the target audience, rather than aggregate alone.
Use SafeEfficiency rather than raw coverage or visual appeal to avoid rewarding hallucinated, unsupported claims.
Extend evaluation to granular user profiles and richer modalities as domain demand expands (Chen et al., 17 Jun 2026).

7. Positioning Within the Slide Generation Benchmark Ecosystem

SlidesGen-Bench represents a rigorous, source- and audience-grounded approach to deck evaluation. Its explicit probe construction and utility-weighting per audience stand in contrast to visually focused protocols such as those in (Yang et al., 14 Jan 2026), and its domain-wise breakdown reveals targeting failures masked by global metrics. Integration wrappers permit conversion between SlidesGen-Bench and scenario-oriented frameworks (e.g., UniPPTBench’s UniPPTEval, which focuses on input regime diversity, scenario-specific metricity, and cross-source synthesis) (Zhao et al., 17 May 2026).

The framework advances the field beyond generic presentation quality, enforcing fidelity to both the source document and the audience’s cognitive and practical priorities. Its methodology guides both development and granular diagnosis for next-generation, audience-aware slide generation agents.