PresentEval: Unified Evaluation Framework

• PresentEval is a unified evaluation framework that quantifies presentation video quality using both fact-based objective quizzes and simulated human-like VLM ratings.
• It employs dual VLM-driven paths by assessing content fidelity through MCQs and scoring visual clarity and audience comprehension via prompt-based segment evaluations.
• Benchmarking on the Doc2Present dataset indicates that PresentEval achieves performance near or surpassing human levels, offering scalable and reproducible insights.

PresentEval is a unified, automated evaluation framework for multimodal presentation video generation, introduced in the context of assessing systems such as PresentAgent. It systematically quantifies both the objective and subjective quality of generated presentation videos by leveraging advanced Vision-LLMs (VLMs) and carefully designed human-comparable metrics. By integrating fact-based quiz accuracy with human-aligned perceptual criteria, PresentEval addresses the unique challenges inherent in evaluating editable, time-synchronized audiovisual slide presentations derived from complex textual documents (Shi et al., 5 Jul 2025).

1. Architecture and Evaluation Workflow

PresentEval operates as the final assessment stage in the PresentAgent pipeline. Once a source document is transformed into a narrated presentation video—comprising a sequence of $K$ visual frames (“slides”) and a time-aligned transcript $T$—PresentEval evaluates the output with two parallel, VLM-driven paths:

• Objective Quiz Evaluation: All slides and the complete narration transcript are concatenated and provided to a large VLM (Qwen-VL-2.5-3B). The VLM answers a canonical set of five multiple-choice questions (MCQs) authored by humans to probe for content fidelity: topical recall, structural understanding, and key argument extraction.
• Subjective Scoring (Simulated Human Preference): Video and/or audio segments are scored by another large VLM (Qwen-Omni-7B) through segment-specific prompts that elicit ratings along three axes: content fidelity, visual clarity, and audience comprehension.

This dual-path design yields both a scalar fact-recall metric and fine-grained quality ratings, facilitating comparisons to both human performance and competing generation systems.

2. Evaluation Dimensions and Definitions

PresentEval formalizes presentation video quality across three complementary dimensions:

• Content Fidelity ($s_c$): Measures the accuracy, logical consistency, and faithful coverage of key facts, argument structure, and terminology. High scores require the absence of hallucination and coherent narrative flow.
• Visual Clarity ($s_v$): Reflects the typographical, layout, and stylistic quality of slides—legibility, balanced whitespace, consistent color palette, image usage, and overall readability.
• Audience Comprehension ($s_u$): Gauges the accessibility and logical structure of the presentation, measuring whether viewers (particularly lay audiences) can follow the material without confusion or contradiction.

These criteria are motivated by the need to distinguish between surface-level visual polish, information grounding, and true communicative efficacy.

3. Prompt-Based VLM Scoring Protocols

Subjective metrics are operationalized via prompt-based, segment-level VLM scoring. Each evaluation segment (one or two slides with corresponding transcript fragment) is scored in isolation along all three axes using fixed natural language prompts:

Dimension	Prompt Template
Content Fidelity	“How coherent and accurate is the content of this presentation? Does it faithfully reflect the source’s key points? Rate 1–5 and explain.”
Visual Clarity	“How would you rate the slide design in terms of layout, readability, and aesthetics? Rate 1–5 and explain.”
Audience Comprehension	“How easy is it for a viewer to understand and follow this presentation? Any confusing parts? Rate 1–5 and explain.”

The VLM produces structured answers beginning with “Score: X/5” and provides brief justifications, which are parsed for the numerical rating.

For the Objective Quiz, five MCQs per source are posed to the VLM. Each question has four options; selection of the reference answer yields a binary indicator $\delta_j\in\{0,1\}$ for each $j$.

4. Aggregation: Mathematical Formulation

Let $V$ be split into $N$ segments with segment-level scores $s^{(i)}_d \in \{1,2,3,4,5\}$ for each dimension $T$0 and segment $T$1. Define:

• Dimension Score: $T$2
• Overall Subjective Score: $T$3
• Quiz Score: $T$4, with normalized accuracy $T$5

These scores are aggregated over the video to produce the final evaluation record, which jointly captures objective and subjective performance.

5. Implementation Details and Datasets

PresentEval is deployed on the Doc2Present dataset, which consists of 30 real long-form documents (3,000–8,000 words) paired with corresponding human-authored slide+speech videos (5–10 slides, 1–2 minutes duration). The evaluation protocol is standardized:

• Objective Quiz: 5 MCQs per document, written by humans to probe nontrivial comprehension—encompassing topics such as argument structure and major claims.
• Subjective Scoring: 5–10 second video or audio segments evaluated by Qwen-Omni-7B, responses parsed for scores.
• Baselines: Human reference (expert-created videos), PresentAgent variants with multiple LLM backends (including GPT-4o, Claude-3.7), all evaluated identically.

6. Comparison to Human Reference and Diagnostic Insights

Empirical results benchmark automatic PresentAgent outputs and human presentations using PresentEval. Human expert videos achieve reference scores of accuracy = 0.56, $T$6 (video mean), and 4.80 (audio mean). Certain agent variants, such as PresentAgent+GPT-4o-Mini, achieve quiz accuracy above human reference (0.64) and subjective scores near or marginally above human levels ($T$7).

This suggests that highly capable multimodal agents may exceed human factual recall on MCQ-based diagnostics while matching (but not decisively exceeding) human-like overall communicative and aesthetic quality. A plausible implication is that present VLM-powered systems can simulate or even outperform human experts on criterion-based recall, but may still display subtle deficiencies on open-ended comprehension or multi-faceted layout tasks (Shi et al., 5 Jul 2025).

7. Worked Examples and Observed Failure Modes

Worked illustrations in PresentEval include the following:

• Objective Quiz: For a technical blog on iPhone features, a sample MCQ asks for the primary feature highlighted, offering four plausible distractors. VLM responses select the reference answer and include a textual rationale, providing both correctness indicators and qualitative interpretability.
• Subjective Scoring: For content fidelity, slides and narration are presented; the VLM scores the segment and explains omissions or inconsistencies (“Score: 4/5. The narration covers all major topics in order, but omits one minor detail about battery life.”). These outputs enable granular error-tracing.

PresentEval, by coupling structured objective checks and elicited VLM judgments, facilitates reproducible, scalable, and human-aligned evaluation for end-to-end presentation video generation systems. This methodological advance directly addresses the structural limitations of generic slide-deck metrics by explicitly incorporating content fidelity, narrative clarity, and audience-centered criteria into compositional scoring (Shi et al., 5 Jul 2025).