PPT-Eval: Benchmark for PowerPoint Agents

• The paper introduces PPT-Eval, a benchmark that evaluates computer-use agents on realistic PowerPoint deck creation and editing tasks using a detailed rubric system.
• It employs a structured, partial-credit rubric for multimodal tasks, assessing content creation, layout manipulation, and aesthetic quality across 120 diverse tasks.
• Empirical results highlight that API-based approaches outperform GUI agents, underlining challenges in achieving human-level performance in complex slide-editing tasks.

PPT-Eval is a PowerPoint-specific benchmark for evaluating computer-use agents on realistic presentation creation and editing tasks in the GUI-based web version of Microsoft PowerPoint, namely PowerPoint Online. It combines a benchmark of 120 tasks across 12 PowerPoint files with a rubric-based evaluation framework designed for multimodal, open-ended tasks in which binary success criteria are often too coarse. The benchmark emphasizes both content creation and deck editing, organizes tasks by difficulty, and scores outputs with partial credit, penalties for unnecessary changes and poor aesthetics, and natural-language feedback. In the reported meta-evaluation, its rubric judgments achieve a Kendall’s $\tau_b$ correlation of $0.77$ with human judgments, while frontier GUI agents remain substantially below human performance (Gandhi et al., 30 Jun 2026).

1. Scope and motivating problem

PPT-Eval is motivated by the claim that slide creation and editing is a rich, multimodal activity that is ubiquitous in professional and educational settings, and therefore an informative testbed for real-world computer-use agents. The benchmark treats PowerPoint work as a mixture of semantic understanding, visual reasoning, layout manipulation, tool use, and task planning. It uses PowerPoint Online rather than a narrow file API, so that agents can interact with the same feature surface available to human users in a browser, including design tools, advanced graphics, SmartArt, themes, advanced layouts, transitions, animations, hyperlinking, and other rich formatting operations (Gandhi et al., 30 Jun 2026).

The benchmark contains 120 tasks drawn from 12 PowerPoint files and spanning 404 unique slides in total, with 10 tasks per file. Each task consists of a natural-language goal, a .pptx file to modify, and a structured rubric. The design is method-agnostic: an agent may solve tasks by GUI interaction or by APIs and CLI tooling, because evaluation depends on the original file and the modified output file rather than on the exact action trace (Gandhi et al., 30 Jun 2026).

PPT-Eval is positioned as filling a gap between broad computer-use benchmarks and PowerPoint-specific but API-constrained evaluations. General computer-use suites such as OSWorld and WindowsAgentArena do not deeply benchmark realistic PowerPoint editing, while OfficeBench excludes PowerPoint-specific presentation manipulation. PPTC evaluates PowerPoint through python-pptx-style APIs, and slide-generation benchmarks such as SlidesBench emphasize from-scratch generation rather than realistic deck editing in a live GUI (Guo et al., 2023).

2. Benchmark construction and task design

The 12 source decks were selected from the Internet Archive and are openly licensed. Their topics are intentionally diverse, including Medicine, Computer Science, Accounting, Life Sciences, History, Aerospace, Architecture, Social Science, Education, and Environmental Science. The benchmark description emphasizes variation in visual style, topic, and slide structure, including images, tables, animations, non-standard layouts, and shapes (Gandhi et al., 30 Jun 2026).

Task curation followed a semi-automatic pipeline. The authors first generated 471 candidate tasks across the 12 files using Claude-4-Sonnet as a task-proposal agent. The agent explored each file for 35 steps per file and was given an augmented action space containing the tool add_tasks_to_dataset(tasks: list[str]). Six human annotators then filtered the 471 candidates down to 10 final tasks per file, refining them for clarity, usefulness, feasibility, automatic gradability, diversity, and balanced difficulty. More than 31% of the final 120 tasks were rephrased during human refinement, with stated reasons including clarifying ambiguous wording, increasing task scope or difficulty, making infeasible tasks feasible in PowerPoint Online, and correcting inaccuracies in the task statement (Gandhi et al., 30 Jun 2026).

Tasks are stratified into three difficulty levels. Easy tasks typically require at most 5 steps or at most 1 minute for a human; medium tasks require about 5–10 steps or 2–5 minutes; hard tasks require at least 10 steps or at least 5 minutes. The released benchmark contains 51 easy tasks, 39 medium tasks, and 30 hard tasks (Gandhi et al., 30 Jun 2026).

The task set mixes content-creation and editing or transformation scenarios. Reported examples include adding text boxes, bullets, tables, SmartArt, diagrams, and new slides; changing titles or wording throughout a deck; replacing images; recoloring or reformatting content; sorting table data; applying themes; adding transitions or animations; hyperlinking text; adjusting slide numbers; and rearranging content into multi-column layouts. Appendix examples include instructions such as “Change the background color of slide 3 to light blue,” “Center align the title text on slide 4,” “Sort the table data on slide 10 alphabetically by isometropia type,” “Replace the existing diagram with a simpler flowchart,” “Change the slide layout to Three Content and reorganize content into a three-column comparison format,” “Add slide numbers to every slide except the title slide,” and “Crop an image to show only the top two rows of buildings” (Gandhi et al., 30 Jun 2026).

3. Execution environment and interaction model

PPT-Eval runs tasks in an Ubuntu-based sandbox using screenenv. Each task is executed by uploading a copy of the task file to OneDrive, obtaining an anonymous editable PowerPoint Online URL, and launching Chromium in the sandbox with that URL. Agents interact through GUI-level actions such as mouse input, keyboard input, and scrolling; each action returns a full-screen screenshot as the observation (Gandhi et al., 30 Jun 2026).

The harness is designed for reproducibility and isolation. Every run begins with a fresh copy of the file and a clean browser session. As a result, tasks derived from the same deck are independent and can be evaluated in parallel. The system also provides per-task timeouts, logs, artifact capture, before-and-after files, and screenshots (Gandhi et al., 30 Jun 2026).

The reported evaluation settings use a 30-step maximum budget per task, 3 concurrent threads, and averages over 3 runs for agent results. The human baseline is averaged over 5 runs. With concurrency set to 3, one benchmark run takes about 3–4 hours. Reported frontier-model costs are about \$54 for Claude-4.5-Opus and about \$62 for Computer-Use-Preview (Gandhi et al., 30 Jun 2026).

A notable empirical detail is the presence of a strong API or CLI baseline. Using Claude-4.5-Opus with Claude Code and Anthropic’s pptx skill, the benchmark includes a non-GUI baseline that edits files programmatically. This baseline is evaluated under the same output-based rubric regime, which makes the benchmark suitable for comparing GUI and non-GUI methods on the same tasks (Gandhi et al., 30 Jun 2026).

4. Rubric-based evaluation framework

The central methodological contribution of PPT-Eval is its task-specific rubric system. The benchmark argues that binary pass/fail evaluation is inadequate for PowerPoint tasks because outputs are multimodal, many tasks admit multiple valid solutions, and agents frequently make only partial progress. A binary metric cannot distinguish between, for example, correct text placed badly, an almost-complete edit with one missing element, or successful local edits accompanied by collateral damage elsewhere in the deck (Gandhi et al., 30 Jun 2026).

Each task therefore has a tree-structured rubric. Internal nodes represent higher-level criteria, and leaf nodes perform concrete checks. Nodes are marked as either critical or non-critical. Critical criteria capture core task requirements, such as modifying the correct slide or inserting a required object. Non-critical criteria capture secondary but still important properties such as formatting consistency, subjective visual quality, aesthetic appropriateness, and avoidance of extraneous edits (Gandhi et al., 30 Jun 2026).

Each leaf node implements

$0.77$5

and returns a numeric score in $[0,1]$ together with a natural-language explanation. Leaf checks may use python-pptx for structural verification, PPTDiff for slide, animation, and transition diffs, and LLM or VLM checks for semantics and subjective visual assessment. PPTDiff is introduced specifically because some information is not visible from screenshots and some PowerPoint properties are buried in XML that python-pptx alone does not expose (Gandhi et al., 30 Jun 2026).

Parent-node aggregation is formalized. Let $s_i \in [0,1]$ be the score of child $i$, let $\mathcal{C}$ be the set of critical children, and let $\mathcal{N}$ be the set of non-critical children. Then

$$\bar{s}_{\mathrm{crit}} = \frac{1}{|\mathcal{C}|} \sum_{i\in\mathcal{C}} s_i, \qquad \bar{s}_{\mathrm{non}} = \frac{1}{|\mathcal{N}|} \sum_{i\in\mathcal{N}} s_i.$$

If both critical and non-critical children exist, the parent score is

$$\bar{s}_{\mathrm{parent}} = \max\big\{0,\; \bar{s}_{\mathrm{crit}} - \lambda\,(1-\bar{s}_{\mathrm{non}})\big\},$$

with

$0.77$0

If all children are of one type, the parent score is the simple average

$0.77$1

The stated desiderata are that zero or irrelevant progress should score 0, meaningful intermediate checkpoints should receive partial credit, and perfect completion should score 1 (Gandhi et al., 30 Jun 2026).

Natural-language feedback is propagated through the tree as well. At each internal node, an LLM is prompted to synthesize a coherent explanation from child scores and rationales. The benchmark therefore produces both scalar scores and readable justifications (Gandhi et al., 30 Jun 2026).

The primary benchmark metrics are Success Rate, defined as the percentage of tasks that receive a perfect score of 1, and Avg. Score, defined as the average rubric score across tasks (Gandhi et al., 30 Jun 2026).

5. Meta-evaluation and empirical results

PPT-Eval includes an explicit rubric validation procedure. The meta-evaluation uses 30 tasks sampled 2–3 per file, with 2–4 human-created attempts per task spanning four completion categories: No Progress, Some Progress, Significant Progress, and Perfect Completion. The reported agreement with human judgments is Kendall’s $0.77$2 and Spearman’s $0.77$3, which the paper characterizes as strong consistency. Category-wise accuracies are 100% for No Progress, 44.44% for Some Progress, 61.54% for Significant Progress, and 88.89% for Perfect Completion (Gandhi et al., 30 Jun 2026).

Robustness checks are also reported. For 61 tasks with non-deterministic rubric components, the authors reran evaluation five times and found low variance. For Computer-Use-Preview the mean variance is 0.0008, median variance 0.0000, and mean coefficient of variation 0.073; for Claude-4-Sonnet the corresponding values are 0.0062, 0.0000, and 0.093. When the VLM used in rubric judgments is swapped from Claude-4-Sonnet to GPT-4.1 on VLM-requiring tasks, the mean absolute error is 0.1 and 78.3% of scores are within $0.77$4 (Gandhi et al., 30 Jun 2026).

The benchmark’s headline system-level results are summarized below.

System	Success rate	Avg. score
Human baseline	0.80	0.90
API baseline (Claude-4.5-Opus + Claude Code + pptx skill)	0.62	0.81
Claude-4.5-Opus	0.45	0.57
Claude-4-Sonnet	0.42	0.53
Computer-Use-Preview	0.38	0.49
OpenCUA-32B	0.28	0.42
OpenCUA-7B	0.24	0.36
Qwen3-VL-8B	0.16	0.27
Qwen3-VL-32B	0.14	0.23

All agent numbers are averages over three runs (Gandhi et al., 30 Jun 2026).

Difficulty trends are pronounced. Claude-4.5-Opus drops from 0.56 success rate on easy tasks to 0.35 on hard tasks; Claude-4-Sonnet drops from 0.55 to 0.17; Computer-Use-Preview drops from 0.47 to 0.20; and the open-weight models are often near or below 0.10 success rate on hard tasks (Gandhi et al., 30 Jun 2026). The human baseline also declines with difficulty, from 0.88 success rate on easy tasks to 0.68 on hard tasks, which the benchmark uses as evidence that the hard tier is meaningfully difficult even for people (Gandhi et al., 30 Jun 2026).

A distinctive empirical result is that the API baseline outperforms the GUI agents overall despite having less access to PowerPoint’s native feature surface. The paper interprets this as evidence that current GUI computer-use agents are still relatively immature compared with CLI or API-based agents. At the same time, it documents tasks that at least one GUI agent could solve but the API baseline struggled with, including adding a hyperlink on “HSC” linking to slide 1, applying a different design theme while maintaining the current greenish background color scheme, and inserting a SmartArt accounting-equation graphic (Gandhi et al., 30 Jun 2026).

6. Position within the PowerPoint evaluation landscape

PPT-Eval belongs to a broader family of PowerPoint benchmarks, but its combination of GUI interaction, realistic deck editing, and task-specific partial-credit grading is distinctive. PPTC evaluates PowerPoint task completion through programmatic APIs and result-based artifact matching; PPTC-R extends that line with robustness stressors such as adversarial instructions, multilingual settings, and API-version shifts (Guo et al., 2023, Zhang et al., 2024). PPTBench evaluates multimodal models on PowerPoint detection, understanding, modification, and generation tasks using screenshot-plus-JSON inputs rather than GUI interaction (Huang et al., 2 Dec 2025). PPTEval in PPTAgent is a deck-level judge-based framework organized around Content, Design, and Coherence for automatic presentation generation, while UniPPTEval in UniPPTBench adds a scenario-aware layer for vague-prompt, long-document, multimodal-document, and multi-source generation (Zheng et al., 7 Jan 2025, Zhao et al., 17 May 2026).

This positioning clarifies what PPT-Eval is and is not. It is not primarily a layout-understanding benchmark for multimodal LLMs, and it is not a generation-quality scorer for text-to-slide systems. Its focus is computer-use agents operating in PowerPoint Online, with evaluation anchored in before-and-after files and detailed rubrics rather than single holistic scores (Gandhi et al., 30 Jun 2026).

The paper also states several limitations. Partial grading remains imperfect in a domain as rich as PowerPoint. Rubric authoring is expensive, requiring about 150 hours of human effort; more than 81% of model-generated rubric drafts had to be fixed. Subjective tasks remain subjective, and the model-based components can hallucinate. The benchmark covers PowerPoint deeply but not cross-application workflows. Finally, the fact that the API baseline outperforms GUI agents is interpreted partly as a statement about current model maturity rather than as proof of the inherent superiority of API-based interaction (Gandhi et al., 30 Jun 2026).

Taken together, PPT-Eval establishes PowerPoint Online as a high-value domain for evaluating computer-use systems. Its principal contribution is not only the task set, but the argument that realistic slide-editing evaluation requires structured partial credit, explicit treatment of collateral damage and aesthetics, and output-based scoring that can compare GUI and non-GUI methods within the same benchmark (Gandhi et al., 30 Jun 2026).