PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks

Abstract: Creating and editing slides is a rich, multimodal activity that is ubiquitous in professional and educational settings, making it an ideal testbed for real-world computer-use agents. Microsoft PowerPoint is among the most widely adopted and feature-rich environments for presentation creation. We introduce PPT-Eval, a benchmark of 120 PowerPoint tasks across 12 files that cover both content creation and presentation editing scenarios, organized by difficulty. A central challenge in this domain is evaluation: tasks are complex, multimodal, and often admit many valid solutions. Moreover, today's agents frequently make only partial progress, which binary success metrics fail to capture. To address this, we design a robust evaluation framework to help create task-specific rubrics for PowerPoint tasks, taking inspiration from and building on past works for rubric-based evaluation. These rubrics award partial credit for intermediate steps, penalize unnecessary changes and poor aesthetics, and provide natural language feedback. This nuanced approach proves highly effective, achieving a Kendall's τ-b correlation of 0.77 with human judgments. We find that existing frontier agents still struggle with solving PowerPoint tasks, with strong models like Claude-4.5-Opus achieving only a 45% success rate and an average partial score of 57%. The benchmark is located at: https://microsoft.github.io/ppteval.

• The paper introduces PPT-Eval, a benchmark that evaluates multimodal, GUI-based interactions in PowerPoint using a detailed rubric system.
• The methodology curates 120 diverse tasks across domains, leveraging LLM pipelines and human annotation to simulate realistic, complex PowerPoint edits.
• Empirical results highlight significant performance gaps between API and GUI agents, underscoring research opportunities in multimodal reasoning and action modeling.

PPT-Eval: A Comprehensive Benchmark for PowerPoint Computer-Use Agents

Motivation and Benchmark Design

PPT-Eval is introduced as a targeted benchmark for evaluating the capabilities of computer-use agents in the context of Microsoft PowerPoint, specifically leveraging the rich, multimodal feature set of PowerPoint Online. Unlike prior benchmarks, which either provide only superficial coverage of presentation tools or restrict agent capabilities to narrow programmatic APIs, PPT-Eval evaluates agents at the GUI level—mirroring realistic user interaction and accessing the entirety of PowerPoint’s functionality. This is accomplished by operating in a sandboxed browser instance with full GUI access, bypassing the limitations of API-driven interaction.

Figure 1: Task-solving process in the PPT-Eval Benchmark, where agents interact directly with a PowerPoint Online instance and are scored using detailed, task-specific rubrics supporting partial credit and feedback.

Tasks within PPT-Eval are curated for diversity and realism. The benchmark encompasses 120 tasks across 12 distinct files, extracted from open-source presentations spanning a wide variety of topics and visual styles, including medicine, computer science, architecture, and social science. Each file introduces slides featuring text-dense, graphics-rich, and highly heterogeneous content formats. The resulting benchmark supports both content creation and iterative editing, with tasks systematically stratified by difficulty to cover the spectrum from trivial modifications to complex, multi-step edits.

Figure 3: Representative slide samples from the benchmark, demonstrating the diversity in topic and visual structure.

Richness of Task Space

The PPT-Eval task suite is constructed to exercise PowerPoint's full feature surface, requiring substantial visual, semantic, and structural reasoning from agents. Tasks are categorized by both difficulty and high-level intent (e.g., layout modifications, image operations, table manipulation, data visualization), capturing the broad operational spectrum needed for real-world productivity enhancement.

Task curation exploits LLM-based exploration pipelines to propose plausible, grounded tasks for each deck, with human annotators subsequently filtering and refining these proposals for relevance, automatic evaluability, and difficulty balance. A sunburst plot illustrates the proportional distribution of tasks by both difficulty (easy, medium, hard) and intent cluster, highlighting the deliberate diversity.

Figure 2: Distribution of tasks by difficulty and high-level intent, evidencing the comprehensive coverage of PowerPoint functionality.

The slides themselves reveal notable heterogeneity in element composition, with some emphasizing shapes and graphics, others dense with tables or non-standard layouts. This diversity ensures agents are challenged across all operational axes of PowerPoint.

Figure 5: Distribution of relevant slide elements, emphasizing the presence of advanced features such as non-standard layouts, images, and shapes.

Rubric-Based Evaluation Framework

A major contribution of PPT-Eval is its rubric-based evaluation paradigm. Recognizing the challenge posed by open-ended, multimodal tasks with numerous valid completion states, the authors implement hierarchical rubric trees, with both critical and non-critical nodes corresponding to required and desirable criteria, respectively. Leaf nodes are implemented algorithmically via Python, LLM, or VLM calls, assessing semantic equivalence and visual correctness where necessary.

Aggregation of partial progress is carefully calibrated through a parametric formula balancing critical vs. non-critical completion, avoiding the coarse binary scoring pitfalls of previous efforts. As a result, nuanced partial credit is awarded, and the system produces natural language feedback at every evaluation node.

An extensive meta-evaluation indicates strong alignment with human judgement, reporting Kendall’s $\tau_b{=}0.77$ and Spearman’s $\rho{=}0.84$. Notably, category-wise accuracy is perfect for the "No Progress" and high for "Perfect Completion" classes, though understandably lower for gradated partial progress categories.

Figure 4: Example rubric tree for a representative edit task, visualizing critical/non-critical node structure and aggregation of partial scores.

Figure 6: Model success rates disaggregated by PowerPoint task structure (images, tables, shapes, etc.), highlighting both global and categorical performance gaps.

Empirical Results: Agent Performance Analysis

The benchmark includes detailed evaluation of both closed-source (frontier) and open-weights models, as well as strong API-based baselines and human users. Notable numerical results include:

• Human participants: 80% average success rate (SR), 0.90 mean rubric score.
• API (CLI) baseline (Claude-4.5-Opus + pptx skill): 62% SR, 0.81 mean score.
• Best frontier GUI model (Claude-4.5-Opus): 45% SR, 0.57 mean score.
• Open-weights models (OpenCUA-32B/7B, Qwen3-VL-32B/8B): Maximum SRs of 28% (OpenCUA-32B) and mean scores of 0.42, with Qwen3-VL variants lagging further behind.

Task breakdowns expose systematic deficiencies: while easy and some medium tasks see more successful completion by leading models, performance sharply degrades for tasks involving advanced object manipulations, layout changes, and holistic multimodal coordination. The partial credit mechanism is particularly vital: even models with similar success rates display differentiated progress as captured by mean scores.

Figure 7: Per-cluster success rates for each model, indexed by high-level task intent, illustrating pronounced model-specific variability.

Several findings stand out:

• Despite GUI agents having access to the full PowerPoint feature set, the best API agent still outperforms GUI models overall. This underscores the present gap in maturity of GUI-based agent training relative to established programmatic manipulation protocols.
• Open-weights models benefit from specialization: OpenCUA, which targets desktop usage, decisively outperforms more generalist vision-LLMs such as Qwen3-VL.
• The rubric system enables fine-grained identification of partial progress, typical errors, and confusion modes, facilitating targeted future improvements.

Meta-Evaluation and Rubric Stability

Rigorous meta-evaluation demonstrates robust reliability of the rubric framework. Detailed analyses identified primary sources of human–rubric disagreement: subjective task elements, VLM hallucinations, and granularity in partial progress categorization. Nevertheless, for deterministic and unambiguous outcomes, agreement is near-perfect. Variance analyses, including Monte Carlo sampling of VLM-judged rubrics, yield very low fluctuation and confirm stability across repeated runs and between different LLM/VLM backends.

Figure 10: Example of perfect human–rubric agreement in a table insertion task, illustrating fine rubric granularity and robust alignment with human assessments.

Figure 9: A failure case: VLM hallucination and human–rubric subjective disagreement in an icon placement task, highlighting unresolved evaluative challenges.

Implications and Future Directions

PPT-Eval establishes a new methodological standard for benchmarking the real-world capabilities of computer-use agents in rich, GUI-based software environments. The persistent headroom in agent performance, combined with high-fidelity partial credit evaluation, reveals critical research opportunities in action modeling, multimodal reasoning, and robust evaluation. The benchmark’s infrastructure is method-agnostic and extensible to other GUI-driven domains.

Key future directions identified include:

• Enhancing the smoothness and coverage of partial credit grading, particularly for tasks with high subjectivity or multiple correct solutions.
• Automating rubric and task generation further to enable scalable reward modeling for agent training via reinforcement learning or off-policy pipelines.
• Extending benchmarking efforts to encompass multi-application workflows, better reflecting real-world productivity settings.

Conclusion

PPT-Eval provides a robust, nuanced, and practically relevant framework for assessing computer-use agents in rich, user-centric environments. Through comprehensive tasks, a rigorous rubric architecture, and systematic model evaluation, it both reveals the limitations of current models and enables discriminative, targeted progress measurement. The benchmark’s approach to partial credit, multimodal evaluation, and human alignment marks a substantive contribution to the assessment of real-world agentic AI, setting the groundwork for accelerated advances in this domain.