SheetBench: Benchmarking Spreadsheet Agents

• SheetBench is a benchmark suite for spreadsheet-agent evaluation, covering manipulation, structural reasoning, and end-to-end workflow tasks.
• It comprises distinct benchmarks—SpreadsheetBench, SheetBench in SheetBrain, and SpreadsheetBench 2—each targeting unique task categories and data sources.
• The benchmarks emphasize real-world challenges such as non-standard layouts, hierarchical headers, and cross-sheet dependencies to drive model improvements.

Searching arXiv for the benchmark lineage and naming variants to ground the article in the cited papers. “SheetBench” is used in the recent spreadsheet-agent literature as a benchmark name with two distinct referents. In "SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows," it is explicitly introduced as a short form for SpreadsheetBench 2, a workflow-level benchmark over generation, debugging, and visualization tasks in authentic business workbooks (Zhu et al., 29 Jun 2026). In "SheetBrain: A Neuro-Symbolic Agent for Accurate Reasoning over Complex and Large Spreadsheets," “SheetBench” denotes a 69-case benchmark for complex tables, multi-table layouts, large sheets, and editing operations (Wang et al., 22 Oct 2025). Both uses are best understood against the earlier "SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation," which established a real-world spreadsheet manipulation benchmark derived from solved online Excel-forum questions (Ma et al., 2024).

1. Nomenclature and benchmark lineage

The name overlap is substantive rather than merely stylistic. The literature uses “SpreadsheetBench” for the 2024 real-world manipulation benchmark, “SheetBench” for the 2025 benchmark released with SheetBrain, and “SpreadsheetBench 2” with “SheetBench” as a short form for the 2026 workflow-level benchmark (Ma et al., 2024).

Benchmark	Scope	Key statistics
SpreadsheetBench	Real-world spreadsheet manipulation from online forums	912 instructions; 2,729 input–output pairs
SheetBench	Complex and large spreadsheet reasoning	69 cases from 11 prior public sources
SpreadsheetBench 2 (“SheetBench”)	End-to-end business spreadsheet workflows	321 tasks; average 11.8 worksheets

This nomenclature has two implications. First, “SheetBench” does not denote a single fixed dataset across papers. Second, the benchmarks reflect different problem formulations: code-generation-based spreadsheet manipulation in the 2024 benchmark, structurally complex spreadsheet reasoning in the 2025 benchmark, and multi-turn workflow execution in the 2026 benchmark. A plausible implication is that the term now refers less to one dataset than to a benchmark line centered on realistic spreadsheet-agent evaluation.

2. SpreadsheetBench (2024): real-world spreadsheet manipulation

The 2024 SpreadsheetBench was introduced to evaluate spreadsheet manipulation in actual user workflows rather than simplified synthetic settings (Ma et al., 2024). It is built from 912 real questions gathered from ExcelForum, MrExcel, ExcelGuru, and Chandoo. The collection process retained only solved threads, removed questions about UI components or other software-specific issues, and filtered out ambiguous cases, threads without attachments, and threads lacking clear solutions.

Its structural diversity is central to the benchmark definition. The associated spreadsheets contain multiple tables per sheet in 35.7% of files, non-standard relational tables in 42.7%, and multi-sheet files in 26.0%. The benchmark comprises 2,729 input–output pairs, 2,729 Excel files, and 3,917 total sheets. The row and column distributions are long-tailed: 80% of sheets have 1–49 rows, but some exceed 1,000 rows; 80% span 1–13 columns, with a tail beyond 100 columns.

The task formulation is explicitly program-synthesis-oriented. Each example is a triplet $(q_i, \{(c_{ij}, a_{ij})\})$, where $q_i$ is a self-contained natural-language instruction and $\{(c_{ij}, a_{ij})\}_{j=1}^{T_i}$ is a set of spreadsheet test cases with $T_i = 3$. Solving a task requires generating Python or VBA code that transforms each input spreadsheet into the expected output. Instructions may be cell-level, such as populating a specified range, or sheet-level, where the entire sheet may be rewritten.

The 912 instructions are grouped into ten high-level categories: find, extract, sum, highlight, remove, modify, count, delete, calculate, and display. Representative tasks include text extraction from non-standard tables, schedule-grid generation with nested INDEX/MATCH logic and weekend highlighting, and multi-sheet lookup conditioned on date ranges with style preservation.

Its evaluation protocol adopts an online-judge-style design. For each instruction, annotators created two additional spreadsheets by perturbing the original input to exercise corner cases. The benchmark then reports a soft restriction score and a hard restriction score:

$$S_{\text{soft}} = \frac{1}{|D|}\sum_{i=1}^{|D|}\left[\frac{1}{|T_i|}\sum_{j=1}^{|T_i|}\mathbf{1}_{r_{ij}=\mathrm{ACC}}\right]$$

$$S_{\text{hard}} = \frac{1}{|D|}\sum_{i=1}^{|D|}\mathbf{1}_{r_{ij}=\mathrm{ACC}\ \forall j\in[1..|T_i|]}$$

Here, $r_{ij}=\mathrm{ACC}$ if the model-generated solution, when applied to $c_{ij}$, exactly matches the ground-truth $a_{ij}$.

The empirical findings establish the benchmark’s difficulty. GPT-4o achieves 18.35 overall soft accuracy and 15.02 overall hard accuracy in single-round inference, while human experts reach 71.33 and 62.00 on a 50-question subset. Multi-round prompting improves several weaker models substantially, but only marginally improves GPT-4o. The reported error modes include mis-locating non-standard headers, failing on non-textual formatting requirements, and producing brittle solutions that fail under minor data perturbations. The paper therefore positions SpreadsheetBench as a test of robust spreadsheet manipulation rather than isolated tabular QA.

3. SheetBench in SheetBrain: structural complexity and neuro-symbolic reasoning

In the SheetBrain paper, “SheetBench” names a benchmark introduced to stress modern LLM-based agents on complex spreadsheet structures rather than end-to-end business workflows (Wang et al., 22 Oct 2025). It consists of 69 spreadsheet cases drawn from 11 prior public sources: HiTab, MiMoTable, RealHiTBench, MultiHiertt, DABstep, SheetCoPilot, SpreadsheetBench, SheetAgent, NCSE, MMQA, and CodaBench.

The benchmark is organized into four task categories: Complex Tables with 21 cases, Multi-Table Layouts with 20 cases, Large Sheets with 20 cases, and Editing Operations with 8 cases. The structural complexities explicitly covered include deeply nested or hierarchical headers, merged cells spanning rows or columns, interleaved multi-table layouts on one sheet, very large row counts, mixed data types, and real-world formatting cues such as cell colors and fonts exposed via the tool-inspector.

The task definitions span three operation types. Question Answering requires extracting, comparing, or computing answers from existing data. Formula Prediction & Propagation requires generating or filling spreadsheet formulas such as SUM and AVERAGE. Data Manipulation (Editing) requires filtering rows, inserting or deleting columns, modifying layouts, and propagating formulas.

Construction emphasizes curation for difficulty. Human annotators selected cases judged to be challenging for existing LLM-based agents such as ChatGPT, BizChat, and SheetAgent. Queries and spreadsheets were then corrected where necessary to remove ambiguity or brokenness, and each case was reviewed by at least one expert annotator. No formal inter-annotator agreement statistics are reported; release followed curator sign-off.

Evaluation uses accuracy, defined as the number of correct predictions divided by the total number of predictions. For SpreadsheetBench, the official scorer was used, whereas for MultiHiertt, RealHiTBench, and SheetBench, correctness was judged by an in-house GPT-4.1-based “LLM-as-judge” prompt. SheetBench is released as a held-out test suite only, with all 69 cases evaluated at test time.

The benchmark is closely tied to the SheetBrain architecture. SheetBrain comprises an understanding module that produces a comprehensive overview of the spreadsheet, an execution module with a Python sandbox and Excel helper toolkit, and a validation module that verifies correctness and can trigger re-execution. On SheetBench, SheetBrain achieves 55/69 overall, compared with 47/69 for BizChat Analyst, 35/69 for SheetAgent, 34/69 for GPT-4o, and 13/69 for StructGPT. The difficulty analysis identifies blind previews using df.head(), local frame-of-reference errors that double-count parent and “of which” sub-rows, failures to filter breakdown rows or propagate formulas correctly, and context-window overflow on large sheets. The paper’s conclusion is explicit: real-world spreadsheets demand an understand–execute–validate workflow, and pure LLM chain-of-thought is insufficient for large or nested tables.

4. SpreadsheetBench 2 (“SheetBench”): end-to-end business spreadsheet workflows

SpreadsheetBench 2, abbreviated in the paper as “SheetBench,” shifts the evaluation target from local spreadsheet operations to complete business workflows (Zhu et al., 29 Jun 2026). It covers three task categories—generation, debugging, and visualization—and is constructed from authentic business data including financial reports and corporate filings.

The benchmark contains 321 tasks. Each instance averages 11.8 worksheets and requires 593.5 cell modifications, with a median of 11.8 sheets per task in the construction description. The tasks are derived from public financial filings from Bloomberg, Screener.in, and BSE India, together with the NYU Stern (Damodaran) dataset. The construction process required over 1,500 expert hours. Financial domain experts first created gold-standard workbooks with correct formulas, cross-sheet references, and domain logic. Tasks were then generated by removing target regions or injecting controlled errors for generation and debugging, or by specifying chart objectives for visualization. Each task includes a natural-language instruction averaging 429 words, an input workbook, and a golden solution. Two independent experts solved every task from scratch; discrepancies triggered iterative revisions until the input, instruction, and solution aligned uniquely.

The three task categories reflect professional spreadsheet practice. Generation includes both “template” and “financial modeling” scenarios. Template tasks use partially completed workbooks with an average of 1.2 sheets, while financial modeling tasks require constructing interlinked components across an average of 15.3 sheets with propagated formulas and cross-sheet dependencies. Debugging tasks ask agents to audit and fix errors such as double counting, broken cross-sheet references, misconfigured INDEX/MATCH lookups, and stray hard-coded values without being told the error locations. The ten-category error taxonomy includes Double Counting, Incorrect References, Formula Errors such as #REF! or #NUM!, Embedded Hardcodes, and Relative vs. Absolute Reference mistakes. Visualization tasks require transforming irregularly structured, multi-sheet financial data into charts and pivot tables such as football-field valuation charts, clustered-stacked bar charts, and sunburst diagrams with correct axes, labels, and color schemes.

Evaluation is performed under a unified multi-turn “SWE-agent” scaffold based on ReAct. The agent interacts with the spreadsheet through exactly three tools: bash, view_xlsx, and submit. The system prompt defines the tools, while the per-task instance prompt recommends the workflow Inspect, Plan, Implement, Execute, Verify, Submit under a fixed 50-turn budget. This design enforces iterative inspection and coding rather than one-shot editing.

The quantitative results show a marked gap between cell-level success and end-to-end reliability. Eight frontier LLMs were evaluated—Claude Opus 4.6, GPT-5.2, Gemini 3.1 Pro, GLM-5, Deepseek V3.2, Kimi K2.5, Qwen3.5-397B, and MiniMax M2.5—along with four commercial spreadsheet products on a 30-task subset. Performance is measured both by Modification, the fraction of target cells matching the golden values or formulas, and by Accuracy, the fraction of tasks where every cell matches exactly, or, for visualization, the fraction of expert rubric criteria passed. The best overall task accuracy is 34.89% for Claude Opus 4.6. Debugging accuracy is as low as 12.00%. Financial modeling reaches 89.7% cell-level modification for Claude Opus but only 34.0% end-to-end accuracy. Template tasks are easiest at 52.6% accuracy, and visualization reaches a 62.5% rubric pass rate. Commercial spreadsheet products do not exceed the foundation models under the same scaffold; Claude for Excel is best among them at 15.4% on the subset.

5. Evaluation regimes and what they measure

The three benchmarks are distinguished not only by data source and task distribution but also by what counts as a correct solution.

SpreadsheetBench (2024) evaluates executable robustness across perturbed spreadsheet files. Its soft and hard scores reward solutions that generalize over three test cases per instruction rather than merely fitting one example (Ma et al., 2024). This makes robustness to value changes part of the target capability.

SheetBench in SheetBrain evaluates case-level correctness on held-out structurally difficult spreadsheets, using either an official scorer or GPT-4.1-based judging depending on dataset. Its reported hit rates are therefore oriented toward successful reasoning over complex layouts, large sheets, and editing tasks rather than exact workbook-wide equivalence under an online-judge perturbation framework (Wang et al., 22 Oct 2025).

SpreadsheetBench 2 evaluates workflow completion under a multi-turn agent scaffold, with separate measurements for cell modification and task accuracy. Its task-accuracy definition is stricter than local edit quality because every target and side-effect cell must match exactly, or, in visualization, expert rubric criteria must be passed (Zhu et al., 29 Jun 2026). The reported gap between high modification rates and much lower end-to-end accuracy makes this distinction analytically important.

These evaluation choices encode different notions of spreadsheet competence. A plausible implication is that the benchmarks probe complementary failure surfaces: robustness to variation in SpreadsheetBench, structural disambiguation in SheetBench, and coherent multi-step execution in SpreadsheetBench 2. The recurring pattern across all three is that local success does not imply reliable spreadsheet agency.

6. Failure modes, misconceptions, and research directions

A common misconception is that spreadsheet benchmarking is primarily about single-formula generation. The benchmark sequence directly contradicts that view. The 2024 SpreadsheetBench emphasizes code that survives multiple perturbed test cases; the 2025 SheetBench stresses nested headers, merged cells, and interleaved multi-table layouts; and the 2026 SpreadsheetBench 2 centers on generation, debugging, and visualization across large multi-sheet business workbooks (Ma et al., 2024).

Another misconception is that commercial spreadsheet products necessarily outperform foundation models on spreadsheet tasks. Under the SpreadsheetBench 2 scaffold, the reported result is the opposite: commercial spreadsheet products do not beat foundation models, and Claude for Excel reaches 15.4% on the 30-task subset while the best model overall reaches 34.89% (Zhu et al., 29 Jun 2026).

The most detailed failure taxonomy appears in SpreadsheetBench 2. Analysis of failed Claude Opus 4.6 trajectories identifies six primary failure modes: Insufficient Inspection, accounting for 45% of failures; Wrong Target Selection, accounting for 30%; Task Misunderstanding; Format/Output Error; Turn Limit Exceeded; and Other rare cases. The paper provides concrete examples. In one financial-modeling failure, the agent wrote =SUM(F95+F90)/F103 instead of the gold-standard =F90/F103, effectively doubling the numerator by analogizing to a neighboring formula. In a debugging case involving double counting, one unambiguous duplicate cross-sheet link remained unpatched. The authors conclude that insufficient spreadsheet inspection and incorrect target-cell selection are the dominant bottlenecks.

The SheetBrain paper identifies a related but distinct set of bottlenecks. Baseline agents fail through blind previews using df.head(), local frame-of-reference errors that double-count hierarchical subtotals, failures in filtering breakdown rows and propagating formulas, and context-window overflow on large sheets. This suggests that structural perception and global validation are as important as raw language reasoning for spreadsheet performance (Wang et al., 22 Oct 2025).

The research directions named in the papers are correspondingly concrete. SpreadsheetBench proposes integrating tabular structure understanding modules, richer interactive execution environments, training corpora that combine real forum threads with synthetic perturbations, and extensions to non-textual elements and GUI automation (Ma et al., 2024). SheetBrain proposes incorporating cross-sheet and workbook-level reasoning, adding formula-generation tasks requiring novel Excel functions, evaluating human-in-the-loop correction workflows, and developing open train/val/test splits for learned spreadsheet agents (Wang et al., 22 Oct 2025). SpreadsheetBench 2 points to more powerful spreadsheet inspection mechanisms, potentially including specialized encoding of cross-sheet dependency graphs, and stronger grounding of domain knowledge such as financial formula definitions in actual workbook layout. It also shows that scaffold design matters: the SWE-agent setup yields similar cell-modification rates but higher end-to-end accuracy than general coding-agent scaffolds such as Claude Code, Cline, and Kilo, reported as 15% versus 8% (Zhu et al., 29 Jun 2026).

Taken together, the benchmark line associated with “SheetBench” documents an increasingly explicit transition in spreadsheet-agent evaluation: from realistic manipulation tasks, to structural complexity and neuro-symbolic reasoning, to end-to-end business workflows. The unifying result is not high absolute performance but a persistent empirical gap between current agent behavior and the reliability required for professional spreadsheet use.