---
title: 'Spreadsheet-RL: RL for Spreadsheet Automation'
url: https://www.emergentmind.com/topics/spreadsheet-rl
type: topic
---

# Spreadsheet-RL: RL for Spreadsheet Automation

Spreadsheet-RL denotes both an emerging research agenda and, in a narrower sense, a specific reinforcement-learning fine-tuning framework for spreadsheet agents. In the broad sense, it treats spreadsheet automation as a sequential decision problem over evolving workbooks rather than as one-shot text generation, spreadsheet question answering, or direct text-to-command translation. In the narrow sense, it names a 2026 framework that trains a policy LLM inside a realistic Microsoft Excel environment with outcome-based reinforcement learning, spreadsheet-native tools, and oracle workbook comparison over manipulation regions [2605.22642]. This framing was prepared by earlier work showing that realistic spreadsheet tasks are long-horizon, multi-step, ambiguity-laden, and dependent on dynamic workbook state, with partial observability arising from context limits and with correctness determined by the final workbook rather than a short textual answer [2403.03636][2406.14991].

## 1. Genealogy and conceptual scope

Before reinforcement learning entered the spreadsheet literature explicitly, several lines of work had already established that spreadsheets are structured computational artifacts rather than passive tables. Logic-programming-based “structure discovery” aimed at recovering the grouping of cells intended by a spreadsheet’s author but not explicitly encoded in the file, using Prolog predicates and spreadsheet grammars such as `DOWN`, `ALONG`, and `AND` [0802.3940]. A separate line showed that ordinary spreadsheet formulas can implement all data transformations definable in SQL, and even spreadsheet-native constructions for \(O(n \log^2 n)\) sorting and graph traversal via BFS and DFS, without macros or external runtimes [1305.2103]. These works did not formulate spreadsheet control as RL, but they clarified that spreadsheets have latent structure, executable semantics, and nontrivial transition dynamics.

Modern spreadsheet-agent work shifted the field from static structure recovery or formula compilation toward interactive control. “SheetAgent” argued that realistic spreadsheet work should be cast as long-horizon interaction over realistic spreadsheet files, often with multiple sheets, evolving contents, and semantic ambiguities in the instruction, rather than as a single-shot spreadsheet-command problem [2403.03636]. “SpreadsheetBench” extended this shift by using 912 authentic questions from Excel forums and by evaluating one generated solution across multiple spreadsheet test cases per instruction in an online-judge-like manner, making robustness a first-class requirement [2406.14991]. Spreadsheet-RL inherits this agentic view and adds explicit policy optimization in a real Excel environment [2605.22642].

A concise view of the lineage is useful.

| Work | Main contribution | Relation to Spreadsheet-RL |
|---|---|---|
| [0802.3940] | Structure discovery with Prolog grammars | Latent state abstraction |
| [1305.2103] | SQL-to-spreadsheet compilation, BFS/DFS | Spreadsheet as executable substrate |
| [2403.03636] | Long-horizon spreadsheet agent and SheetRM | Sequential-control framing |
| [2406.14991] | Real-world benchmark with multiple test cases | Robustness-sensitive evaluation |
| [2605.22642] | RL fine-tuning in real Excel | Explicit Spreadsheet-RL method |

The term itself also admits a persistent ambiguity. “Spread2RML” is relevant to spreadsheet automation, but its “RL” relevance is through the RDF Mapping Language rather than reinforcement learning; it is a heuristic, template-driven system for predicting RML mappings on messy spreadsheets [2110.12829]. In the spreadsheet-agent literature, therefore, Spreadsheet-RL refers specifically to reinforcement-learning-style control of spreadsheet environments, not to RML-based semantic lifting.

## 2. Task formulation as sequential spreadsheet control

The spreadsheet-RL view emerges most clearly when spreadsheet manipulation is formalized as a sequence of observations, actions, and workbook transitions. SheetRM, the benchmark introduced with SheetAgent, uses a three-part task schema consisting of spreadsheet assets, a natural-language task instruction, and a checklist [2403.03636]. The checklist decomposes success into tuples of the form \((\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})\), so evaluation already has a structured subgoal character rather than a pure end-state binary label. SheetRM emphasizes reasoning-dependent manipulation, multi-category coverage, long-horizon tasks, and procedure evaluation; it covers five broad manipulation categories and 36 subtypes, and most tasks range from 3 to 7 subtasks.

SpreadsheetBench sharpens the same formulation in a different direction. It defines the dataset as
$$
\mathcal{D}=\{(q_i, c_i, a_i)\},
$$
where \(q_i\) is an instruction, \(c_i\) is a spreadsheet, and \(a_i\) is the answer of \(q_i\) on \(c_i\); after multi-test-case construction this becomes
$$
\mathcal{D}=\{(q_i,\{(c_{ij},a_{ij})\})\}.
$$
The answer is not a formula string or text span, but “the modified spreadsheet resulting from the application of a solution” [2406.14991]. Its soft and hard metrics make robustness explicit:
$$
S_{soft}=\frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\left(\frac{1}{|T_i|}\sum_{j=1}^{|T_i|}\mathds{1}_{r_{ij} = ACC}\right),
$$
$$
S_{hard}=\frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\mathds{1}_{r_{ij} = ACC, \forall j = 1, 2, \ldots, |T_i|}.
$$

Spreadsheet-RL adopts SpreadsheetBench’s task formulation. Each task consists of initial spreadsheet(s) \(D_i\), natural-language instruction \(T\), oracle final spreadsheet \(D_O\), and manipulation regions \(M\), and the agent must produce a final spreadsheet \(D_A\) matching \(D_O\) over the specified regions [2605.22642]. This does not amount to an explicit MDP or POMDP formalization in the paper. However, the surrounding literature repeatedly identifies the ingredients of such a formulation: evolving workbook state, partial observability caused by context limits, delayed success criteria, and compounding errors from early incorrect edits [2403.03636][2406.14991]. This suggests that Spreadsheet-RL is best understood as an operational sequential-control framework first and a formal control-theoretic object second.

## 3. Spreadsheet-RL framework and Spreadsheet Gym

The 2026 “Spreadsheet-RL” framework combines a scalable data pipeline, a multi-turn Excel environment called Spreadsheet Gym, a spreadsheet-native tool harness, and asynchronous outcome-based RL using GRPO [2605.22642]. The policy LLM interacts with Spreadsheet Gym in a multi-turn manner, interleaving reasoning with tool calls. The reward is terminal and outcome-based:
$$
\mathcal{R}(o)=
\begin{cases}
0, & \text{if no valid output},\\
\mathrm{allcellsmatch}(D_{\mathrm{pred}}, D_o), & \text{otherwise}.
\end{cases}
$$
The high-level objective is KL-regularized:
$$
\max_{\pi_\theta}\; \mathbb{E}_{[D_i,T]\sim\mathcal{D},\,y\sim\pi_\theta(\cdot\mid D_i,T;G)}
\left[\mathcal{R}(D_i,T,D_o)\right]
-\beta\,\mathbb{D}_{\mathrm{KL}}\!\left[
\pi_\theta(\cdot\mid D_i,T;G)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid D_i,T;G)
\right].
$$
Optimization is performed with GRPO, using grouped rollouts and group-relative advantages; the paper states that \(\widehat{A}_i\) is computed by normalizing outcome rewards within the sampled group, but does not provide the exact normalization formula [2605.22642].

Spreadsheet Gym is the environment in which this training and evaluation occur. It uses Microsoft Excel itself as the runtime—specifically Microsoft Excel 365, version 2512—rather than an approximation [2605.22642]. The environment exposes spreadsheet functionality through a Python sandbox and a spreadsheet-native tool set comprising `find_cells`, `inspect_range`, `fill_formula`, `clear_range`, `delete_rows`, `delete_columns`, `recalculate_and_read`, and `code_interpreter`. The harness also imposes routing rules: use `find_cells` for headers or anchors, `inspect_range` for relevant ranges, `fill_formula` for formula-filled targets, `clear_range` for blank cells, structural delete tools for row or column deletion, `recalculate_and_read` after custom formula edits, and `code_interpreter` for fallback logic. Read-only calls are allowed up to 20 concurrent per assistant turn, write-related calls must be issued one at a time, and read-only and write calls must not be mixed in the same turn.

A defining design feature is that the harness encodes spreadsheet semantics directly. The paper gives concrete failure modes it is meant to suppress: deleting columns left-to-right while indices shift, hand-writing formula strings with incorrect escaping, failing to translate relative references when copying formulas, and confusing blanking cells with deleting rows or columns [2605.22642]. It also promotes an explicit workflow prior:
$$
\text{Inspect small relevant workbook ranges} \rightarrow
\text{plan the smallest necessary edit} \rightarrow
\text{modify the workbook} \rightarrow
\text{verify values, formulas, or formatting} \rightarrow
\text{fix iteratively if needed} \rightarrow
\text{save data.xlsx}.
$$
The framework further emphasizes that RL proceeds without SFT warm-up; the claim is that the harness and tools provide a sufficiently strong initial interaction policy to make RL viable from a weak but nontrivial starting point [2605.22642].

## 4. Data construction and benchmark ecology

Spreadsheet-RL’s training data are produced by an automated “Spreadsheet Data Agent” that harvests seed metadata from ExcelForum posts after 01/01/2024 [2605.22642]. The raw crawl yields 18,855 discussion threads, 32,691 spreadsheet attachments, and 144,694 user replies, with an average of 7.67 replies per thread. After coding-agent execution, oracle construction, and rule-based filtering, the final training set contains 5,928 high-quality tasks; 2,417 tasks have more than one initial spreadsheet.

The framework evaluates on two benchmarks. The first is SpreadsheetBench, with 912 unique tasks, each instantiated into three distinct but highly similar test cases [2605.22642]. The second is Domain-Spreadsheet, introduced as the first open-source domain-specific spreadsheet benchmark, containing 1,660 spreadsheet tasks across Finance-B, Finance-I, Finance-A, Supply Chain, HR, Sales, and Real Estate [2605.22642]. The paper reports no formal train/validation/test splits for Domain-Spreadsheet.

The per-domain difficulty profile makes clear that Domain-Spreadsheet is not merely a re-labeled manipulation benchmark. Overall, the benchmark has mean sheets \(2.7\), input median size \(10.3\) KB, baseline Pass@1 \(8.4\), and RL Pass@1 \(17.2\) [2605.22642].

| Domain | #Eval. | Base → RL Pass@1 |
|---|---:|---:|
| Finance-B | 597 | 15.6 → 29.3 |
| Finance-I | 388 | 7.7 → 16.2 |
| Finance-A | 135 | 8.1 → 19.3 |
| Supply Chain | 180 | 1.1 → 5.0 |
| HR | 185 | 0.5 → 3.2 |
| Sales | 86 | 1.2 → 5.8 |
| Real Estate | 89 | 1.1 → 1.1 |

The surrounding benchmark literature shows why such data are needed. SpreadsheetBench documents real spreadsheets with multiple tables in one sheet, non-standard relational tables, free-form text, and non-textual elements; 35.7% of spreadsheets contain multiple tables in one sheet, and 42.7% contain non-standard relational tables [2406.14991]. WorkstreamBench, although not an RL paper, extends the benchmark horizon further by evaluating end-to-end financial workbook construction under a rubric that weights Accuracy at 50%, Formula at 35%, and Format at 15%, thereby shifting evaluation from local cell correctness to professional spreadsheet quality [2605.22664]. This suggests that the current Spreadsheet-RL benchmarks capture realistic manipulation and domain workflows, but not yet the full artifact-level standards of professional finance modeling.

## 5. Empirical results and comparative position

The headline empirical claim of Spreadsheet-RL is that RL substantially improves a small open-source spreadsheet agent on realistic tasks [2605.22642]. On SpreadsheetBench, Qwen3-4B-Thinking-2507 improves from 12.0% to 23.4% Pass@1. The ablation chain is especially informative: the raw base model scores 12.0, adding the spreadsheet-native interaction harness raises this to 15.6, adding comprehensive spreadsheet-tool access raises it further to 19.3, and Spreadsheet-RL post-training reaches 23.4. The decomposition indicates that environment design and structured tools are not auxiliary conveniences; they are major determinants of the gains.

On Domain-Spreadsheet, overall Pass@1 rises from 8.4% to 17.2%, with the largest improvements in finance domains: 15.6 to 29.3 in Finance-B, 7.7 to 16.2 in Finance-I, and 8.1 to 19.3 in Finance-A [2605.22642]. Real Estate remains unchanged at 1.1, which the paper treats as evidence of unresolved domain gaps or task complexity. Training dynamics over the 60-step run show smoothed training reward rising from roughly 0.21 to 0.33, SpreadsheetBench accuracy rising from 19.3% at step 0 to 23.4% at step 60, mean response length dropping from about 16k tokens to about 11k, and mean interaction turns dropping from roughly 20 to about 11.

These results should be read against both stronger closed-source agents and earlier non-RL spreadsheet agents. In the same Spreadsheet-RL evaluation table, Copilot Agent Mode reaches 57.7 on SpreadsheetBench, Claude Files Opus 4.1 reaches 42.9, ChatGPT agent with `.xlsx` access reaches 45.5, and OpenAI o3 reaches 23.3 [2605.22642]. Spreadsheet-RL therefore narrows, but does not close, the gap to the strongest proprietary systems. Earlier prompting-based work had already shown that architecture matters on spreadsheet tasks: SheetAgent improved Pass@1 from 44.3 to 61.1 on SCB under GPT-3.5 and from 1.7 to 30.8 on the harder SheetRM benchmark, with especially large gains under reasoning-heavy long-horizon conditions [2403.03636]. SpreadsheetBench had also shown a large human gap: human experts reached 71.33% soft and 62.00% hard overall, far above the best reported LLM systems in that benchmark’s original study [2406.14991].

The comparative picture is therefore consistent across papers. Prompting alone is not enough; tool design and interaction structure matter; RL adds further gains when paired with faithful environment semantics and spreadsheet-native actions; and even then, realistic spreadsheet automation remains far from solved [2403.03636][2406.14991][2605.22642].

## 6. Related directions, misconceptions, and limitations

Spreadsheet-RL sits within a broader ecosystem of agentic spreadsheet research that is not itself RL but supplies many of the abstractions RL systems require. SpreadsheetAgent uses a two-stage multi-agent framework with code execution, images, LaTeX tables, and a verification module to construct a verified YAML structural representation before solving downstream tasks; the paper is explicit that it is not a reinforcement-learning method, but its localized reading, multi-format state, and verification loop are directly relevant to spreadsheet control under partial observability [2604.12282]. BRTR likewise replaces single-pass retrieval with an iterative tool-calling loop over rows, columns, windows, and images, and extends this into a planner-executor architecture that can support spreadsheet editing, formula writing, formatting, and charts; again, it is not RL, but it provides an explicit action vocabulary, observation model, and auditable trace structure [2603.06503]. WorkstreamBench contributes a different missing piece: a multidimensional artifact-level quality taxonomy for end-to-end finance workbooks, emphasizing readability, auditability, range hygiene, avoidance of hardcoding, and professional format [2605.22664].

Several misconceptions recur in this area. First, Spreadsheet-RL is not simply any spreadsheet-capable LLM agent. SheetAgent, SpreadsheetAgent, BRTR, and SpreadsheetBench are highly relevant, but none of them is itself a reinforcement-learning training method [2403.03636][2604.12282][2603.06503][2406.14991]. Second, Spreadsheet-RL is not the same as spreadsheet semantic lifting to RDF; Spread2RML is relevant through spreadsheet automation and messy-sheet interpretation, but its “RL” means RML, the RDF Mapping Language, not reinforcement learning [2110.12829]. Third, Spreadsheet-RL is not yet a full formal theory of spreadsheet control. The 2026 paper does not present an explicit MDP or POMDP notation, does not provide the exact closed-form normalization for GRPO advantages, and does not report train/validation/test splits for Domain-Spreadsheet [2605.22642].

The limitations identified in the core framework are substantial. The experiments focus on relatively lightweight open-source models; larger dense models or MoE models are not trained. Reward is terminal and outcome-based, so credit assignment remains difficult. Deployment risk is explicit: spreadsheet agents can make incorrect formulas, structural edits, or subtle formatting mistakes, and such errors matter in high-stakes domains [2605.22642]. The authors therefore frame Spreadsheet-RL as a research foundation rather than a fully deployable decision-making system, and recommend human review, transparent edit logs, stronger verification, privacy safeguards, and domain-specific safety checks.

A plausible implication is that Spreadsheet-RL is best viewed, at present, as a synthesis of four ingredients: realistic spreadsheet benchmarks, faithful Excel execution, structured tool interfaces, and verifiable outcome rewards. The field’s central open problems follow directly from that synthesis: better formal state abstractions, denser and more reliable reward design, safer edit verification, stronger domain generalization, and evaluation regimes that move from region-level correctness toward full professional workbook quality [2605.22642][2605.22664].

Source: https://www.emergentmind.com/topics/spreadsheet-rl