Spreadsheet-RL: RL for Spreadsheet Automation

• Spreadsheet-RL is a reinforcement learning framework that models spreadsheet automation as a sequential control task over evolving workbook states.
• It integrates Excel-native tools and a dedicated Spreadsheet Gym to enable precise formula edits, range inspections, and structural modifications.
• Empirical evaluations show significant performance boosts over baseline agents, highlighting the benefits of RL in managing long-horizon spreadsheet tasks.

Spreadsheet-RL denotes both an emerging research agenda and, in a narrower sense, a specific reinforcement-learning fine-tuning framework for spreadsheet agents. In the broad sense, it treats spreadsheet automation as a sequential decision problem over evolving workbooks rather than as one-shot text generation, spreadsheet question answering, or direct text-to-command translation. In the narrow sense, it names a 2026 framework that trains a policy LLM inside a realistic Microsoft Excel environment with outcome-based reinforcement learning, spreadsheet-native tools, and oracle workbook comparison over manipulation regions (Chi et al., 21 May 2026). This framing was prepared by earlier work showing that realistic spreadsheet tasks are long-horizon, multi-step, ambiguity-laden, and dependent on dynamic workbook state, with partial observability arising from context limits and with correctness determined by the final workbook rather than a short textual answer (Chen et al., 2024, Ma et al., 2024).

1. Genealogy and conceptual scope

Before reinforcement learning entered the spreadsheet literature explicitly, several lines of work had already established that spreadsheets are structured computational artifacts rather than passive tables. Logic-programming-based “structure discovery” aimed at recovering the grouping of cells intended by a spreadsheet’s author but not explicitly encoded in the file, using Prolog predicates and spreadsheet grammars such as DOWN, ALONG, and AND (0802.3940). A separate line showed that ordinary spreadsheet formulas can implement all data transformations definable in SQL, and even spreadsheet-native constructions for $O(n \log^2 n)$ sorting and graph traversal via BFS and DFS, without macros or external runtimes (Sroka et al., 2013). These works did not formulate spreadsheet control as RL, but they clarified that spreadsheets have latent structure, executable semantics, and nontrivial transition dynamics.

Modern spreadsheet-agent work shifted the field from static structure recovery or formula compilation toward interactive control. “SheetAgent” argued that realistic spreadsheet work should be cast as long-horizon interaction over realistic spreadsheet files, often with multiple sheets, evolving contents, and semantic ambiguities in the instruction, rather than as a single-shot spreadsheet-command problem (Chen et al., 2024). “SpreadsheetBench” extended this shift by using 912 authentic questions from Excel forums and by evaluating one generated solution across multiple spreadsheet test cases per instruction in an online-judge-like manner, making robustness a first-class requirement (Ma et al., 2024). Spreadsheet-RL inherits this agentic view and adds explicit policy optimization in a real Excel environment (Chi et al., 21 May 2026).

A concise view of the lineage is useful.

Work	Main contribution	Relation to Spreadsheet-RL
(0802.3940)	Structure discovery with Prolog grammars	Latent state abstraction
(Sroka et al., 2013)	SQL-to-spreadsheet compilation, BFS/DFS	Spreadsheet as executable substrate
(Chen et al., 2024)	Long-horizon spreadsheet agent and SheetRM	Sequential-control framing
(Ma et al., 2024)	Real-world benchmark with multiple test cases	Robustness-sensitive evaluation
(Chi et al., 21 May 2026)	RL fine-tuning in real Excel	Explicit Spreadsheet-RL method

The term itself also admits a persistent ambiguity. “Spread2RML” is relevant to spreadsheet automation, but its “RL” relevance is through the RDF Mapping Language rather than reinforcement learning; it is a heuristic, template-driven system for predicting RML mappings on messy spreadsheets (Schröder et al., 2021). In the spreadsheet-agent literature, therefore, Spreadsheet-RL refers specifically to reinforcement-learning-style control of spreadsheet environments, not to RML-based semantic lifting.

2. Task formulation as sequential spreadsheet control

The spreadsheet-RL view emerges most clearly when spreadsheet manipulation is formalized as a sequence of observations, actions, and workbook transitions. SheetRM, the benchmark introduced with SheetAgent, uses a three-part task schema consisting of spreadsheet assets, a natural-language task instruction, and a checklist (Chen et al., 2024). The checklist decomposes success into tuples of the form $$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$, so evaluation already has a structured subgoal character rather than a pure end-state binary label. SheetRM emphasizes reasoning-dependent manipulation, multi-category coverage, long-horizon tasks, and procedure evaluation; it covers five broad manipulation categories and 36 subtypes, and most tasks range from 3 to 7 subtasks.

SpreadsheetBench sharpens the same formulation in a different direction. It defines the dataset as

$\mathcal{D}=\{(q_i, c_i, a_i)\},$

where $q_i$ is an instruction, $c_i$ is a spreadsheet, and $a_i$ is the answer of $q_i$ on $c_i$; after multi-test-case construction this becomes

$\mathcal{D}=\{(q_i,\{(c_{ij},a_{ij})\})\}.$

The answer is not a formula string or text span, but “the modified spreadsheet resulting from the application of a solution” (Ma et al., 2024). Its soft and hard metrics make robustness explicit:

$S_{soft}=\frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\left(\frac{1}{|T_i|}\sum_{j=1}^{|T_i|}\mathds{1}_{r_{ij} = ACC}\right),$

$$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$0

Spreadsheet-RL adopts SpreadsheetBench’s task formulation. Each task consists of initial spreadsheet(s) $$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$1, natural-language instruction $$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$2, oracle final spreadsheet $$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$3, and manipulation regions $$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$4, and the agent must produce a final spreadsheet $$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$5 matching $$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$6 over the specified regions (Chi et al., 21 May 2026). This does not amount to an explicit MDP or POMDP formalization in the paper. However, the surrounding literature repeatedly identifies the ingredients of such a formulation: evolving workbook state, partial observability caused by context limits, delayed success criteria, and compounding errors from early incorrect edits (Chen et al., 2024, Ma et al., 2024). This suggests that Spreadsheet-RL is best understood as an operational sequential-control framework first and a formal control-theoretic object second.

3. Spreadsheet-RL framework and Spreadsheet Gym

The 2026 “Spreadsheet-RL” framework combines a scalable data pipeline, a multi-turn Excel environment called Spreadsheet Gym, a spreadsheet-native tool harness, and asynchronous outcome-based RL using GRPO (Chi et al., 21 May 2026). The policy LLM interacts with Spreadsheet Gym in a multi-turn manner, interleaving reasoning with tool calls. The reward is terminal and outcome-based:

$$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$7

The high-level objective is KL-regularized:

$$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$8

Optimization is performed with GRPO, using grouped rollouts and group-relative advantages; the paper states that $$(\text{Criterion}, \text{Sheet Index}, \text{Area}, \text{Properties})$$9 is computed by normalizing outcome rewards within the sampled group, but does not provide the exact normalization formula (Chi et al., 21 May 2026).

Spreadsheet Gym is the environment in which this training and evaluation occur. It uses Microsoft Excel itself as the runtime—specifically Microsoft Excel 365, version 2512—rather than an approximation (Chi et al., 21 May 2026). The environment exposes spreadsheet functionality through a Python sandbox and a spreadsheet-native tool set comprising find_cells, inspect_range, fill_formula, clear_range, delete_rows, delete_columns, recalculate_and_read, and code_interpreter. The harness also imposes routing rules: use find_cells for headers or anchors, inspect_range for relevant ranges, fill_formula for formula-filled targets, clear_range for blank cells, structural delete tools for row or column deletion, recalculate_and_read after custom formula edits, and code_interpreter for fallback logic. Read-only calls are allowed up to 20 concurrent per assistant turn, write-related calls must be issued one at a time, and read-only and write calls must not be mixed in the same turn.

A defining design feature is that the harness encodes spreadsheet semantics directly. The paper gives concrete failure modes it is meant to suppress: deleting columns left-to-right while indices shift, hand-writing formula strings with incorrect escaping, failing to translate relative references when copying formulas, and confusing blanking cells with deleting rows or columns (Chi et al., 21 May 2026). It also promotes an explicit workflow prior:

$\mathcal{D}=\{(q_i, c_i, a_i)\},$0

The framework further emphasizes that RL proceeds without SFT warm-up; the claim is that the harness and tools provide a sufficiently strong initial interaction policy to make RL viable from a weak but nontrivial starting point (Chi et al., 21 May 2026).

4. Data construction and benchmark ecology

Spreadsheet-RL’s training data are produced by an automated “Spreadsheet Data Agent” that harvests seed metadata from ExcelForum posts after 01/01/2024 (Chi et al., 21 May 2026). The raw crawl yields 18,855 discussion threads, 32,691 spreadsheet attachments, and 144,694 user replies, with an average of 7.67 replies per thread. After coding-agent execution, oracle construction, and rule-based filtering, the final training set contains 5,928 high-quality tasks; 2,417 tasks have more than one initial spreadsheet.

The framework evaluates on two benchmarks. The first is SpreadsheetBench, with 912 unique tasks, each instantiated into three distinct but highly similar test cases (Chi et al., 21 May 2026). The second is Domain-Spreadsheet, introduced as the first open-source domain-specific spreadsheet benchmark, containing 1,660 spreadsheet tasks across Finance-B, Finance-I, Finance-A, Supply Chain, HR, Sales, and Real Estate (Chi et al., 21 May 2026). The paper reports no formal train/validation/test splits for Domain-Spreadsheet.

The per-domain difficulty profile makes clear that Domain-Spreadsheet is not merely a re-labeled manipulation benchmark. Overall, the benchmark has mean sheets $\mathcal{D}=\{(q_i, c_i, a_i)\},$1, input median size $\mathcal{D}=\{(q_i, c_i, a_i)\},$2 KB, baseline Pass@1 $\mathcal{D}=\{(q_i, c_i, a_i)\},$3, and RL Pass@1 $\mathcal{D}=\{(q_i, c_i, a_i)\},$4 (Chi et al., 21 May 2026).

Domain	#Eval.	Base → RL Pass@1
Finance-B	597	15.6 → 29.3
Finance-I	388	7.7 → 16.2
Finance-A	135	8.1 → 19.3
Supply Chain	180	1.1 → 5.0
HR	185	0.5 → 3.2
Sales	86	1.2 → 5.8
Real Estate	89	1.1 → 1.1

The surrounding benchmark literature shows why such data are needed. SpreadsheetBench documents real spreadsheets with multiple tables in one sheet, non-standard relational tables, free-form text, and non-textual elements; 35.7% of spreadsheets contain multiple tables in one sheet, and 42.7% contain non-standard relational tables (Ma et al., 2024). WorkstreamBench, although not an RL paper, extends the benchmark horizon further by evaluating end-to-end financial workbook construction under a rubric that weights Accuracy at 50%, Formula at 35%, and Format at 15%, thereby shifting evaluation from local cell correctness to professional spreadsheet quality (Yen et al., 21 May 2026). This suggests that the current Spreadsheet-RL benchmarks capture realistic manipulation and domain workflows, but not yet the full artifact-level standards of professional finance modeling.

5. Empirical results and comparative position

The headline empirical claim of Spreadsheet-RL is that RL substantially improves a small open-source spreadsheet agent on realistic tasks (Chi et al., 21 May 2026). On SpreadsheetBench, Qwen3-4B-Thinking-2507 improves from 12.0% to 23.4% Pass@1. The ablation chain is especially informative: the raw base model scores 12.0, adding the spreadsheet-native interaction harness raises this to 15.6, adding comprehensive spreadsheet-tool access raises it further to 19.3, and Spreadsheet-RL post-training reaches 23.4. The decomposition indicates that environment design and structured tools are not auxiliary conveniences; they are major determinants of the gains.

On Domain-Spreadsheet, overall Pass@1 rises from 8.4% to 17.2%, with the largest improvements in finance domains: 15.6 to 29.3 in Finance-B, 7.7 to 16.2 in Finance-I, and 8.1 to 19.3 in Finance-A (Chi et al., 21 May 2026). Real Estate remains unchanged at 1.1, which the paper treats as evidence of unresolved domain gaps or task complexity. Training dynamics over the 60-step run show smoothed training reward rising from roughly 0.21 to 0.33, SpreadsheetBench accuracy rising from 19.3% at step 0 to 23.4% at step 60, mean response length dropping from about 16k tokens to about 11k, and mean interaction turns dropping from roughly 20 to about 11.

These results should be read against both stronger closed-source agents and earlier non-RL spreadsheet agents. In the same Spreadsheet-RL evaluation table, Copilot Agent Mode reaches 57.7 on SpreadsheetBench, Claude Files Opus 4.1 reaches 42.9, ChatGPT agent with .xlsx access reaches 45.5, and OpenAI o3 reaches 23.3 (Chi et al., 21 May 2026). Spreadsheet-RL therefore narrows, but does not close, the gap to the strongest proprietary systems. Earlier prompting-based work had already shown that architecture matters on spreadsheet tasks: SheetAgent improved Pass@1 from 44.3 to 61.1 on SCB under GPT-3.5 and from 1.7 to 30.8 on the harder SheetRM benchmark, with especially large gains under reasoning-heavy long-horizon conditions (Chen et al., 2024). SpreadsheetBench had also shown a large human gap: human experts reached 71.33% soft and 62.00% hard overall, far above the best reported LLM systems in that benchmark’s original study (Ma et al., 2024).

The comparative picture is therefore consistent across papers. Prompting alone is not enough; tool design and interaction structure matter; RL adds further gains when paired with faithful environment semantics and spreadsheet-native actions; and even then, realistic spreadsheet automation remains far from solved (Chen et al., 2024, Ma et al., 2024, Chi et al., 21 May 2026).

6. Related directions, misconceptions, and limitations

Spreadsheet-RL sits within a broader ecosystem of agentic spreadsheet research that is not itself RL but supplies many of the abstractions RL systems require. SpreadsheetAgent uses a two-stage multi-agent framework with code execution, images, LaTeX tables, and a verification module to construct a verified YAML structural representation before solving downstream tasks; the paper is explicit that it is not a reinforcement-learning method, but its localized reading, multi-format state, and verification loop are directly relevant to spreadsheet control under partial observability (Ren et al., 14 Apr 2026). BRTR likewise replaces single-pass retrieval with an iterative tool-calling loop over rows, columns, windows, and images, and extends this into a planner-executor architecture that can support spreadsheet editing, formula writing, formatting, and charts; again, it is not RL, but it provides an explicit action vocabulary, observation model, and auditable trace structure (Gulati et al., 6 Mar 2026). WorkstreamBench contributes a different missing piece: a multidimensional artifact-level quality taxonomy for end-to-end finance workbooks, emphasizing readability, auditability, range hygiene, avoidance of hardcoding, and professional format (Yen et al., 21 May 2026).

Several misconceptions recur in this area. First, Spreadsheet-RL is not simply any spreadsheet-capable LLM agent. SheetAgent, SpreadsheetAgent, BRTR, and SpreadsheetBench are highly relevant, but none of them is itself a reinforcement-learning training method (Chen et al., 2024, Ren et al., 14 Apr 2026, Gulati et al., 6 Mar 2026, Ma et al., 2024). Second, Spreadsheet-RL is not the same as spreadsheet semantic lifting to RDF; Spread2RML is relevant through spreadsheet automation and messy-sheet interpretation, but its “RL” means RML, the RDF Mapping Language, not reinforcement learning (Schröder et al., 2021). Third, Spreadsheet-RL is not yet a full formal theory of spreadsheet control. The 2026 paper does not present an explicit MDP or POMDP notation, does not provide the exact closed-form normalization for GRPO advantages, and does not report train/validation/test splits for Domain-Spreadsheet (Chi et al., 21 May 2026).

The limitations identified in the core framework are substantial. The experiments focus on relatively lightweight open-source models; larger dense models or MoE models are not trained. Reward is terminal and outcome-based, so credit assignment remains difficult. Deployment risk is explicit: spreadsheet agents can make incorrect formulas, structural edits, or subtle formatting mistakes, and such errors matter in high-stakes domains (Chi et al., 21 May 2026). The authors therefore frame Spreadsheet-RL as a research foundation rather than a fully deployable decision-making system, and recommend human review, transparent edit logs, stronger verification, privacy safeguards, and domain-specific safety checks.

A plausible implication is that Spreadsheet-RL is best viewed, at present, as an overview of four ingredients: realistic spreadsheet benchmarks, faithful Excel execution, structured tool interfaces, and verifiable outcome rewards. The field’s central open problems follow directly from that synthesis: better formal state abstractions, denser and more reliable reward design, safer edit verification, stronger domain generalization, and evaluation regimes that move from region-level correctness toward full professional workbook quality (Chi et al., 21 May 2026, Yen et al., 21 May 2026).