FormulaCode: Structured Symbolic Computation

• FormulaCode is a research framing that treats formulas as formal code artifacts, enabling execution, structural comparison, and precise evaluation across various domains.
• Empirical studies show that employing concise comments, descriptive variable naming, and hardcoded solutions significantly boosts LLM accuracy in mathematical reasoning and code generation.
• Applications of FormulaCode include executable Python rationales, spreadsheet formula generation, image-to-LaTeX conversion, and benchmarks for optimizing performance in scientific codebases.

FormulaCode denotes a research framing in which formulas are treated as formal code artifacts rather than as purely informal notation. In the cited literature, this framing appears in several distinct forms: executable Python rationales for mathematical reasoning in LLMs, spreadsheet formulas generated from natural language or inferred from semi-structured tabular context, LaTeX source recovered from document images, and, in a separate usage, a benchmark named FormulaCode for repository-level performance optimization in scientific Python codebases. The shared premise is that formula-like reasoning can be represented in structured symbolic languages and then evaluated through execution, exact structural comparison, or workload-based measurement (Wei et al., 2024, Zhao et al., 2024, Chen et al., 2021, Liu et al., 24 Mar 2025, Sehgal et al., 16 Mar 2026).

1. Conceptual scope and formal representations

In the mathematical-reasoning setting, FormulaCode corresponds to code-based rationales: solutions expressed as executable programs, typically Python, rather than pure text. CoinMath explicitly treats mathematical reasoning as learning from code-like implementations of formulas and systematically optimizes how such code is written so that LLMs learn math better. The paper characterizes this as a concrete realization of the FormulaCode idea, with Program-of-Thought reasoning delegating numeric computation to a Python interpreter while the model generates the code implementing the reasoning steps (Wei et al., 2024).

In spreadsheet research, FormulaCode denotes a domain-specific programming language. NL2Formula defines the target as a Microsoft Excel formula $F = \{y_1, y_2, \ldots, y_K\}$, generated from a natural-language query $N$ grounded in a spreadsheet table $T$, with the task formulated as $F = f_\theta(N; T)$. The formulas follow an extended BNF grammar supporting constants, arithmetic, cell references, ranges, and function calls such as SUM, FILTER, MINIFS, and SORT, and are executable in Microsoft Excel (Zhao et al., 2024). SpreadsheetCoder adopts the same basic view, but its specification is implicit in the surrounding spreadsheet context: headers, nearby rows, nearby columns, and the missing target cell whose formula is to be predicted (Chen et al., 2021).

In document intelligence, FormulaCode is instantiated as image-to-LaTeX conversion. PP-FormulaNet formalizes the task as learning $f_\theta : \mathcal{X} \to \mathcal{Y}$, where $\mathcal{X}$ is a space of formula images and $\mathcal{Y}$ is a sequence of LaTeX tokens. The output is directly usable symbolic code containing operators, structural markup such as \frac and \sum, and start, end, and padding tokens (Liu et al., 24 Mar 2025).

In software engineering, FormulaCode is the title of a benchmark for evaluating LLM-based coding agents on repository-level performance optimization. Here the object being optimized is not a mathematical or spreadsheet formula, but performance-critical code in large scientific Python repositories. The benchmark packages baseline repository snapshots, expert optimization patches, correctness tests, and large sets of Airspeed Velocity workloads, and evaluates agents under multi-objective correctness and performance constraints (Sehgal et al., 16 Mar 2026).

This breadth suggests that FormulaCode is best understood as a general representational stance: formulas, symbolic expressions, or optimization-relevant program structures are encoded in languages that admit parsing, execution, or downstream measurement.

2. Mathematical reasoning as executable FormulaCode

CoinMath studies which coding styles are most beneficial when mathematical solutions are represented as executable code. It starts from MathInstruct code-based rationales and transforms them with GPT-4o into stylistic variants along three axes: comment usage, naming convention, and solution generality. The three attributes selected as beneficial are concise comments, descriptive variable naming, and hardcoded solutions tailored to the concrete problem instance (Wei et al., 2024).

The empirical results are specific. For comment usage, concise comments and detailed comments both substantially outperform no comments. For Llama-3.1-8B, the average accuracy over Arithmetic, SVAMP, GSM, and MATH is 34.0 for no comments, 68.4 for concise comments, and 68.3 for detailed comments. For CodeLlama-Python-7B, the corresponding averages are 56.7, 59.9, and 58.9. For naming conventions, descriptive naming outperforms obscure naming: on Llama-3.1-8B, average accuracy is 68.2 versus 66.5, and on Gemma-2-2B, 56.1 versus 54.2. For solution generality, hardcoded solutions outperform generalized ones on Llama-3.1-8B, with averages of 67.6 versus 63.7 (Wei et al., 2024).

CoinMath also investigates whether additional supervision sources help once math code is already present. General-domain coding instructions improve valid code generation substantially relative to vanilla models, but Math-Code alone yields higher math accuracy than General Code, and mixing General Code with Math-Code hurts relative to Math-Code alone. For zero-shot Llama-3.1-8B, the average accuracy is 4.1 for the vanilla model, 27.2 for +G.C., 66.8 for +M.C., and 58.2 for +Mix. Textual rationales are similarly limited: for Llama-3.1-8B, Math-Text yields 44.0, Math-Code 66.8, and Mix 67.9, whereas for CodeLlama-7B, Math-Text yields 26.9, Math-Code 60.2, and Mix 58.6 (Wei et al., 2024).

The resulting training strategy outperforms prior math LLM tuning recipes. CoinMath, trained on diversified code styles, exceeds MAmmoTH by an average of +5.9% accuracy across math benchmarks. In the Llama-3.1 setting, average accuracy rises from 66.2 for MAmmoTH to 69.2 for CoinMath, and in the CodeLlama-Python setting from 53.6 to 62.4. Ablation results further show that the curated combination of concise comments, descriptive naming, and hardcoded solutions is more effective than indiscriminately mixing all styles (Wei et al., 2024).

A central implication is that FormulaCode in math LLMs is not merely executable output. The structure and style of the code act as part of the supervision signal. Concise comments align problem text with computation, descriptive identifiers encode semantic roles, and hardcoded instances reduce abstraction burden in zero-shot inference. The paper explicitly limits its evaluation to datasets requiring concrete calculations and notes that code-based solutions may be less effective for abstract reasoning, theorem comprehension, or areas such as abstract algebra (Wei et al., 2024).

3. Spreadsheet FormulaCode: generation from language and context

Spreadsheet FormulaCode appears in two complementary formulations. NL2Formula studies natural-language-to-formula generation: given a natural-language query and a spreadsheet table, generate an executable Excel formula. The dataset contains 70,799 paired natural-language queries and spreadsheet formulas over 21,670 tables, covering 37 types of formula functions. It is constructed from WikiSQL, Spider, and TAT-QA, and includes both Analysis Query formulas and Calculation formulas (Zhao et al., 2024).

The baseline model for NL2Formula is fCoder, a T5-based sequence-to-sequence Transformer that serializes the natural-language query and flattened table into a single input sequence and trains with autoregressive cross-entropy. Evaluation uses Exact Match and Execution Result Assessment. On the test set, fCoder-Large achieves 70.6% Formula EM and 77.1% ERA, substantially outperforming ForTap at 24.2% EM and GPT-3.5 in a 10-shot setting at 21.4% EM and 25.2% ERA. Performance varies by difficulty: fCoder-Large reaches 45.4 EM on Simple, 76.0 on Medium, 58.4 on Complex, and 76.5 on Calculation formulas; ERA is consistently higher than EM because distinct formulas can be semantically equivalent (Zhao et al., 2024).

SpreadsheetCoder addresses a different specification regime. Instead of taking an explicit natural-language query, it predicts a missing Google Sheets formula from semi-structured tabular context, including headers and surrounding cells. Its architecture combines a row-based BERT encoder, a column-based BERT encoder, convolution layers that aggregate broader context across bundles, and a two-stage LSTM decoder that first predicts a formula sketch and then predicts concrete relative cell ranges. The range grammar uses bounded relative offsets $R[-10] \ldots R[10]$ and $C[-10] \ldots C[10]$, allowing formulas such as SUM(C2:C6) to be represented compactly as a sketch followed by relative range tokens (Chen et al., 2021).

The dataset for SpreadsheetCoder is large-scale and production-oriented: 46k spreadsheets yield about 20M formulas initially, and after filtering and deduplication the retained corpus contains 770k training samples, 42k validation samples, and 34k test samples. On the Google Sheets test set, SpreadsheetCoder achieves top-1 formula accuracy of 42.51%, top-5 of 54.41%, and top-10 of 58.57%, compared with 31.14% top-1 for row-based RobustFill, 20.65% for column-based RobustFill, and 10.56% for a no-context model. In production-style evaluation, SpreadsheetCoder assists 82% more users in composing formulas on Google Sheets than a rule-based system (Chen et al., 2021).

The two spreadsheet lines expose two recurring difficulties in FormulaCode research. First, exact string match is often too strict. NL2Formula gives the example of =MINIFS(G2:G6, J2:J6, "5th") and =MIN(FILTER(G2:G6, J2:J6="5th")), which yield the same execution result but differ syntactically (Zhao et al., 2024). Second, layout sensitivity is fundamental. In NL2Formula, shifting a table from A1 to B2 requires cell references to update accordingly, and a model trained only on A1-anchored tables drops to an average EM of 6.7% under such shifts. SpreadsheetCoder mitigates related issues with relative range encoding and explicit row/column context, but still restricts references to within $\pm 10$ rows and columns and excludes cross-tab and absolute references (Zhao et al., 2024, Chen et al., 2021).

4. Visual FormulaCode: formula recognition as image-to-LaTeX

PP-FormulaNet formulates formula recognition as conversion from document images to LaTeX source code. The output vocabulary includes symbols, operators, structural markup, and special tokens, and the system is designed for printed formulas extracted from PDFs and LaTeX documents, with some reported generalization to handwritten expressions. Accuracy is measured with BLEU on simple and complex test sets from UniMERNet-1M and easy, middle, and hard splits from arXiv-4M (Liu et al., 24 Mar 2025).

The model family has two variants. PP-FormulaNet-L is optimized for accuracy and uses a Vary-ViT-B visual encoder together with an mBART-style Transformer decoder of hidden dimension 512. PP-FormulaNet-S is optimized for efficiency and uses a PP-HGNetV2-B4 CNN encoder, knowledge distillation from Vary-ViT-B, a 384-dimensional decoder, and multi-token parallel prediction. On the main comparison table, PP-FormulaNet-L achieves SPE-BLEU 0.9055, CPE-BLEU 0.9207, Easy 0.9392, Middle 0.9273, Hard 0.9142, and Avg-BLEU 0.9213, with inference times of 1976.52 ms at batch size 1 and 332.12 ms at batch size 15. PP-FormulaNet-S achieves Avg-BLEU 0.8712 and runs in 202.25 ms at batch size 1 and 32.85 ms at batch size 15, making it over 16 times faster than UniMERNet at batch size 15 while still exceeding UniMERNet’s Avg-BLEU of 0.8613 (Liu et al., 24 Mar 2025).

A major part of the contribution is the Formula Mining System that creates arXiv-4M. The pipeline reconstructs source order across multiple .tex files, extracts formulas from a widened set of math environments, recovers user-defined commands such as \newcommand and \DeclareMathOperator, normalizes source code with KaTeX, and then renders formulas to images through pdflatex, fitz, and OpenCV cropping. Combined with UniMERNet-1M, this yields 5M formula-image pairs for training (Liu et al., 24 Mar 2025).

The training objective is standard sequence cross-entropy,

$N$0

augmented in PP-FormulaNet-S by a knowledge distillation loss that matches projected student features to teacher features, and by multi-token prediction using a parallel causal mask. The multi-token ablation quantifies the speed–accuracy tradeoff: with step = 1, CPE-BLEU is 0.9092 and batch-1 inference time is 2779.76 ms; with step = 3, CPE-BLEU is 0.8689 and inference time is 969.30 ms; with step = 5, CPE-BLEU falls to 0.8048 while inference time drops to 600.08 ms. The chosen operating point for PP-FormulaNet-S is step = 3 (Liu et al., 24 Mar 2025).

This branch of FormulaCode makes explicit that symbolic code recovery is not limited to textual or tabular inputs. It also shows that canonicalization matters. By normalizing visually equivalent LaTeX variants into a more uniform representation, the system reduces the many-to-one ambiguity between rendered formulas and source code. The paper nonetheless notes remaining challenges for noisy scans, arbitrary handwritten styles, and very complex or long expressions (Liu et al., 24 Mar 2025).

5. FormulaCode as a benchmark for agentic optimization

In 2026, FormulaCode also becomes the name of a benchmark for evaluating agentic optimization on large codebases. The benchmark comprises 957 real performance bottlenecks mined from 70 open-source scientific Python repositories, each corresponding to a real GitHub pull request whose primary intent was performance optimization and whose effect is statistically validated with Airspeed Velocity workloads. Each task includes the baseline repository snapshot, the expert patch, correctness tests, and, on average, 264.58 community-maintained performance workloads (Sehgal et al., 16 Mar 2026).

The mining pipeline has four stages: scraping ASV-compliant repositories, attribute and LLM-based filtering of pull requests, environment synthesis through a reflexive LLM agent that creates docker_build.sh scripts, and statistical plus correctness validation. From 766 candidate repositories and 26,717 pull requests across 127 repositories, the pipeline retains 3,181 candidate performance-improving PRs from 101 repositories after filtering, produces reproducible Docker environments for 1,232 tasks from 75 repositories, and finally keeps 957 tasks across 70 repositories after significance testing and strict correctness checks. The resulting benchmark is live and, as of 2025-11-30, projects to add an average of 27.00 new tasks per month (Sehgal et al., 16 Mar 2026).

Evaluation is multi-objective. For each workload $N$1, the per-workload speedup is

$N$2

Task-level performance is the geometric mean across workloads,

$N$3

and the human-relative score is

$N$4

The benchmark further defines stratified advantage over hierarchical groups of workloads and a normalized advantage that divides $N$5 by $N$6, thereby penalizing highly variable gains. If the agent fails correctness tests for a workload, that workload’s speedup is set to 1 (Sehgal et al., 16 Mar 2026).

The benchmark-level results show that current frontier agents improve runtimes relative to baseline but remain below human experts. On FormulaCode-V, all evaluated agents achieve $N$7, but all have negative advantage relative to the expert baseline with $N$8. The best reported agent, OpenHands with Claude 4.0 Sonnet, attains $N$9 and normalized advantage $T$0. Stratified analyses show that agents are generally better at function-level local optimizations than at module-level improvements, and per-tag analysis indicates relative strength on Parallelization and Batching but marked weakness on tasks classified as Use Lower Level System (Sehgal et al., 16 Mar 2026).

This benchmark-specific usage narrows FormulaCode to repository-scale, workload-validated software optimization. Yet the connection to the broader FormulaCode framing remains clear: performance-critical behavior is evaluated not by a single unit test or exact patch match, but by structured execution across a large space of formal workloads. The benchmark therefore extends the FormulaCode idea from symbolic artifacts to multi-objective operational behavior in real codebases (Sehgal et al., 16 Mar 2026).

6. Evaluation regimes, misconceptions, and open problems

A recurring issue across FormulaCode research is that exact symbolic match and semantic correctness are not identical. NL2Formula addresses this directly with Execution Result Assessment, because multiple spreadsheet formulas may compute the same answer even when Exact Match fails. SpreadsheetCoder, by contrast, reports formula, sketch, and range accuracy but notes that semantic equivalence is difficult to define for arbitrary spreadsheet formulas. This difference is not merely metric engineering; it reflects a fundamental ambiguity in symbolic targets whose denotation may be invariant under multiple surface forms (Zhao et al., 2024, Chen et al., 2021).

Another common misconception is that more auxiliary supervision is always helpful. CoinMath shows the opposite for several cases. General-domain coding instructions improve valid code rate but remain inferior to math-specific code for mathematical reasoning, and indiscriminately mixing General Code with Math-Code hurts relative to Math-Code alone. Textual rationales provide only slight gains for general-purpose models and may reduce performance for code-specialized models (Wei et al., 2024).

Layout and grounding remain unresolved bottlenecks. NL2Formula reports that models trained on A1-anchored tables collapse to average EM of 6.7% when tables are shifted, revealing memorization of absolute positions rather than robust addressing logic. SpreadsheetCoder improves on context-free baselines through row-wise and column-wise modeling, relative range grammar, and header-aware encoding, but still excludes long-distance, absolute, and cross-sheet references. In formula recognition, PP-FormulaNet reduces code ambiguity through LaTeX normalization but still faces accuracy–speed tradeoffs under multi-token decoding and remains primarily trained on printed formulas rather than arbitrary handwriting (Zhao et al., 2024, Chen et al., 2021, Liu et al., 24 Mar 2025).

The 2026 FormulaCode benchmark introduces a different class of limitation. Its domain is strongly biased toward scientific and numeric Python, because that ecosystem provides mature ASV suites and dense performance workloads. The benchmark also inherits the trust boundary of community-maintained workloads and is computationally expensive to evaluate at scale. The authors therefore recommend treating FormulaCode scores as complementary signals rather than as sole criteria for code quality, maintainability, or security (Sehgal et al., 16 Mar 2026).

Taken together, these works suggest several open directions. For mathematical FormulaCode, the central unanswered question is how well executable code rationales extend beyond concrete calculations to abstract reasoning, proofs, or theorem comprehension. For spreadsheet FormulaCode, position invariance, richer function coverage, messy multi-table layouts, and longer input contexts remain open. For visual FormulaCode, explicit structure-aware decoding and broader handwritten support remain unresolved. For repository-scale FormulaCode, the frontier challenge is holistic optimization across many workloads under strict correctness constraints, where current agents still underperform expert humans (Wei et al., 2024, Zhao et al., 2024, Liu et al., 24 Mar 2025, Sehgal et al., 16 Mar 2026).