SymbArena: SR Benchmark & Fine-Tuning Corpus

• SymbArena is a symbolic regression benchmark and training corpus featuring 148,102 unique equations paired with numerical data for LLM fine-tuning.
• It employs a two-stage equation generation process using tree-based synthesis, complexity filtering, and unique skeleton splitting to ensure valid and diverse formulas.
• Evaluation combines numerical accuracy with form-level consistency using both GPT-4o and rule-based metrics, supporting comparisons across LLM-based and traditional methods.

Searching arXiv for the primary SymbArena paper and related benchmark context. arxiv_search(query="SymbArena symbolic regression benchmark LLM fine-tuning", max_results=5) Searching arXiv for related work on symbolic regression benchmarks and the SymbArena paper. SymbArena is a benchmark and training corpus for symbolic regression that was introduced to make LLM fine-tuning for symbolic regression both possible and meaningful. It is defined around 148,102 diverse equations, a corpus of about 1.83 billion tokens, paired numerical data for each equation, train/test support for both LLM-based and traditional symbolic regression methods, and an evaluation scheme that combines numerical accuracy with form-level consistency. In the associated study, SymbArena is the substrate on which the baseline system SymbolicChat is developed and evaluated, with the paper framing the benchmark as a response to “an inherent tension between pre-trained LLMs' proficiency in approximate reasoning and the high-precision demands of SR tasks” (Hua et al., 13 Aug 2025).

1. Origin, scope, and benchmark philosophy

SymbArena was introduced against a specific diagnosis of prior LLM-based symbolic regression. The paper argues that existing methods relied primarily on direct inference or iterative prompt engineering, repeatedly sampling equations and retaining those with good fit, but that this strategy suffered from ineffectiveness, poor generalization, lack of precision, and masked errors under numerical metrics. The central claim is that symbolic regression requires exact structural recovery, coefficient sensitivity, and mathematically valid expression generation, whereas general-purpose pretrained LLMs are optimized for broad, approximate, and ambiguous reasoning (Hua et al., 13 Aug 2025).

Within that framing, SymbArena is not presented as a conventional test-only benchmark. It is simultaneously a training resource for SR-oriented fine-tuning and an evaluation benchmark designed to detect cases where a model is numerically plausible but symbolically wrong. The paper contrasts it with Nguyen, R rational, SRbench, and LLM-SRbench, arguing that SymbArena differs by scale, explicit train/test splitting, support for both traditional and LLM-based methods, and richer evaluation including symbolic similarity. This suggests a deliberate shift from benchmarking symbolic regression as pure equation search toward benchmarking it as a trainable structure-recovery problem.

2. Dataset construction and curation

The benchmark dataset is defined as

$D = \{(d_i, f_i)\}_{i=1}^N,$

where $f_i$ is the symbolic equation and $d_i \in \mathcal{R}^{K \times (C_i+1)}$ is the data matrix containing $K$ samples, each with $C_i$ independent variables and one dependent output. Construction proceeds in two stages: equation generation and numerical data sampling (Hua et al., 13 Aug 2025).

Equations are represented as trees. Each node is either a terminal—a variable placeholder or constant placeholder—or an operator node—a unary or binary operator. The generation pipeline first samples the number of independent variables $C_i$, creates variable placeholder nodes, adds a constant placeholder node, and samples operators from a predefined symbol library. The reported library includes binary operators such as +, -, *, /, unary or transcendental functions such as exp, log, sqrt, sin, arcsin, and examples also include arctan, while exponentiation appears as **. Trees are expanded recursively so that syntactic validity is guaranteed “by construction,” excluding malformed expressions such as unmatched parentheses or invalid operator placements.

After generation, equations are filtered by uniqueness and complexity. Uniqueness is checked using skeleton string similarity. Complexity is controlled by retaining only equations with tree depth between 4 and 8, thereby removing formulas that are described as too trivial or excessively hard. Numerical data are then produced by sampling inputs from either $\mathcal{U}(-dom, dom)$ or $\mathcal{N}(0, dom)$, with the fixed domain parameter

$dom = 10.$

The sampled inputs are substituted into the generated equation, and the resulting input-output tuples are assembled into $d_i$.

A major curation decision is the train-test split by unique form/skeleton, rather than by instance. The stated purpose is to prevent leakage in which the same structural formula appears in train and test with only different coefficients. For the test set only, the authors apply a “reality enhancement” procedure: for each synthetic equation they retrieve the 3 most similar human-known scientific equations using an LLM “deep research” function, obtain similarity scores and explanations, manually inspect those retrievals, and filter out equations with low similarity to existing scientific knowledge. The benchmark is therefore synthetic in origin but partially curated to make evaluation closer to realistic scientific formulas.

3. Symbolic representation and evaluation methodology

SymbArena adopts an explicit textual symbolic format intended for LLM consumption. Variables appear as placeholders such as x_0, x_1, and x_2, while constants are abstracted canonically as C. Outputs are expressed as equations for y, and prompts present numerical data as key-value text rows such as x_0=..., x_1=..., y=.... In experiments, prompting and evaluation consistently use 200 input-output pairs per problem (Hua et al., 13 Aug 2025).

Several preprocessing and canonicalization steps are integral to both training and evaluation. Equations are abstracted into skeletons, which remove coefficient identity and focus on structure. Coefficients are replaced by C, and variables may be normalized for pattern comparison. Deduplication uses skeleton string similarity, complexity filtering constrains tree depth to the interval from 4 to 8, validity filtering excludes malformed syntax, and implementation additionally uses five distinct prompt variations per expression during instruction tuning.

Evaluation is explicitly two-dimensional: numerical accuracy and form-level consistency. Numerical accuracy is measured by $f_i$0 and by tolerance-based accuracy $f_i$1. Form-level consistency is measured in two ways: a GPT-4o-based semantic adjudication and a rule-based heuristic metric intended to be reproducible and LLM-independent.

The rule-based structural metric averages six similarity components:

Feature	Comparison rule
Operators	Jaccard similarity
Functions	Jaccard similarity
Variables	Jaccard similarity
Constants	Ratio similarity
Structural pattern	Character-by-character normalized match
Complexity	Ratio similarity

For operators, functions, and variables, similarity is based on set overlap. For constants and complexity, similarity is defined by the ratio between the smaller and larger scalar value, with similarity equal to 1 if both are zero. Structural pattern similarity compares normalized pattern strings character-by-character and normalizes by the longer string length. The final score is clipped to $f_i$2. The paper’s rationale is that numerical metrics can be misleading because a structurally wrong formula may still fit sampled data reasonably well after coefficient optimization. In that sense, SymbArena treats structural fidelity as a first-class benchmark target rather than as an auxiliary diagnostic.

4. Fine-tuning substrate and the SymbolicChat pipeline

SymbArena is intended to support multiple workflows: instruction tuning / supervised fine-tuning of LLMs for symbolic regression, reinforcement fine-tuning with structure-aware rewards, inference benchmarking for LLM-based methods, evaluation of traditional symbolic regression methods, iterative search/refinement methods, and unified comparison across symbolic regression paradigms (Hua et al., 13 Aug 2025).

The paper’s primary backbone is Qwen2.5-7B-Instruct. The baseline system, SymbolicChat, is organized in three stages: instruction tuning, rule-based reward tuning via Form-GRPO, and Scientific Generative Agent refinement during inference. Instruction tuning uses 146,590 unique expressions with five distinct prompt variations per expression and LoRA fine-tuning with rank 8 and alpha 32. The supervised objective is cross-entropy over prompt-equation pairs.

After instruction tuning, the model undergoes reinforcement fine-tuning with GRPO adapted to symbolic regression. The reward has four components. A format reward tests whether the generated expression can be compiled into executable form using numerical libraries such as NumPy or math. A form-similarity reward uses the structural similarity metric. A numerical reward uses nonnegative $f_i$3. An equivalence reward checks whether the predicted and target equations share the same coefficient-abstracted skeleton. The final reward is a weighted sum with ablation-selected weights

$f_i$4

Reward tuning uses 1,000 expressions, samples 8 candidate responses per input, and sets group size to $f_i$5.

Inference-time refinement is performed by a Scientific Generative Agent procedure. The process initializes with a primitive equation, maintains a population of candidate equations, inserts top-scoring candidates into later prompts as in-context examples, generates offspring for exploitation and exploration, evaluates candidates by $f_i$6, optionally optimizes constants, and repeats. The reported implementation uses 5 iterations, generating 2 expressions for exploitation and 4 for exploration per cycle. The paper’s characterization is that SymbolicChat remains architecturally simple—standard open-source LLM backbone, SFT via LoRA, GRPO-style tuning, handcrafted rewards, and iterative refinement—but aligns those components to the symbolic validity and structural consistency demands of symbolic regression.

5. Experimental results and ablation structure

The principal benchmark comparison reports results for PySR, prompt-based and search-based LLM baselines, and SymbolicChat. On SymbArena, the headline figures are as follows (Hua et al., 13 Aug 2025):

Method	$f_i$7 rule	$f_i$8
PySR	0.382	0.663
LLM-SR (Qwen2.5-7B)	0.404	0.313
SGA (gpt-3.5-turbo)	0.504	0.327
SymbolicChat	0.602	0.799

The full table additionally reports GPT-4o structural scores, tolerance-based accuracy, and runtime. SymbolicChat attains $f_i$9 GPT-4o 0.429, $d_i \in \mathcal{R}^{K \times (C_i+1)}$0 rule 0.602, $d_i \in \mathcal{R}^{K \times (C_i+1)}$1 0.799, $d_i \in \mathcal{R}^{K \times (C_i+1)}$2 0.412, and time 92.05 s. PySR reports $d_i \in \mathcal{R}^{K \times (C_i+1)}$3 GPT-4o 0.368, $d_i \in \mathcal{R}^{K \times (C_i+1)}$4 rule 0.382, $d_i \in \mathcal{R}^{K \times (C_i+1)}$5 0.663, $d_i \in \mathcal{R}^{K \times (C_i+1)}$6 0.398, and time 6.85 s. The paper interprets these results as evidence that, with SymbArena-based fine-tuning, an LLM can exceed a traditional symbolic regression system in both structural fidelity and numerical accuracy.

The ablation results are used to separate the contributions of fine-tuning and iterative refinement. Base Qwen2.5-7B without fine-tuning reports rule-based $d_i \in \mathcal{R}^{K \times (C_i+1)}$7 and $d_i \in \mathcal{R}^{K \times (C_i+1)}$8. Applying iterative strategies directly to the base model improves numerical fit somewhat, but not decisively. After instruction tuning, Qwen2.5-7B-SFT reaches rule-based $d_i \in \mathcal{R}^{K \times (C_i+1)}$9 and $K$0 without iterative search, while SGA on top of SFT yields $K$1 GPT-4o 0.452, rule-based $K$2 0.601, and $K$3. After SFT + GRPO, direct generation improves further, and SGA on the tuned model reaches the full SymbolicChat result: rule-based $K$4, $K$5, and $K$6. The paper summarizes this as showing that SFT teaches task format and structural priors, GRPO improves direct generation quality, and iterative refinement yields the strongest numerical fit.

The qualitative examples are aligned with the metric design. For a target such as

$K$7

the paper states that only SymbolicChat recovers the exact symbolic form, while competing methods produce polynomial, interaction, or sinusoidal approximations that may partially fit the data but are structurally wrong. Common failure modes include adding extra polynomial terms, dropping divisive structure, replacing nested expressions with simpler surrogates, and overproducing terms.

6. Limitations, reporting issues, and terminological distinctions

The benchmark’s limitations are explicit even within the presentation of positive results. SymbArena is synthetically generated, although the test set is reality-enhanced. Its rule-based structural metric is deliberately useful and reproducible, but it remains heuristic rather than a proof of symbolic equivalence. One of the two structural scores depends on GPT-4o, which introduces evaluator dependence. Input sampling uses the fixed normalization domain $K$8, and the operator library is controlled rather than unrestricted. These design choices make the benchmark operationally coherent, but they also bound the notion of symbolic regression being evaluated (Hua et al., 13 Aug 2025).

The paper also contains reporting inconsistencies. One section states that 11,000 equations are used for training and 512 for testing, whereas the implementation section states that instruction tuning uses 146,590 unique expressions, reward tuning uses 1,000 expressions, and the test set contains 512 equations, summing to 148,102 total equations. A related discrepancy appears in the headline claim of an 8.37% improvement in form-level consistency over the second-best LLM baseline: the text notes that this figure is not directly reconstructible from the table as written. These issues do not negate the benchmark design, but they qualify how precisely individual summary numbers should be interpreted.

The name SymbArena can also invite confusion because several arXiv works use “arena” terminology for unrelated problem settings. DIAMBRA Arena is a Gym-compatible reinforcement-learning platform built around retro fighting games (Palmas, 2022). Arena is a toolkit for multi-agent reinforcement learning centered on the Interface abstraction (Wang et al., 2019). Alexa Arena is a user-centric embodied-AI simulation and benchmark platform for human-robot interaction (Gao et al., 2023). SymbArena is distinct from all three: it is an SR benchmark and fine-tuning corpus for equation inference rather than an RL, MARL, or embodied-AI platform. This distinction is substantive, because SymbArena’s central objects are symbolic equations, paired numerical datasets, structure-aware metrics, and LLM fine-tuning procedures, not action spaces, embodied worlds, or interactive control loops.

In that form, SymbArena’s main significance lies in reframing LLM symbolic regression as a training-and-evaluation problem rather than only a prompting problem. The benchmark’s combination of scale, skeleton-aware splitting, and form-level evaluation is designed to measure whether a model discovers the right law rather than merely interpolating a sampled dataset. That is the core scientific role it occupies in current symbolic regression research.