SymbolBench: Time-Series Symbolic Evaluation

• SymbolBench is a benchmark assessing LLMs' ability to extract symbolic structures from real-world time series, including differential equations, Boolean rules, and causal graphs.
• It leverages curated datasets from 156 multivariate differential equations, 65 Boolean networks, and 190 SCM samples to test both in-distribution and out-of-distribution performance.
• The framework integrates LLMs with genetic programming through a closed-loop system that iteratively refines symbolic proposals via quantitative and qualitative verification.

SymbolBench is a benchmark for evaluating whether LLMs can recover symbolic structure from real-world time series in a way that resembles scientific discovery rather than mere pattern fitting. It is organized around three task families—multivariate symbolic regression, Boolean network inference, and causal discovery—and is presented simultaneously as an evaluation suite, a curated dataset/resource, and the basis for a unified closed-loop framework that combines LLMs with genetic programming (Liu et al., 5 Aug 2025).

1. Conceptual scope and motivation

SymbolBench was introduced in response to a specific limitation in earlier LLM-oriented symbolic regression evaluations: prior efforts largely concentrated on relatively narrow families of algebraic equations, often without rich real-world context, without explicit reasoning-oriented evaluation, and without coverage of symbolic targets such as logical rules and causal graphs (Liu et al., 5 Aug 2025). The benchmark is therefore intended to assess whether models can move from temporal observations to interpretable symbolic hypotheses across multiple symbolic forms, rather than only generating equations that fit a narrow class of static data.

The benchmark is explicitly designed around four objectives: real-world relevance, varied task difficulty, scale/balance, and a unified framework that can support multiple symbolic-discovery tasks (Liu et al., 5 Aug 2025). In the paper’s comparison against prior resources such as LLM-SRBench, ODEBench, and RealTCD, SymbolBench is characterized by three symbolic structure types rather than one, by support for both in-distribution and out-of-distribution evaluation, and by inclusion of reasoning and contextual or multimodal settings. The same comparison reports 156 scientific equations, 65 logical-expression tasks, and 190 SCM tasks (Liu et al., 5 Aug 2025).

A central design choice is that SymbolBench includes structured textual context, and for multimodal models also visual and temporal inputs such as trajectory plots and encoded temporal embeddings (Liu et al., 5 Aug 2025). This makes the benchmark not only a test of symbolic output accuracy but also a test of whether contextual grounding changes symbolic hypothesis generation.

2. Task families and dataset composition

SymbolBench contains three datasets, each corresponding to a different symbolic object class encountered in scientific analysis of time series (Liu et al., 5 Aug 2025).

Task family	Symbolic target	Dataset scope
Coupled Differential Equations (CDEs)	Multivariate continuous-time governing equations	Over 156 samples, extended from ODEBench
Boolean Networks (BNs)	Node-wise Boolean update rules	65 networks curated from the BioDivine repository
Structured Causal Models (SCMs)	Lagged directed causal graphs	190 samples derived from curated CDEs and Physiome models

For CDEs, the target is a continuous dynamical system of the form

$$\frac{dx_i}{dt} = f_i(x_1, x_2, ..., x_n), \quad i = 1, ..., n.$$

The unknown symbolic objects are the functions $f_i$, inferred from multivariate trajectories $\{x_i(t)\}$ (Liu et al., 5 Aug 2025). SymbolBench starts from ODEBench and extends it, especially with more high-dimensional ODEs, to obtain a more balanced distribution. The paper states that each sample is accompanied by variable descriptions and, when available, a domain name (Liu et al., 5 Aug 2025).

For Boolean networks, each node is binary and evolves according to a Boolean formula,

$$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$

with $f_i$ composed of operators such as AND, OR, and NOT (Liu et al., 5 Aug 2025). The data are sequences of binary state vectors over time, and the task is to infer the node-wise logical update rules. The benchmark uses 65 Boolean networks from the BioDivine repository, each with a short domain description and variable names (Liu et al., 5 Aug 2025).

For SCMs, the target is a lagged directed causal graph rather than a full generative equation. The formulation is

$$x_i \leftarrow \{ (x_j, l) \mid x_j \in X,\; l \in [1, M] \},$$

meaning that $x_j$ at time $t-l$ may causally influence $x_i$ at time $t$ (Liu et al., 5 Aug 2025). These samples are derived by functional analysis of curated CDEs and additional Physiome models with more than three variables. In the reported experiments, the maximum lag is set to $f_i$0 for fair comparison with Tigramite-based baselines (Liu et al., 5 Aug 2025).

A further structural distinction is that CDE evaluation includes both in-distribution and out-of-distribution settings, where the out-of-distribution condition is created by using new initial conditions (Liu et al., 5 Aug 2025). For BNs, inference is based on state transitions from various initial conditions, effectively giving state-transition graphs rather than a single smooth trajectory. For SCMs, evaluation is graph-structural rather than trajectory-simulation based (Liu et al., 5 Aug 2025).

3. Evaluation protocol and metrics

SymbolBench combines numerical, structural, and qualitative evaluation (Liu et al., 5 Aug 2025). For CDEs, the numerical fit is based on averaged $f_i$1:

$f_i$2

The main reported benchmark score is

$f_i$3

together with $f_i$4, the proportion of samples with $f_i$5 (Liu et al., 5 Aug 2025). Symbolic proximity is also reported through expression tree edit distance, and expression complexity through SymPy’s count_ops (Liu et al., 5 Aug 2025).

For BNs, the benchmark uses macro-averaged F1 over transitions:

$f_i$6

It additionally reports precision, recall, bookmaker informedness $f_i$7, symbolic proximity, expression complexity, and thresholded accuracies $f_i$8, $f_i$9, and $\{x_i(t)\}$0 (Liu et al., 5 Aug 2025).

For SCMs, the benchmark reports graph-classification metrics including F1, precision, recall, false discovery rate, thresholded accuracies, and Structural Hamming Distance (SHD) (Liu et al., 5 Aug 2025). It also introduces a structural plausibility proxy based on conditional independence:

$\{x_i(t)\}$1

and

$\{x_i(t)\}$2

The text states that “a lower CI-score suggests stronger conditional independence and thus better structural plausibility” (Liu et al., 5 Aug 2025).

A distinctive feature is the qualitative stage. SymbolBench uses an LLM as judge and assigns rubric-based scores from 1 to 5 for context alignment, scientific plausibility, conciseness or clarity, and logical coherence (Liu et al., 5 Aug 2025). This extends evaluation beyond fit or graph overlap to include whether a symbolic hypothesis is contextually sensible.

4. Closed-loop symbolic reasoning framework

The benchmark is paired with a unified closed-loop symbolic reasoning system in which LLMs can act as both predictors and judges (Liu et al., 5 Aug 2025). The input is a time series with $\{x_i(t)\}$3 time points $\{x_i(t)\}$4, where $\{x_i(t)\}$5. The framework iterates through proposal generation, verification, history storage, and context selection (Liu et al., 5 Aug 2025).

The workflow consists of seven stages: proposal generation; cleaning and parsing; quantitative verification; qualitative verification; history pool update; context management and selection; and iterative refinement (Liu et al., 5 Aug 2025). In proposal generation, candidates can come from an LLM-as-Predictor or from GP operations such as mutation and crossover on expression trees. In quantitative verification, CDE candidates are coefficient-fitted and simulated with SciPy solve_ivp(..., method=LSODA), BN candidates are simulated as Boolean transition systems, and SCM candidates are scored structurally (Liu et al., 5 Aug 2025). All expressions, reasoning paths, and scores are stored in a history pool $\{x_i(t)\}$6, and a context manager ranks candidates and selects top-$\{x_i(t)\}$7 items for the next round (Liu et al., 5 Aug 2025).

The framework defines four prompting strategies for the LLM-as-Predictor: Naive prompt, Base prompt, Context prompt, and CoT prompt (Liu et al., 5 Aug 2025). The Naive prompt asks for symbolic expressions without context or history; Base adds prior generated expressions from the history pool; Context adds contextual information such as variable descriptions; CoT extends Context with step-by-step reasoning, which is itself judged qualitatively (Liu et al., 5 Aug 2025).

The system is also explicitly multimodal for suitable models. Text-only models receive serialized time series and metadata, while multimodal models can additionally receive visual trajectory plots and temporal embeddings; the paper specifically mentions this for GPT-4o-mini and ChatTS-14B (Liu et al., 5 Aug 2025). The evaluated model set includes Qwen2.5-14B, Llama 3.2-3B, Mathstral-7B, GPT-4o-mini, and ChatTS-14B, while task-specific baselines include PySR, ProGED, ODEformer, LogicGep, PCMCI, LPCMCI, and j-PCMCI+ (Liu et al., 5 Aug 2025).

5. Empirical profile

The empirical findings are organized around a clear asymmetry between task types. On CDEs and SCMs, LLMs are often competitive with or stronger than classical baselines, whereas on Boolean networks they are not (Liu et al., 5 Aug 2025). In the paper’s summary, current LLMs outperform classical baselines on multivariate symbolic regression and causal discovery, but not on Boolean network inference.

For CDEs, the paper reports that in dimension-4 in-distribution evaluation, Qwen2.5 with Base or Context prompting reaches around $\{x_i(t)\}$8 and $\{x_i(t)\}$9 around $$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$0, substantially exceeding PySR on that setting (Liu et al., 5 Aug 2025). For SCMs, ChatTS-14B is reported as the best overall model in Table 4, with F1 $$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$1, precision $$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$2, false discovery rate $$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$3, $$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$4, and SHD $$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$5, while LPCMCI retains the best SHD at $$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$6 (Liu et al., 5 Aug 2025). By contrast, all LLMs trail LogicGep by a large margin on BN inference, although the paper notes that they still achieve positive bookmaker informedness and therefore outperform random guessing (Liu et al., 5 Aug 2025).

The paper’s prompt analysis is equally diagnostic. Context prompting consistently improves results across all three tasks, whereas naïve chain-of-thought is not reliably helpful and can hurt on many CDE settings; ChatTS-14B is the main exception identified in the text (Liu et al., 5 Aug 2025). Longer or more structured reasoning, implemented through iterative refinement, yields moderate but consistent gains rather than dramatic jumps (Liu et al., 5 Aug 2025).

A further result concerns complexity-based model selection. The reported correlations between expression complexity and out-of-distribution $$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$7 are weak or inconsistent and can even become positive in higher dimensions. Without context, the reported correlations are $$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$8 for dimensions 1–4; with context, they are $$x_i^{(t+1)} = f_i(x_1^{(t)}, x_2^{(t)}, ..., x_n^{(t)}), \quad x_i \in \{0, 1\},$$9 (Liu et al., 5 Aug 2025). This suggests that simplicity alone is not a stable selector for scientific symbolic hypotheses in this benchmark.

The hybrid LLM-GP results are among the most consequential. For CDEs, GPLearn alone reaches $f_i$0 and $f_i$1, GPT-4o-mini alone reaches $f_i$2 and $f_i$3, while LLM-as-Predictor + GPLearn reaches $f_i$4 in-distribution and $f_i$5 out-of-distribution, together with the best symbolic proximity (Liu et al., 5 Aug 2025). For BNs, LogicGep remains the strongest standalone method, LogicGep + Judge improves over LogicGep alone, and Predictor + LogicGep improves over pure GPT-4o-mini but remains below LogicGep (Liu et al., 5 Aug 2025). The paper interprets this as evidence that LLMs and GP are complementary rather than substitutable.

6. Benchmark position, limitations, and future trajectory

Within the broader benchmark landscape, SymbolBench occupies the niche of symbolic reasoning over time series rather than static equations, isolated algebraic manipulation, or domain-specific executable QA. ASyMOB concentrates on university-level symbolic mathematics and perturbation robustness across 17,092 symbolic manipulation challenges (Shalyt et al., 28 May 2025). ERBench evaluates equation discovery with a public development set of 10,000 formulas, a secret evaluation set of 1,000 formulas, and robustness axes such as dimensionality, sample size, domain, distribution, and noise (Kahlmeyer et al., 8 Jun 2026). SymPyBench focuses on undergraduate physics problems with parameterized executable Python solutions, 15,045 problems, and robustness metrics such as Consistency Score and Complete Failure Rate (Imani et al., 5 Dec 2025). This suggests that SymbolBench’s distinctive contribution lies in bringing together continuous equations, Boolean update rules, and lagged causal graphs under a single time-series-centered evaluation design (Liu et al., 5 Aug 2025).

The paper is explicit about several limitations. Model coverage is limited and does not include larger or newer open models such as DeepSeek-R1; the search budget is capped at 100 generation epochs and 20 retries per epoch per run, with failed samples rerun once; the hybrid study examines only one LLM and one GP configuration; and there is no systematic analysis of model size or reasoning-strategy variants beyond the tested prompting settings (Liu et al., 5 Aug 2025). Additional validity considerations are also visible in the benchmark construction: SCM ground truths are derived from functional analysis of curated CDEs and Physiome models rather than from unrestricted observational settings, the LLM-as-Judge stage introduces a model-based evaluative layer, and out-of-distribution testing is defined through new initial conditions rather than a broader family of shifts (Liu et al., 5 Aug 2025).

The stated future directions are correspondingly methodological: task-specific scaling of test-time compute and reasoning depth, richer and more knowledge-heavy context, candidate-selection criteria beyond syntactic simplicity to include plausibility, robustness, and causal faithfulness, and broader symbolic targets with more verifiable reasoning traces (Liu et al., 5 Aug 2025). In that sense, SymbolBench is not only a benchmark of current LLM behavior on time-series symbolic reasoning; it is also a proposal for evaluating iterative, context-grounded, and tool-coupled scientific hypothesis generation.