Symbolic Regression: Analytic Model Discovery

• Symbolic regression is the process of discovering closed-form analytic expressions from data by exploring a vast, combinatorial space of mathematical models.
• It employs genetic programming, Bayesian frameworks, and deep learning techniques to optimize both functional form and continuous parameters.
• Despite its potential in scientific discovery and interpretable machine learning, symbolic regression faces NP-hard challenges that spur ongoing research in efficient heuristics and benchmarking.

Symbolic regression (SR) is the problem of identifying closed-form analytic expressions that best fit a dataset, given a predefined library of operators and functions. Unlike standard regression, which assumes a fixed model structure, SR searches over the combinatorial space of mathematical expressions, aiming to jointly optimize both functional form and continuous parameters for the highest explanatory or predictive power. SR is of significant importance in scientific discovery, interpretable machine learning, and inverse modeling, but is computationally challenging: it is provably NP-hard in the general case and necessitates sophisticated algorithmic strategies to be tractable on problems of practical size (Tohme et al., 2022, Song et al., 22 Apr 2024, Virgolin et al., 2022).

1. Problem Definition and Computational Hardness

Symbolic regression seeks

$$f^* = \operatorname{arg\,min}_{f \in S} \sum_{i=1}^N \left[ f(x_i) - y_i \right]^2$$

where $S$ denotes the set of all expressions that can be formed from a user-specified grammar of functions (e.g., arithmetic, transcendental, and other operators). The search space is combinatorially large, admitting arbitrary compositions and nestings of primitives, and grows super-exponentially with expression size. SR is NP-hard, as shown by reductions to classic hard problems such as the degree-constrained Steiner tree and unbounded subset sum (Song et al., 22 Apr 2024, Virgolin et al., 2022). The core complexity arises from two sources:

• Discrete structure: selecting the optimal tree-structured composition of operators,
• Continuous parameters: fitting constants embedded in the expression.

These results imply that unless P=NP, there is no polynomial-time algorithm guaranteed to find the globally optimal symbolic expression for arbitrary operator sets and data, and thus all practical SR implementations rely on heuristics or are constrained to tractable subspaces (Virgolin et al., 2022, Song et al., 22 Apr 2024).

2. Algorithmic Methodologies

2.1 Genetic Programming (GP) and Matrix-based Models

Most traditional methods employ genetic programming (GP), in which populations of candidate expressions, represented as trees or alternative encodings (e.g., integer matrices in GSR), are evolved via crossover, mutation, and selection (Tohme et al., 2022). The GSR algorithm demonstrates a distinctive variant, where the optimization is formulated as minimizing

$\sum_{i=1}^N \left[ f(x_i) - g(y_i) \right]^2$

with both $f$ and $g$ expressed as sparse linear combinations of basis functions, and the sparse Lasso problem solved via ADMM. GSR uses a matrix-based representation for basis functions, enabling straightforward genetic operators and robust search, and achieves strong empirical recovery rates on standard and challenging benchmarks (Tohme et al., 2022).

2.2 Random Search Baseline

Uniform random global search (SRURGS) is a conceptually simple yet robust baseline, establishing the use of pure random sampling over all expressions up to a complexity bound and providing a "null" benchmark. SRURGS is especially robust in highly complex expression spaces where local search methods struggle, although it converges slowly in simple settings (Towfighi, 2019).

2.3 Bayesian and Information-theoretic Models

Bayesian symbolic regression frameworks use hierarchical priors over expression trees, additive models, and reversible-jump MCMC to sample from the posterior over models and parameters (Jin et al., 2019). These methods allow explicit incorporation of prior domain knowledge (e.g., preference for certain operators or features), directly penalize complexity, and typically achieve more concise and accurate fits than standard GP for structured problems.

2.4 Multi-modal, Deep, and Neural SR

Recent advances target scalability and data efficiency by employing deep generative models (e.g., sequence-to-sequence transformer architectures, set transformers), embedding invariances in both data and expression encodings, and leveraging pretraining on synthetic corpora (Li et al., 28 Feb 2024, Li et al., 2022, Holt et al., 2023). These models unify prior approaches under a probabilistic generative framework, support end-to-end symbolic prediction from data, and can be sampled or fine-tuned via policy gradient or PQT-based refinement.

2.5 Reinforcement Learning and Flow-based SR

Reinforcement learning (RL) formulations cast SR as a sequential decision process, where the action space encompasses grammar-consistent expression building steps, and rewards correspond to normalized error or information metrics. Advanced RL-based approaches integrate auxiliary gates (e.g., for noise-robust variable selection (Sun et al., 2 Jan 2025)) and entropy bonuses to balance exploration and exploitation. Generative flow networks (GFlowNet-SR) traverse expression DAGs to model distributions over high-reward symbolic forms, outperforming RL and GP in noisy regimes (Li et al., 2023).

3. Benchmarks, Metrics, and Community Standards

Recent efforts emphasize comprehensive, standardized benchmarking (e.g., SRBench 2.0 (Aldeia et al., 6 May 2025)). Key metrics include:

• Accuracy: RMSE, mean absolute error, $R^2$ on held-out data.
• Model complexity: tree size/depth, number of nodes/basis functions, or coding length.
• Energy and resource usage: e.g., compute costs tracked via standardized energy metrics.
• Pareto-optimality: trade-off frontiers between accuracy and complexity.

To address inconsistency, SRBench 2.0 specifies uniform APIs, resource/time limits, rigorous cross-validation, and explicit deprecation criteria.

Metric	Definition	Notes
RMSE	$\sqrt{\frac{1}{N}\sum_{i} (y_i-\hat y_i)^2}$	Primary accuracy metric
$R^2$	$1 - SS_{res}/SS_{tot}$	Used for per-task success
Model Size	Node count or coding length	Complexity–parsimony trade
Energy (kWh)	Measured/extrapolated per run	Resource-awareness standard

Benchmarks cover both "black-box" regression datasets and proctored scientific laws, supporting robust cross-method and cross-problem comparisons.

4. Applications and Domain-specific Adaptations

SR has been successfully deployed in various scientific contexts, including:

• LHC and QFT phenomenology: Automated recovery or parametrization of analytic results and approximations to unknown or nonperturbative structure functions (e.g., Drell-Yan process), achieving high accuracy and compactness using Pareto-optimized GP (Morales-Alvarado et al., 10 Dec 2024).
• High-dimensional and multi-variable systems: Approaches like ScaleSR and decomposable neuro-symbolic SR decompose SR into sequential or modular problems, improving tractability and interpretability in settings with tens of variables (Chu et al., 2023, Morales et al., 6 Nov 2025).
• Physics and chemistry: Hybrid MINLP-based SR supports exact recovery of known physical laws, with dimensional-constraint enforcement to ensure physical plausibility (Austel et al., 2020).
• Particle physics model exploration: Symbolic surrogates accelerate Bayesian inference and sensitivity analysis in beyond-standard-model contexts, enabling gradient-based MCMC and outperforming neural networks in extrapolation and globality (AbdusSalam et al., 23 Oct 2025).

5. Strengths, Limitations, and Open Challenges

Strengths:

• Explicit, interpretable closed-form model discovery; not reliant on black-box approximators.
• Recover known or plausible scientific laws from empirical data.
• Methods exist for incorporating prior knowledge, handling noise, scaling to more variables, and regularizing complexity.

Limitations:

• SR is NP-hard in the general case; search is intractable for unconstrained expression spaces.
• Scalability to high dimensions, large operator sets, or deep/nested expressions remains a challenge.
• Robustness to noise varies; strong performance in high-noise requires specialized architectures (e.g., gating RL modules or GFlowNets).
• Selection of hyperparameters (e.g., model complexity penalization, search depth, or population size) can strongly affect performance and reproducibility.
• Expressions may become unwieldy/overfit without explicit parsimony constraints or complexity-informed regularization.

Open challenges and trends:

• Scaling laws (e.g., transformer performance vs. compute) are beginning to be mapped for SR, mirroring trends in language modeling (Otte et al., 30 Oct 2025).
• Unified benchmarking and sustainable infrastructure for reproducibility, parameter-free operation, and energy efficiency (Aldeia et al., 6 May 2025).
• Advanced modalities (e.g., image-based SR, multi-modal fusion, probabilistic/uncertainty quantification) are areas of active research (Li et al., 2022, Li et al., 28 Feb 2024).

6. Future Directions

Ongoing research is focused on:

• Improving scalability to more complex and high-dimensional tasks via composite, hybrid, or modular SR frameworks.
• Integrating domain knowledge formally and automatically, including units, constraints, and invariances.
• Exploring universal scaling behavior and systematic transfer learning regimes for pre-trained symbolic models (Otte et al., 30 Oct 2025).
• Developing comprehensive, interpretable toolchains for model inspection, candidate set exploration, and synthesis of novel scientific hypotheses (e.g., via e-graphs (Franca et al., 29 Jan 2025)).
• Extending uncertainty quantification, active learning, and multi-output symbolic regression to support scientific workflows.

Symbolic regression thus remains a fertile intersection of theory, algorithm design, and domain-scientific application, demanding continued advances at the interface of combinatorial optimization, probabilistic modeling, and scalable machine learning.