Why Architecture Choice Matters in Symbolic Regression

Abstract: Symbolic regression discovers mathematical formulas from data. Some methods fix a tree of operators, assign learnable weights, and train by gradient descent. The tree's structure, which determines what operators and variables appear at each position, is chosen once and applied to every target. This paper tests whether that choice affects which targets are actually recovered. Three structures are compared, all sharing the same operator and target language but differing in how variables enter the tree; one is strictly more expressive. Across over 12,700 training runs, one structure recovers a target at 100% while another scores 0%, and the ranking reverses on a different target. Expressiveness guarantees that a solution exists in the search space, but not that gradient descent finds it: the most expressive structure fails on targets that a restricted alternative solves reliably. Switching the operator changes which targets succeed; reversing its gradient profile collapses recovery entirely. Balanced (non-chain) tree shapes are never recovered. These findings show that the optimization landscape, not expressiveness alone, determines what gradient-based symbolic regression recovers.

• The paper shows that architecture choice significantly impacts recovery, with distinct outcomes based on variable routing and tree configurations.
• It demonstrates that even more expressive architectures can fail where less expressive ones succeed, emphasizing gradient direction and optimization landscapes.
• The study finds balanced tree shapes consistently hinder recovery, highlighting the need for architecture-aware designs and adaptive ensemble strategies.

Architecture Effects in Gradient-Based Symbolic Regression

Overview and Motivation

The paper "Why Architecture Choice Matters in Symbolic Regression" (2604.23256) presents a systematic investigation into the specific influence of network architecture on the recoverability of symbolic expressions via gradient descent. Rather than conflating architecture with grammar or search protocol, the study isolates architectural variation—specifically variable routing and tree shape—in symbolic regression models based on parameterized operators. The research is executed primarily within the Exp-Minus-Log (EML) operator framework (where each node computes $e^a - \ln b$), though the results extend to analogous operators with gradient asymmetry.

Prior literature rarely disentangles the direct effect of architecture from other factors, with many benchmarks mixing multiple sources of variability. Here, by fixing both operator and target language (elementary functions constructed from EML), the study exposes critical interactions between architecture and target formula, quantifying exact match recovery rates over $12,700$+ training runs.

Architectures and Experimental Setup

Three architectures are evaluated at depth-3 EML trees: Eq.\,6 (softmax routing—most expressive), V16 (sigmoid internal, softmax leaf—symmetric and restricted), and Hybrid (combining V16 internal structure with broader root routing). All architectures can represent all benchmark targets, but differ in how variables $x$ and $y$ flow through the computation graph.

A distinctive aspect is the focus on chains versus balanced tree shapes. Chain types allow a single branch to carry the main computation, whereas balanced trees distribute computation equally between child subtrees. The experimental protocol employs Adam (lr=0.01) to fit the architectures on a dense grid, uses a staged training and weight-hardening schedule, and counts exact match recovery via stringent RMSE thresholding on test data.

Architecture--Target Interaction and Recovery Rates

The results reveal dramatic architecture--target dependent recovery outcomes. Eq.\,6 achieves $100\%$ exact recovery on two targets but $0\%$ on others; V16 is a generalist, achieving up to $99.6\%$ recovery consistently across chain shapes but never dominating all targets. Hybrid is highly specialized, favoring RL chain shapes (near $96\%$). Critically, expressiveness does not guarantee optimization success: Eq.\,6, strictly more expressive than V16, fails entirely on targets that V16 recovers.

Figure 1: Recovery profiles across architectures, tree shapes, and operators, emphasizing sharp outcome disparities and architecture specialization.

This matrix demonstrates that the optimization landscape induced by variable routing determines practical recoverability, not the representational capacity alone.

Operator Swaps and Gradient Direction Effects

Two alternate operators—SML ($\sinh(a) - \arctan(b)$, left amplifier) and RML ($\arctan(a) - \sinh(b)$, right amplifier)—are examined to generalize findings. Under SML, Eq.\,6 inverts its shape preferences: LR drops from $12,700$0 to $12,700$1, and RL/RR rise to $12,700$2. V16 maintains high rates ($12,700$3) in chains under SML and collapses under RML. Operator-specific gradient direction is thus tightly coupled to architecture–target compatibility.

Recovery Impossibility in Balanced Trees

Balanced tree shapes are universally hard, with $12,700$4 recovery in every architecture–operator–leaf variant across $12,700$5 trials. This persists under all tested hyperparameter configurations, suggesting an inherent optimization barrier. The contrast with chain shapes points to distributed gradient dilution in balanced trees, where neither branch receives the concentrated signal necessary for successful symbolic recovery.

Gradient Measurements and Optimization Landscape Analysis

Leaf parameter gradient norms $12,700$6 and $12,700$7 are measured mid-training. For Eq.\,6, variable gradient dominance ($12,700$8 vs. $12,700$9) aligns perfectly with recovery outcomes: a high gradient on the correct variable predicts success. SML operator swaps corroborate directionality: gradient ratio reversals correspond to sharp changes in recoverability.

Figure 2: Gradient ratio trajectories during training, distinguishing successful from failed recovery cases by early gradient dominance reversal.

V16, due to symmetric variable access, manifests narrower and stable ratios, consistent with broader recoverability regardless of target shape.

Implications and Extensions

These findings highlight an underappreciated dimension in gradient-based symbolic regression: the landscape geometry induced by architecture and variable routing can create insurmountable barriers to exact recovery even when the formula is representable. The implication for practical SR systems is that selecting a "one-size-fits-all" architecture can severely constrain performance, and architecture-aware ensemble or adaptive schemes are required.

Theoretical implications extend to neural architecture search (NAS), where optimization landscape geometry must be considered alongside expressiveness. Results also stress the need for loss-landscape analysis in balanced trees, and motivate further study into multiplicative and deeper architectures.

Conclusion

Gradient-based symbolic regression cannot rely exclusively on expressiveness; the optimization landscape induced by architecture is decisive in determining which targets are recovered. Ensembles and adaptive architectures offer viable strategies, and future work should extend analysis to more complex operators, tree shapes, and benchmark formulae, as well as undertake mechanistic loss-landscape scrutiny to rigorously characterize architecture–target interactions.