SymbolNet: Neural-Symbolic Models and Compression

• SymbolNet is a family of neural-symbolic frameworks characterized by dynamic pruning and explicit symbolic composition for efficient regression and compression.
• One approach employs adaptive MLP layers with trainable unary and binary operators to jointly optimize expression complexity and prediction accuracy in high-dimensional tasks.
• Another variant, MetaSymNet, uses a tree-structured network to evolve symbolic expressions through real-time structural adaptation and operator selection.

SymbolNet is an overloaded term in the arXiv literature rather than a single canonical model. In the most specific contemporary usage, it denotes a neural symbolic regression framework for compression that performs dynamic pruning of model weights, input features, and mathematical operators in a single training process (Tsoi et al., 2024). Closely related work uses the name for a tree-like symbolic regression network with adaptive architecture and activation functions, introduced as MetaSymNet and explicitly described as “aka SymbolNet” (Li et al., 2023). Earlier, the same label also appears as an alias for the Deep Symbolic Network, a hierarchical graph of symbols and links intended as a white-box alternative to deep neural networks (Zhang et al., 2017), and for the Symbolic Tensor Neural Network formalism, a BNF-based symbolic specification of tensor computation graphs and their dual gradient flows (Skarbek, 2018). This multiplicity of meanings is central to understanding the topic: “SymbolNet” designates a family of symbolic or neural-symbolic formalisms whose common concern is explicit structure, interpretability, and symbolic composition, but whose concrete architectures, objectives, and deployment regimes differ substantially.

1. Terminological scope and historical usages

The term “SymbolNet” has been attached to at least four distinct constructs in the cited literature.

Usage	Paper	Core characterization
SymbolNet	(Tsoi et al., 2024)	Neural symbolic regression with adaptive dynamic pruning for compression
MetaSymNet (aka SymbolNet)	(Li et al., 2023)	Tree-like symbol network with adaptive architecture and activation functions
DSN or SymbolNet	(Zhang et al., 2017)	Deep hierarchical symbolic network of symbols and links
STNN, aka SymbolNet	(Skarbek, 2018)	Symbolic tensor-network formalism specified by BNF and DAG constraints

The 2024 SymbolNet paper defines the term narrowly as a model compression technique for high-dimensional symbolic regression on custom hardware such as FPGAs, with simultaneous optimization of training loss and expression complexity (Tsoi et al., 2024). By contrast, MetaSymNet uses a rooted directed acyclic graph represented as a tree, in which internal nodes are meta-neurons that evolve into unary operators, binary operators, or direct variable copies, and the final output is an extracted closed-form expression (Li et al., 2023).

The earlier Deep Symbolic Network literature uses “DSN or SymbolNet” to denote a broad white-box knowledge representation framework in which symbols correspond to “coarse-grained” pieces of matter or concepts and are connected by composition, inheritance, dependence, causality, abstraction, and higher-order links (Zhang et al., 2017). The Symbolic Tensor Neural Network literature uses “SymbolNet” in yet another sense, as a grammar-driven symbolic notation for CNN-like DAGs, modularized by user-defined units and paired with a dual symbolic representation for backpropagation (Skarbek, 2018).

A common misconception is that these papers describe incremental versions of the same architecture. The record instead indicates a naming collision. A plausible implication is that any technical discussion of “SymbolNet” should identify the exact paper and objective before comparing methods.

2. SymbolNet as neural symbolic regression for compression

In the 2024 formulation, SymbolNet is a multi-layer perceptron in which each hidden symbolic layer performs a learnable linear transform followed by a bank of heterogeneous activations composed of unary and binary operators (Tsoi et al., 2024). If the input to layer $\ell$ is $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$, the layer uses weights $$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$ and bias $b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$ to form

$a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$

The pre-activations are then partitioned into channels assigned to $u$ unary operators $f_1,\dots,f_u$ and $b$ binary operators $g_1,\dots,g_b$. The dimensionalities satisfy

$\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$

This construction differs from standard homogeneous MLP layers because operator choice is built into the layer semantics rather than being fixed globally. Operator selection is not performed through a separate combinatorial search; instead, each operator node has a trainable threshold that can switch it off or downgrade it. Unary operators can revert to identity, and binary operators can revert to addition. In that sense, symbolic structure emerges by pruning within a differentiable training loop rather than by population-based equation search (Tsoi et al., 2024).

The paper positions this design against the limitations of genetic programming for datasets with more than $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$0 inputs. It explicitly reports effectiveness on LHC jet tagging with 16 inputs, MNIST with 784 inputs, and SVHN with 3072 inputs (Tsoi et al., 2024). This suggests that the central contribution is not only interpretability but also symbolic regression under high input dimensionality, with an explicit deployment target in custom hardware.

3. Dynamic pruning and adaptive sparsity control

The defining mechanism of the 2024 SymbolNet is single-phase end-to-end dynamic pruning over three modalities: weights, input features, and operators (Tsoi et al., 2024). For each parameter $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$1, a trainable threshold $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$2 induces a mask

$z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$3

For each input feature $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$4, a threshold $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$5 induces

$z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$6

For each unary operator $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$7, a threshold $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$8 replaces

$z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$9

and for each binary operator $$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$0 with threshold $$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$1,

$$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$2

The sparsity schedule is adaptive rather than epoch-based. If the current fraction of pruned elements of type $$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$3 is $$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$4, the decay factor is

$$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$5

with $$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$6 and “in practice $$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$7” (Tsoi et al., 2024). The regularizers are

$$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$8

and the total loss is

$$W^{(\ell)} \in \mathbb{R}^{d_{\ell-1}\times \tilde d_\ell}$$9

with

$b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$0

The significance of this construction lies in how sparsity targets are enforced. The paper states that when $b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$1, the regularizer pushes thresholds upward, and when $b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$2, the decay $b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$3 and the penalty is switched off (Tsoi et al., 2024). A plausible implication is that pruning is regulated as a closed-loop control process around target sparsity ratios rather than as a fixed penalty schedule. The paper also contrasts this with multistage pruning plus fine-tuning, which it describes as often suffering a “bump” in validation loss after pruning and requiring threshold sweeps (Tsoi et al., 2024).

4. Optimization procedure and reported empirical performance

The 2024 SymbolNet training loop initializes weights and thresholds, computes masks for weights, inputs, and operators on each minibatch, performs a masked forward pass, computes task loss and sparsities, forms the adaptive regularizers, and updates all parameters by backpropagation; thresholds are then clipped to their allowed ranges (Tsoi et al., 2024). The reported optimization details are Adam with default $b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$4, $b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$5, learning rate “typically $b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$6–$b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$7,” batch size $b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$8 for LHC and $b^{(\ell)} \in \mathbb{R}^{\tilde d_\ell}$9 for MNIST/SVHN, and epochs “200 (LHC), $a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$0–$a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$1 (MNIST/SVHN)” (Tsoi et al., 2024).

The experimental summary reports three benchmark problems. For LHC jet tagging, the baseline is a DNN $a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$2, 90 % pruned, $a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$3-bits, with $a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$4 and latency $a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$5 ns, while SymbolNet has complexity $a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$6, $a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$7, and latency $a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$8 ns, corresponding to “$a^{(\ell)} = z^{(\ell-1)} W^{(\ell)} + b^{(\ell)}.$9 pp acc, $u$0 speed” (Tsoi et al., 2024). For MNIST, the baseline CNN with 92 % pruning and $u$1 quantization has $u$2 and latency $u$3s, while SymbolNet has mean complexity $u$4, $u$5, and latency $u$6 ns, corresponding to “$u$7 pp acc, $u$8 speed” (Tsoi et al., 2024). For SVHN binary classification (“1” vs “7”), the baseline CNN with 92 % pruning and $u$9 quantization has $f_1,\dots,f_u$0 and latency $f_1,\dots,f_u$1s, while SymbolNet has complexity $f_1,\dots,f_u$2, $f_1,\dots,f_u$3, and latency $f_1,\dots,f_u$4 ns, corresponding to “$f_1,\dots,f_u$5 pp AUC, $f_1,\dots,f_u$6 speed” (Tsoi et al., 2024).

The same paper states that SymbolNet yields compact symbolic expressions whose FPGA implementations consume an order of magnitude fewer LUTs/DSPs and run tens of times faster, with only minor loss in accuracy or even slight improvements in AUC (Tsoi et al., 2024). It also reports that on scientific benchmarks the method “consistently dominates a dense-fine-tuned EQL baseline and is competitive with expensive GP systems (e.g. PySR), yet in a single 1–2 hour GPU run instead of many-hour CPU evolution” (Tsoi et al., 2024). Since the latter comparison is summarized qualitatively in the supplied data, it is most precise to treat it as a characterization of the reported trend rather than as a substitute for benchmark-by-benchmark tabulation.

5. MetaSymNet: tree-structured SymbolNet and the PANGU meta-function

A distinct but closely related use of the name appears in MetaSymNet, described in the supplied material as “aka SymbolNet” (Li et al., 2023). Here, a mathematical expression is represented as a rooted directed acyclic graph, specifically a tree. Each internal node corresponds to an operator neuron, each leaf corresponds to a variable neuron, and the root output is the network prediction $f_1,\dots,f_u$7 (Li et al., 2023). During training, every node is maintained as a meta-neuron whose activation can evolve into a binary operator $f_1,\dots,f_u$8, a unary operator such as $f_1,\dots,f_u$9, $b$0, $b$1, $b$2, or $b$3, or a direct copy of a single input variable $b$4.

The core activation is the PANGU meta-function. At each internal node $b$5,

$b$6

where $b$7 are learned amplitude and bias scalars, $b$8 is a fixed library of function evaluations on the child activations, $b$9 are learned selection logits, and $g_1,\dots,g_b$0 (Li et al., 2023). For leaf node $g_1,\dots,g_b$1,

$g_1,\dots,g_b$2

where $g_1,\dots,g_b$3 are selection logits and $g_1,\dots,g_b$4 (Li et al., 2023).

Training combines MSE with an entropy penalty,

$g_1,\dots,g_b$5

where $g_1,\dots,g_b$6, $g_1,\dots,g_b$7, and $g_1,\dots,g_b$8 is a small coefficient (Li et al., 2023). Optimization alternates between updating amplitudes and biases, updating operator- and variable-selection logits, extracting the current formula by snapping each meta-neuron to the argmax-selected operator or variable, and then re-optimizing constants on the extracted symbolic expression. The stopping rule is reported as repeating “until $g_1,\dots,g_b$9 (e.g. $\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$0)” (Li et al., 2023).

A defining feature is real-time structural adaptation. If a meta-neuron becomes unary, one child is pruned; if it becomes a variable copy, both inputs are pruned; if a variable leaf upgrades to a function, new child nodes are inserted and the tree grows (Li et al., 2023). This growth-and-pruning cycle differs from the 2024 SymbolNet’s threshold-based pruning over a fixed layered scaffold. The two methods therefore share a symbolic-regression objective but differ in representational bias: MetaSymNet is tree-native and operator-centric, whereas the 2024 SymbolNet begins from an MLP-like architecture with symbolic layers and explicit compression targets.

6. Evaluation of MetaSymNet and its relation to broader symbolic-network frameworks

MetaSymNet is evaluated on public symbolic-regression suites: Nguyen (12 formulae), Keijzer (15), Korns (15), Constant (8), Livermore (22), Vladislavleva (8), R (6), Jin (6), Neat (9), AIFeynman (103), and Others (9) (Li et al., 2023). The reported metric is $\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$1 on test data in the no-noise case and average $\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$2 under additive uniform noise of ten levels from 0 % to 10 % of the $\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$3-span, with extrapolation error assessed as $\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$4 inside the training interval versus outside it, specifically “$\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$5” (Li et al., 2023).

The symbolic-regression comparison reports MetaSymNet average $\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$6, with next best DSO at $\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$7, EQL at $\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$8, GP at $\tilde d_\ell = u + 2b, \qquad d_\ell = u + b.$9, and NeSymReS at $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$00 (Li et al., 2023). On Nguyen problems and 100 trials per noise level, MetaSymNet “maintains $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$01 up to 5% noise and outperforms DSO/EQL/GP/NeSymReS at all noise levels” (Li et al., 2023). In comparisons to MLP and SVR, all three models reach $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$02 inside $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$03, but outside $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$04 MetaSymNet has $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$05, while MLP and SVR “plunge to large negative $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$06” (Li et al., 2023). For Nguyen problems with target $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$07, MetaSymNet uses on average 6.75 nodes and 27 parameters, whereas MLP with iterative pruning uses 28.16 nodes and 376 parameters (Li et al., 2023). On 100-point subsets of the 103 AIFeynman formulas, it achieved $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$08 on 85 formulas, with average $z^{(\ell-1)} \in \mathbb{R}^{d_{\ell-1}}$09 (Li et al., 2023).

The broader symbolic-network lineage helps situate these results. Deep Symbolic Networks present a transparent graph of symbols, identifying operators, and probabilistic links for composition, inheritance, dependence, causality, and abstraction, with layer-by-layer discovery driven by singularity detection and clustering (Zhang et al., 2017). Symbolic Tensor Neural Networks provide a BNF grammar for composing decorated unit symbols into DAGs, a module system via UNITDEF, and a dual grammar that mirrors forward units to generate gradient flow for CNN-like models (Skarbek, 2018). These earlier works are not symbolic regression methods in the narrow sense of either MetaSymNet or the 2024 SymbolNet. Nevertheless, they establish a recurrent theme behind the “SymbolNet” label: replacing opaque parameterizations with explicit symbolic structure, whether that structure is a knowledge graph, a syntax for tensor graphs, a dynamically pruned symbolic MLP, or an adaptive expression tree.

This suggests that the most defensible encyclopedic understanding of SymbolNet is not a single architecture but a research motif. Across the cited papers, that motif combines explicit symbolic composition with neural optimization or hierarchical structure, while differing in whether the primary target is knowledge representation, graph specification, formula discovery, or hardware-efficient compression.