Symbolic Feedforward Neural Networks

• Symbolic feedforward neural networks are architectures that embed explicit symbolic computations within standard feedforward designs.
• They realize automata simulation via deterministic and probabilistic constructions, enabling exact state transitions or rule extraction post training.
• Key implementations span DFA emulation, affine-region decomposition, and tree-structured operator networks that enhance neural interpretability.

Searching arXiv for papers on symbolic feedforward neural networks and adjacent neuro-symbolic feedforward formulations. Symbolic feedforward neural networks are feedforward architectures, representations, or derived models in which the forward computation is given an explicit symbolic interpretation rather than being treated solely as a distributed latent transformation. In current arXiv usage, the term covers several non-identical constructions: exact depth-unrolled simulation of deterministic and probabilistic automata, post hoc rule or decision-model extraction from trained feedforward networks, exact symbolic decompositions of piecewise-linear networks into affine regions, and tree-structured symbolic-regression networks whose units are mathematical operators rather than generic activations (Dhayalkar, 16 May 2025). The literature therefore does not define a single canonical formalism; instead, it organizes a family of approaches for making feedforward computation exact, interpretable, or analytically manipulable in symbolic terms (Dhayalkar, 12 Sep 2025).

1. Conceptual scope and main interpretations

The phrase denotes at least four recurring ideas. First, a feedforward network may be symbolic by construction, as in explicit automaton simulators or operator trees. Second, it may be symbolic post hoc, through rule extraction or decision-tree derivation from an already trained network. Third, it may admit a symbolic semantic representation, such as a finite partition of input space into affine regions. Fourth, it may participate in a modular neural-symbolic pipeline in which a feed-forward perception stage interfaces with a symbolic learner or a symbolically synthesized structural scaffold (Kamruzzaman et al., 2010).

Mode of symbolism	Representative papers	Characteristic mechanism
By-construction state machines	(Dhayalkar, 16 May 2025, Dhayalkar, 12 Sep 2025)	Hidden states encode automaton states or state distributions
Post hoc symbolicization	(Kamruzzaman et al., 2010, Seidel et al., 16 Apr 2025)	Rules or decision trees are derived from trained FNNs
Exact symbolic semantics	(Sotoudeh et al., 2019)	Piecewise-linear FNNs become finitely many affine maps on polytopes
Symbolic operator networks	(0808.1378, Li et al., 2023, Boddupalli et al., 5 May 2026)	Units are elementary functions or learned symbolic operators

This breadth matters because several adjacent papers are explicitly careful about what they are not. FF-NSL is described as a feed-forward neural perception front-end plus a symbolic learner rather than a unified symbolic feedforward architecture (Cunnington et al., 2021). G-SSNNs use a synthesized symbolic program to inject graph structure before graph-neural processing, which places them closer to symbolically synthesized inductive bias than to a classical symbolic multilayer perceptron (Whitehouse, 2023). The operator-theoretic kernel paper is relevant only in an indirect, analytic sense, treating “symbolic” as formal operator-level characterization rather than symbolic reasoning in the logic or automata sense (Jorgensen et al., 2023).

2. Deterministic finite-state constructions

A central formalization treats feedforward neural networks as exact simulators of deterministic finite automata on bounded-length inputs. Dhayalkar’s construction begins from a DFA

$\mathcal{A} = (Q, \Sigma, \delta, q_0, F),$

with extended transition

$$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$

and indicator

$$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$

For any DFA and any finite input length $T$, the paper states that there exists a feedforward neural network $f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$ such that $f_\theta(x)=\phi(x)$ exactly for every $x\in \Sigma^T$. The construction is explicit rather than existence-based: states are encoded either one-hot as $e_i\in\mathbb{R}^n$ or in binary as $b_i\in\{0,1\}^{\lceil \log_2 n\rceil}$, symbols are one-hot encoded, and one layer corresponds to one automaton transition step (Dhayalkar, 16 May 2025).

In the one-hot ReLU construction, the transition module is

$$g:\mathbb{R}^{n+k}\to \mathbb{R}^n,\qquad g([e_i;u_j]) = e_{\delta(q_i,s_j)}.$$

The network computes

$$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$0

followed by an acceptance readout

$$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$1

where $$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$2 is the accepting-state indicator. The appendix describes a depth-$$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$3 network, and the most explicit size statement is that hidden layer width $$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$4 suffices for transition lookup on the concatenated one-hot state-symbol space. The paper’s “linear separability” claim is therefore a finite-domain selector construction: each valid state-symbol codeword can be isolated by affine inequalities, allowing a two-layer ReLU MLP to realize the DFA transition table exactly (Dhayalkar, 16 May 2025).

The threshold-network variant replaces one-hot state vectors by binary codes

$$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$5

and computes

$$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$6

Because each next-state bit is a Boolean function of previous-state bits and current-symbol bits, a two-layer threshold network computes the transition exactly. This yields the paper’s compression claim: $$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$7 The paper describes this as “exponential compression” under binary threshold activations, although it does not provide a matching lower bound (Dhayalkar, 16 May 2025).

The same construction is extended to latent symbolic structure through Myhill–Nerode equivalence. If $$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$8 computes the one-hot DFA state reached by string $$\hat{\delta}(q_0, x) = \delta(\dots \delta(\delta(q_0,s_1),s_2)\dots,s_T),$$9, then an injective state embedding $$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$0 yields

$$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$1

so that strings in the same equivalence class map to the same latent vector, while distinct DFA states remain separated. Corollary 2.1 further claims an $$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$2-dimensional continuous embedding with state separation by $$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$3, justified by a Johnson–Lindenstrauss-style projection on the finite state set. This gives the paper its strongest neural-symbolic reading: hidden vectors can be interpreted as exact realizations of regular-language residual structure, not merely as probing-detected correlates (Dhayalkar, 16 May 2025).

3. Probabilistic finite-state constructions

A probabilistic extension replaces one-hot state tracking by simplex-valued state propagation. In the PFA formulation

$$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$4

each symbol $$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$5 has a row-stochastic transition matrix $$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$6, where

$$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$7

The hidden state is a row-vector distribution

$$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$8

updated by

$$\phi(x)= \begin{cases} 1,& \hat{\delta}(q_0,x)\in F,\ 0,& \text{otherwise}. \end{cases}$$9

Acceptance is the final mass on accepting states,

$T$0

The construction is entirely feedforward and entirely symbolic: each hidden coordinate is the marginal probability of an automaton state, each matrix row is a transition distribution, and no explicit softmax is required at inference if the matrices are already row-stochastic (Dhayalkar, 12 Sep 2025).

The paper states a bidirectional equivalence theorem. In one direction, every PFA induces a symbolic feedforward network with symbol-selected row-stochastic transition matrices shared across time, and the network output equals the PFA acceptance probability for every string. In the reverse direction, every feedforward network of this specific form corresponds to a PFA $T$1 with

$T$2

This is stronger than universal approximation: within the constrained architecture class, the feedforward network and the PFA are representationally identical (Dhayalkar, 12 Sep 2025).

The probabilistic construction also formalizes two extensions. First, “probabilistic subset construction” interprets

$T$3

as a differentiable analogue of NFA subset propagation, with simplex-valued beliefs replacing Boolean reachable sets. Second, probabilistic $T$4-closure is handled by iterating a row-substochastic matrix $T$5: $T$6 with $T$7, converging in at most $T$8 steps to a fixed point $T$9 according to the authors’ lemma (Dhayalkar, 12 Sep 2025).

Size and learnability are treated explicitly. Storing one $f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$0 transition matrix per symbol yields parameter complexity $f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$1, independent of sequence length $f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$2. Proposition 5.1 states that if training labels are ground-truth PFA acceptance probabilities, then a symbolic feedforward network with trainable $f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$3 can be optimized to arbitrarily small empirical error on the dataset. The paper is careful that this is a realizability-and-optimization statement, not a PAC-style sample-complexity or identifiability theorem. It also notes that thresholded PFA languages may be non-regular, unlike the DFA setting (Dhayalkar, 12 Sep 2025).

4. Symbol extraction and symbolic analysis of trained feedforward networks

A separate tradition starts from an ordinary feedforward network and derives a symbolic model after training. REANN begins with a standard three-layer backpropagation ANN, then performs constructive hidden-unit growth, pruning of redundant inputs and connections, discretization of hidden activations by heuristic clustering, and rule extraction via the REx algorithm. The extracted output is an unordered set of IF–THEN rules plus a default rule. On the benchmark problems reported in the paper, the extracted rule sets are small: breast cancer uses 2 rules with training accuracy 96.28% and testing accuracy 93.43%; iris uses 3 rules with 98.67% training and 97.33% testing; diabetes uses 2 rules with 72.14% training and 76.56% testing; season uses 5 rules with 100% training and 100% testing (Kamruzzaman et al., 2010).

The discretization step is the critical bridge from numeric hidden activations to symbolic conditions. Hidden activations are grouped into clusters with threshold $f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$4, using updates such as

$f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$5

and each continuous activation is replaced by the nearest cluster center. This converts hidden-unit behavior into a finite symbolic alphabet. The final rules are stated directly in terms of input attributes and class labels, such as petal-length and petal-width thresholds for iris classification. The symbolism here is explicitly post hoc rather than intrinsic: the ANN remains a conventional numeric feedforward model during learning (Kamruzzaman et al., 2010).

A more recent extraction framework derives “equivalent symbol-based decision models” from trained FNNs by tracing sufficiently relevant weighted activations backward from the output layer. The target symbolic formalism is primarily a decision tree whose nodes are tuples $f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$6, with information state

$f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$7

tests over information items, and edges associated with assignments $f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$8. The method computes forward activations for a selected input, then includes only those hidden neurons whose products $f_\theta:\mathbb{R}^{Tk}\to \mathbb{R}$9 make sufficiently relevant contributions to downstream decisions. For deeper networks, edges are recursively refined into subpaths layer by layer, and multiple decision paths are merged into a hierarchical tree. The prototype operates on Keras .h5 models, supports dense networks and CNNs converted to equivalent feedforward form, and is presented as a feasibility demonstration rather than a formally quantified fidelity benchmark (Seidel et al., 16 Apr 2025).

A third line of work gives a symbolic representation for piecewise-linear feedforward networks themselves: $f_\theta(x)=\phi(x)$0 where the $f_\theta(x)=\phi(x)$1 partition the domain into convex polytopes and each $f_\theta(x)=\phi(x)$2 is affine. For networks composed of sequentially applied FullyConnected, 2DConvolution, BatchNorm, ReLU, and MaxPool layers, the symbolic representation is computable over a restriction domain of interest $f_\theta(x)=\phi(x)$3, yielding $f_\theta(x)=\phi(x)$4. This converts network analysis into reasoning over finitely many affine pieces and is then used for exact weakest preconditions on inputs, strongest postconditions on outputs, bounded model checking of neural controllers, and patching via masking networks. In this formulation, symbolism is neither post hoc explanation nor a symbolic operator library; it is an exact semantic decomposition of a trained piecewise-linear FNN into affine cases (Sotoudeh et al., 2019).

5. Symbolic reasoning and symbolic regression architectures

Several architectures are symbolic at the unit level rather than only at the semantic or extracted level. In algebraic reasoning, a deep feedforward network is trained to guide rewrite-rule application from human-like reasoning traces. Expressions are encoded as reduced partial trees, optionally centralized to include parent context, enriched with symbolic association vectors

$f_\theta(x)=\phi(x)$5

and rule-application records $f_\theta(x)=\phi(x)$6. The network predicts the next $f_\theta(x)=\phi(x)$7 pair with an averaged Softmax layer. Depth matters sharply: a 5-hidden-layer feedforward network substantially outperforms shallower variants, and the best configuration, C-RPT2 + SAV + RAR, reaches a 4.6% step-level reasoning error rate on the reported test set (Cai et al., 2017).

Symbolic-regression architectures make the symbolic status of hidden units even more direct. The Symbolic Function Network models a target function as a tree whose leaves are input variables and whose internal nodes are elementary functions

$f_\theta(x)=\phi(x)$8

The structure is built incrementally by forward, backward, forward-backward, or reduced-random-set variants; parameters are trained by a tree-propagation steepest-descent rule derived from

$f_\theta(x)=\phi(x)$9

The model is feedforward in the acyclic sense rather than the conventional layered-MLP sense, and the paper emphasizes sparsity and symbolic readability of the resulting tree expression (0808.1378).

MetaSymNet also uses a tree-like feedforward graph, but each node carries a differentiable operator-selection mechanism. A PANGU node computes

$x\in \Sigma^T$0

where the $x\in \Sigma^T$1 are candidate operator outputs and the $x\in \Sigma^T$2 arise from a softmax over learnable logits. Leaf nodes similarly perform soft variable selection. The architecture then expands or contracts depending on whether a node resolves to a unary operator, a binary operator, or a variable. Across more than 10 public datasets comprising 222 formulas, the paper reports average $x\in \Sigma^T$3 of $x\in \Sigma^T$4, ahead of DSO, EQL, GP, and NeSymReS in its table, and it reports far smaller average structural complexity than an MLP under the same goodness of fit criterion (Li et al., 2023).

SymANNTEx frames symbolic regression as a deep feedforward network whose hidden units are symbolic primitives. Each operational layer outputs four symbolic features, including linear combinations, generalized power products, simple products, and operator combinations involving $x\in \Sigma^T$5, $x\in \Sigma^T$6, and $x\in \Sigma^T$7. The overall mapping is

$x\in \Sigma^T$8

trained with MAE plus symbolic-layer-specific regularizers and finalized by SymPy-based simplification and AIC model selection. The paper reports exact recovery of Lorenz and Rössler, near recovery for Takens–Bogdanov and the pendulum, and approximate surrogates for more difficult cases such as Arrhenius kinetics and Chua when the true primitive is absent from the operator library (Boddupalli et al., 5 May 2026).

6. Expressive boundaries, empirical patterns, and adjacent directions

A recurrent misconception is that symbolic feedforward networks are claimed to be unrestricted symbolic reasoners. The strongest automata-theoretic result argues the opposite: exact constructive simulation is finite-state and length-bounded. In the DFA setting, the positive theorem requires depth $x\in \Sigma^T$9 for strings of length $e_i\in\mathbb{R}^n$0, while the negative theorem states that a fixed finite-depth, finite-width feedforward network cannot exactly recognize languages such as

$e_i\in\mathbb{R}^n$1

for unbounded input lengths. The paper summarizes this boundary as “Neural FSMs simulate exactly and only regular languages” (Dhayalkar, 16 May 2025).

The PFA framework broadens expressivity under threshold semantics, since thresholded PFA languages need not be regular, but it retains analogous structural limits: depth still grows with sequence length, the alphabet and state set remain finite, and the equivalence theorem applies only to row-stochastic, symbol-conditioned, linear-transition architectures rather than arbitrary multilayer perceptrons (Dhayalkar, 12 Sep 2025). This suggests that symbolic feedforward computation is most rigorous when the symbolic process itself is finite-state or otherwise explicitly constrained.

Empirically, the constructive automata papers present proof-of-concept rather than emergent-symbolism experiments. The DFA paper reports perfect parity accuracy for sequence lengths $e_i\in\mathbb{R}^n$2–$e_i\in\mathbb{R}^n$3, then $e_i\in\mathbb{R}^n$4 at $e_i\in\mathbb{R}^n$5, $e_i\in\mathbb{R}^n$6 at $e_i\in\mathbb{R}^n$7, and $e_i\in\mathbb{R}^n$8 at $e_i\in\mathbb{R}^n$9, attributing the degradation to optimization rather than theory. Its ReLU transition-lookup experiments achieve perfect accuracy across all tested DFAs up to 8 states and 3 symbols, while binary-encoded threshold-like training degrades as $b_i\in\{0,1\}^{\lceil \log_2 n\rceil}$0 grows. The PFA paper reports perfect accuracy in a smaller learnability setting and $b_i\in\{0,1\}^{\lceil \log_2 n\rceil}$1 mean accuracy in a larger one, while carefully distinguishing empirical behavior recovery from formal identifiability (Dhayalkar, 16 May 2025).

Several adjacent works delimit the topic by contrast. The parameter-conditioned random-feature FFNN for logistic and Stuart–Landau dynamics is explicitly described as not proposing a symbolic method, not attempting exact equation recovery, and not including interpretability analysis, even though it is relevant as a baseline precursor for later symbolic extraction workflows (Sakaguchi, 2024). FF-NSL and G-SSNNs likewise show that feed-forward neural modules can interface productively with symbolic learning or symbolic synthesis, but they preserve a modular separation between continuous perception and downstream symbolic induction or graph-structured symbolic bias rather than turning the feedforward network itself into a fully symbolic machine (Cunnington et al., 2021). The operator-theoretic kernel formulation, finally, provides a symbolic-analytic calculus of feedforward composition through composition operators, RKHS structure, and spectral iteration, but it does not address symbolic reasoning or symbolic rule execution in the neuro-symbolic sense (Jorgensen et al., 2023).

Taken together, the literature supports a precise but plural understanding of symbolic feedforward neural networks. In one regime, they are exact feedforward realizations of automata-like symbolic state transitions. In another, they are ordinary feedforward networks whose behavior is rendered symbolic through rule extraction, decision-tree derivation, or affine-region decomposition. In a third, they are tree-like or operator-based architectures whose units are themselves symbolic functions. The common denominator is not a single architecture family but an insistence that feedforward computation can be made structurally interpretable, symbolically expressible, or formally exact—provided the symbolic regime, representational assumptions, and expressive limits are made explicit.