SymFormer: Structural Biases in Transformers

• SymFormer is a family of transformer models that integrates explicit structural and symmetry priors for diverse tasks including symbolic regression, circuit structure learning, PDE solving, and TB diagnosis.
• In symbolic regression, it uniquely predicts both formula symbols and constants via an encoder–decoder architecture with local gradient refinement, achieving near-perfect R² scores.
• Other variants employ tree-relative self-attention, grammar-constrained decoding, and symmetry-aware mechanisms to improve inference efficiency and accuracy in their specific domains.

Searching arXiv for papers using the term "SymFormer" to ground the article. SymFormer is a model name used in multiple arXiv works for transformer-based architectures that encode domain structure directly into generation or perception pipelines. Existing uses include end-to-end symbolic regression, grammar-constrained generation of probabilistic-circuit structures inside SymCircuit, structure-aware symbolic PDE solving inside SymPlex, and bilateral-symmetry-aware chest X-ray diagnosis (Vastl et al., 2022, Ju, 20 Mar 2026, Park et al., 3 Feb 2026, Liu et al., 2023). The shared label does not denote a single canonical architecture; rather, it has been attached to several systems that replace or augment vanilla sequence modeling with explicit inductive biases such as joint symbol-and-constant prediction, tree-relative self-attention, grammar masking, traversal-aware positional encoding, or symmetry-constrained attention.

1. Scope of the name

A common source of confusion is that "SymFormer" refers to several distinct models rather than one standardized design. In the cited literature, the name appears in four technically separate settings: symbolic regression, tractable probabilistic circuits, symbolic PDE solving, and computer-aided tuberculosis diagnosis (Vastl et al., 2022, Ju, 20 Mar 2026, Park et al., 3 Feb 2026, Liu et al., 2023).

Paper	Domain	Defining mechanism
"SymFormer: End-to-end symbolic regression using transformer-based architecture" (Vastl et al., 2022)	Symbolic regression	Encoder–decoder transformer that predicts formula symbols and corresponding constants simultaneously
"SymCircuit: Bayesian Structure Inference for Tractable Probabilistic Circuits via Entropy-Regularized Reinforcement Learning" (Ju, 20 Mar 2026)	Probabilistic circuit structure learning	Grammar-constrained autoregressive Transformer with tree-relative self-attention
"SymPlex: A Structure-Aware Transformer for Symbolic PDE Solving" (Park et al., 3 Feb 2026)	Symbolic PDE solving	Structure-aware Transformer with tree-relative self-attention and grammar-constrained autoregressive decoding
"Revisiting Computer-Aided Tuberculosis Diagnosis" (Liu et al., 2023)	Chest X-ray TB diagnosis	Unified two-head network with Symmetric Search Attention and Symmetric Positional Encoding

This multiplicity is not merely terminological. Each SymFormer instantiation is tied to a different object of prediction: formula tokens and real constants, probabilistic-circuit structures, symbolic expression trees for PDEs, or chest X-ray feature maps and diagnosis heads. A plausible implication is that the name functions as a family resemblance marker for structure-aware or symmetry-aware Transformer variants rather than as a unique model specification.

2. SymFormer for symbolic regression

In symbolic regression, SymFormer is an encoder–decoder transformer that, in one forward pass, takes a bag of observed $(x,y)$ pairs and produces an autoregressive sequence of formula symbols together with a set of real-valued constants (Vastl et al., 2022). The encoder first projects each $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$ by a learned affine layer into a latent vector of dimension $D_{\mathrm{enc}}$. It then applies several induced-set cross-attention blocks, following Lee et al. 2019, to compress the variable number of points into a fixed set of $M$ seed vectors, and a final cross-attention with a small set of $K$ trainable queries yields a $K\times D_{\mathrm{enc}}$ matrix $H_{\mathrm{enc}}$ summarizing the dataset. The decoder is autoregressive, with masked self-attention over past tokens, and at each step receives the embedding of the previously generated symbol plus positional encoding and the latest real-valued constant after projection into the same model dimension. Two heads are placed on top of the final decoder layer: a symbol head, implemented as linear plus softmax over vocabulary $V$, and a constant head, implemented as a linear layer outputting a scalar when the symbol head chooses a constant token.

The paper’s main architectural departure from earlier transformer-based symbolic regression is joint symbol-and-constant prediction. Formulae are represented in prefix notation, and every constant is replaced by a token $C_E$ with exponent index $E\in[-10,\ldots,+10]$. A true constant $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$0 is decomposed approximately as

$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$1

with $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$2 and $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$3, where $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$4. The decoder predicts $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$5 and outputs $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$6 as the real constant. Small integers $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$7 are directly in the vocabulary so that exact small constants need not pass through continuous regression.

The probabilistic factorization is defined over both token sequence and constants:

$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$8

At constant positions, $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$9 is taken to be a Gaussian with mean given by the constant head. Training uses teacher forcing with a symbolic cross-entropy loss and a constant regression loss,

$D_{\mathrm{enc}}$0

where $D_{\mathrm{enc}}$1 is ramped from $D_{\mathrm{enc}}$2 to $D_{\mathrm{enc}}$3 over the first $D_{\mathrm{enc}}$4 steps via a cosine schedule. Gaussian noise is added to each ground-truth constant during training and annealed to $D_{\mathrm{enc}}$5. The synthetic training corpus comprises $D_{\mathrm{enc}}$6 univariate and $D_{\mathrm{enc}}$7 bivariate formulas, simplified via SymPy and filtered if too large, numerically singular, or constant. Training uses Adam, batch size $D_{\mathrm{enc}}$8, a Vaswani-style learning-rate schedule divided by $D_{\mathrm{enc}}$9, and $M$0 epochs on $M$1A100 GPUs in approximately $M$2.

Inference is staged. Symbolic search uses Top-$M$3 sampling with $M$4 to generate candidate token sequences. Constant initialization reads off the predicted mantissas directly. Local refinement then holds the tree structure fixed and optimizes real constants with BFGS or simple Adam to minimize MSE on the observed points. Final selection chooses the refined formula with the lowest MSE or highest $M$5.

Empirically, the paper reports on $M$6 in-domain test formulas: univariate SymFormer without refinement reaches $M$7 and $M$8, while adding local gradient search yields $M$9 and $K$0; bivariate SymFormer plus local gradient search yields $K$1 and $K$2. On standard benchmarks—Nguyen, R, Livermore, Koza, Keijzer, Constant, and Koza-2—the reported overall median is $K$3 at $K$4 per equation, compared with $K$5 at $K$6 for NSRS and $K$7 at $K$8 for DSO. The ablation on constant encoding reports that "Full SymFormer" (extended scientific encoding plus local gradient search) achieves $K$9 and $K\times D_{\mathrm{enc}}$0. The paper states that SymFormer outperforms two state-of-the-art methods while having faster inference. Its listed limitations are a fixed sampling distribution, a maximum input dimension of $K\times D_{\mathrm{enc}}$1, and difficulty with very deep or highly nested functions.

3. SymFormer inside SymCircuit

In SymCircuit, SymFormer is the core generative policy network for probabilistic-circuit structure learning (Ju, 20 Mar 2026). It implements a distribution $K\times D_{\mathrm{enc}}$2 over PC structures $K\times D_{\mathrm{enc}}$3 by emitting a pre-order traversal sequence of tokens from a fixed vocabulary derived from a context-free grammar $K\times D_{\mathrm{enc}}$4. The grammar enforces the two PC invariants—decomposability and smoothness—by construction:

$K\times D_{\mathrm{enc}}$5

To eliminate arity ambiguity, each $K\times D_{\mathrm{enc}}$6 and $K\times D_{\mathrm{enc}}$7 is a distinct token.

Architecturally, SymFormer is a standard Transformer decoder with three key augmentations. First, tree-relative self-attention adds a learned scalar bias $K\times D_{\mathrm{enc}}$8 to the standard attention score, where $K\times D_{\mathrm{enc}}$9 is the tree relation in the partial parse:

$H_{\mathrm{enc}}$0

Second, dynamic grammar masking maintains a grammar state $H_{\mathrm{enc}}$1 and computes the valid next-token set $H_{\mathrm{enc}}$2 in $H_{\mathrm{enc}}$3; all logits for invalid tokens are set to $H_{\mathrm{enc}}$4, so $H_{\mathrm{enc}}$5 for invalid choices. Third, traversal-aware positional encoding augments token embeddings with a sinusoidal encoding of depth in the partial tree rather than absolute sequence position. The paper states that these mechanisms ensure that SymFormer can represent any distribution over valid PC structures up to a bounded depth, citing Theorem 4.1 in Park et al. 2026, and that every sampled sequence is guaranteed to decode to a decomposable, smooth PC.

The training framework is entropy-regularized RL-as-inference. SymFormer is trained jointly with per-structure parameters $H_{\mathrm{enc}}$6 by maximizing

$H_{\mathrm{enc}}$7

where $H_{\mathrm{enc}}$8 is an imitation-pretrained prior policy. The unique optimal policy under this objective is

$H_{\mathrm{enc}}$9

with $V$0. Since $V$1, the policy may be written as

$V$2

Setting $V$3 recovers exactly the Bayesian posterior, while the implementation uses $V$4 so that $V$5, yielding a tempered posterior that encourages exploration and smooths learning dynamics.

The paper’s main variance-reduction device is option-level REINFORCE. Rather than applying REINFORCE to all token decisions,

$V$6

it restricts the estimator to structural decisions $V$7:

$V$8

Under mild i.i.d. assumptions, the signal-to-noise ratio improves by a factor $V$9 over the token-level estimator. On NLTCS, the paper reports $C_E$0–$C_E$1 and $C_E$2–$C_E$3, implying a $C_E$4–$C_E$5 SNR boost and a $C_E$6 sample-efficiency gain in circuits evaluated to reach a target likelihood. More concretely, option-level REINFORCE reaches the same held-out log-likelihood in approximately $C_E$7 circuits ($C_E$8 epochs) that token-level methods require approximately $C_E$9 circuits to match. The paper also reports that adding an option-level critic head yields near-zero explained variance on NLTCS and brings no improvement.

The empirical characterization extends beyond likelihood. On NLTCS, SymCircuit closes $E\in[-10,\ldots,+10]$0 of the gap to LearnSPN. A three-layer uncertainty decomposition with $E\in[-10,\ldots,+10]$1 samples attributes $E\in[-10,\ldots,+10]$2 of total predictive variance to structural variance, approximately $E\in[-10,\ldots,+10]$3 to parameter variance, and $E\in[-10,\ldots,+10]$4 to leaf variance. On Plants, with $E\in[-10,\ldots,+10]$5 variables, the paper reports that after $E\in[-10,\ldots,+10]$6 epochs it closes $E\in[-10,\ldots,+10]$7 of the gap to LearnSPN and that the same option-level REINFORCE setup scales to token sequences of length approximately $E\in[-10,\ldots,+10]$8–$E\in[-10,\ldots,+10]$9. Vocabulary expansion to larger-arity sum nodes slows convergence because of the larger action space, and preliminary results suggest scalability rather than full parity with greedy baselines.

4. SymFormer inside SymPlex

In SymPlex, SymFormer is a structure-aware Transformer for generating symbolic expression trees under PDE constraints (Park et al., 3 Feb 2026). The model is described as a Transformer-decoder stack with three modifications relative to a vanilla sequence-based Transformer: traversal-aware positional encoding that preserves prefix-order information, tree-relative multihead self-attention that injects discrete learnable biases based on the current partial AST, and grammar- and depth-constrained autoregressive decoding that guarantees every generated prefix is a valid partial syntax tree.

The partial AST is reconstructed from the current prefix by reading off node arities and filling frontier slots in prefix order. For every pair of token positions $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$00, SymFormer computes a discrete relation

$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$01

Each relation type is embedded in a learnable vector $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$02, one per attention head. In head $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$03, the model uses projections $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$04, $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$05, and $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$06, and adds the relation bias to the keys before taking dot products:

$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$07

The paper also gives an equivalent scalar-bias view, emphasizing that attention is conditioned on the evolving tree structure rather than on sequence order alone.

Autoregressive decoding is constrained in three ways. The arity grammar restricts the next choice to tokens of the required arity. The remaining depth budget ensures that the minimal subtree rooted at the chosen token fits within the maximum depth $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$08. Degeneracy filtering can reject trivial subtrees such as $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$09 or $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$10. Because only valid tokens survive the mask, every prefix remains a well-formed partial AST. The paper further states that the model’s hidden representations depend only on the underlying partial AST up to bounded depth, citing Theorem 3.1 in Appendix A.

Training is reinforcement learning without access to ground-truth symbolic solutions. A training iteration samples expression trees from the policy, optimizes continuous constants by gradient descent on the PDE and boundary-condition residuals, computes

$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$11

with an additional $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$12 term if the PDE is time-dependent, and converts this to reward

$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$13

The algorithm also updates a top-$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$14 memory with structural and semantic diversity, applies a policy loss

$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$15

and adds an imitation loss over the memory. The depth-weight $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$16 discourages overly deep trees, and $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$17 is the state-wise entropy.

The paper positions this design against numerical and neural approaches that approximate PDE solutions in discretized or implicit function spaces. SymPlex instead operates directly in symbolic expression space, and the abstract reports exact recovery of non-smooth and parametric PDE solutions using deep learning-based symbolic methods. This suggests a specific role for SymFormer in domains where grammatical validity and hierarchical symbolic dependencies are central, and where sparse, delayed rewards must be integrated into end-to-end learning.

5. SymFormer for computer-aided tuberculosis diagnosis

In chest X-ray analysis, SymFormer is a unified two-head network for TB infection-area detection and CXR-image classification (Liu et al., 2023). The architecture comprises a backbone plus Feature Pyramid Network, a Symmetric Abnormity Search module on every FPN output, and two diagnosis heads. The backbone may be CNN- or transformer-based, including ResNet-50 and P2T-Small, and the FPN produces four feature maps. Each SAS block shares weights across scales and contains Symmetric Positional Encoding, Symmetric Search Attention, and a small feed-forward network with MLP and residual connection. The detection head is based on RetinaNet or Deformable DETR and includes a box-classification branch for active TB and latent TB using Focal Loss and a box-regression branch using Smooth $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$18 Loss. The image-classification head consists of five $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$19 Conv$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$20ReLU layers, global average pooling, one fully connected layer, and a softmax over healthy, sick non-TB, and TB.

The key inductive bias is bilateral symmetry. SymAttention is motivated by the observation that chest X-rays exhibit approximate left-right symmetry and that radiologists often compare symmetric locations to identify anomalies. Accordingly, each spatial location attends only to a small neighborhood around its bilaterally symmetric counterpart rather than to the whole image. If $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$21 is the reference position, the bilaterally symmetric center is

$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$22

The module learns coordinate offsets and attention weights, samples around the symmetric center, and fuses the result with residual and MLP layers.

Because clinical chest X-rays are not perfectly symmetric, the model adds Symmetric Positional Encoding. SPE computes an absolute positional encoding, splits it into left and right parts, predicts a small affine transform with an STN, transforms and flips the right-side encoding, and reassembles a symmetry-aware positional prior that is added to the feature map. The STN micro-architecture is specified as two repetitions of MaxPool$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$23Conv$(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$24ReLU, followed by flattening and a small MLP initialized to the identity transform.

Training proceeds in two stages. Stage 1 trains backbone, SAS, and detection head on TB images only and optimizes the detection loss. Stage 2 freezes the backbone and detection head, trains only the image-classification head on all images, and optimizes the image-level cross-entropy. At test time, images classified as non-TB have their detected boxes discarded, which the paper states reduces false positives. The implementation uses input resizing to $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$25, random horizontal flip, and Stage 2 augmentation with half of MC+Shenzhen and DA+DB data.

The empirical evaluation is centered on the TBX11K dataset. On the TBX11K test set, SymFormer with RetinaNet achieves $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$26 accuracy, $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$27 AUC, specificity $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$28, and AP/AR approximately $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$29; with Deformable DETR it achieves $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$30 accuracy and $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$31 AUC. The paper reports that this outperforms experienced radiologists at $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$32 accuracy. For detection, baseline RetinaNet yields category-agnostic $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$33 of approximately $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$34 on all images and $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$35 on TB images only, while SymFormer with RetinaNet reaches $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$36 and $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$37, respectively. Active TB $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$38 improves from $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$39 to $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$40 on all images and from $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$41 to $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$42 on TB images only; latent TB remains difficult, improving from $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$43 to $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$44 on all images and from $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$45 to $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$46 on TB images only. The ablation table reports the best validation result for category-agnostic TB as $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$47 $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$48 with SymAttention plus SPE with STN. Four-fold cross-validation yields approximately $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$49 accuracy and approximately $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$50 $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$51.

The limitations are domain-specific. The paper states that the method relies on roughly frontal, reasonably aligned CXR images; severe rotation or off-center crops may break the symmetry prior. Latent TB detection remains weak because of few training examples, and localization is coarse, with high $(x_i,y_i)\in\mathbb{R}^d\times\mathbb{R}$52 but low COCO-style AP.

6. Recurrent motifs and terminological ambiguity

Across the four works, "SymFormer" denotes different architectures, but each version introduces an explicit structural prior in place of unconstrained token or feature processing (Vastl et al., 2022, Ju, 20 Mar 2026, Park et al., 3 Feb 2026, Liu et al., 2023). In symbolic regression, the prior takes the form of joint symbol-and-constant prediction with a specialized constant encoding. In SymCircuit and SymPlex, it appears as tree-relative self-attention and grammar-constrained autoregressive decoding. In chest X-ray diagnosis, it appears as bilateral-symmetry-aware feature recalibration and sparse attention around mirrored coordinates.

The overlap between SymCircuit and SymPlex is especially close, though not identical. Both use tree-relative self-attention and grammar-constrained decoding for hierarchical symbolic objects. However, SymCircuit uses a learned scalar bias over five tree relations—parent, child, sibling, ancestor, and other—together with dynamic grammar masking over probabilistic-circuit productions, whereas SymPlex uses learnable relation vectors over six relation types, including self, and adds depth-budget and degeneracy constraints for expression trees. These are not interchangeable formulations, even though they share a related design vocabulary.

A common misconception is therefore that SymFormer names a single model family with stable semantics across papers. The literature summarized here does not support that view. A more accurate reading is that the label has been reused for transformer variants specialized to symbolic structure or symmetry structure. This suggests that any reference to "SymFormer" requires immediate disambiguation by domain and paper identifier: symbolic regression (Vastl et al., 2022), probabilistic circuits in SymCircuit (Ju, 20 Mar 2026), symbolic PDE solving in SymPlex (Park et al., 3 Feb 2026), or TB diagnosis on TBX11K (Liu et al., 2023).