Latent Equation Embedding (LEE) Framework

• Latent Equation Embedding (LEE) is a symbolic regression framework that jointly embeds symbolic tokens and numerical observations in a shared latent space.
• It employs an encoder and dual decoders to reconstruct expression syntax and predict function values, effectively grounding representations in both structure and behavior.
• LEE leverages iterative and gradient-based refinement to overcome one-shot amortization gaps, producing low-complexity, high-parsimony symbolic expressions.

Searching arXiv for the cited papers and closely related terminology. Latent Equation Embedding (LEE) is a symbolic-regression framework that learns a shared latent space $\mathcal{Z}\subset\mathbb{R}^{d_z}$ in which symbolic expressions and numerical observations are jointly represented and iteratively refined. In its official usage, LEE was introduced for symbolic regression in "Symbolic Regression via Latent Iterative Refinement" (Chu et al., 26 May 2026). Its defining components are an encoder $f_\theta$ that jointly embeds symbolic tokens and observed data, an expression decoder $g_{\text{expr}}$ that reconstructs formulas, and an evaluation decoder $g_{\text{eval}}$ that predicts function values, thereby functionally grounding the latent space. The framework is designed to close the amortization gap of one-shot neural symbolic regression by reusing the encoder as a learned inference optimizer through iterative decode–re-encode updates.

1. Terminology and scope

The expression Latent Equation Embedding (LEE) properly refers to the 2026 symbolic-regression framework described above, not to the earlier 2018 paper "Equation Embeddings" (Krstovski et al., 2018). The 2018 work introduced EqEmb and EqEmb-U, which learn semantic representations of equations from surrounding text in scientific corpora, but it does not define a model called LEE. A separate 2024 paper uses the acronym LEE for Latent Embedding Exploitation in wearable gesture recognition; that usage is unrelated to equations (Rafiq et al., 2024).

Term	Meaning	Source
LEE	Shared latent-space symbolic-regression framework with iterative refinement	(Chu et al., 26 May 2026)
EqEmb / EqEmb-U	Context-based semantic representations of equations in scientific text	(Krstovski et al., 2018)
LEE	Latent Embedding Exploitation for wearable gesture recognition	(Rafiq et al., 2024)

This distinction matters because the acronym can obscure two substantially different research programs. In the 2026 usage, equations are candidate symbolic programs to be inferred from numerical observations. In the 2018 usage, equations are semantic objects embedded from textual context. A plausible implication is that both lines of work treat equations as latent objects, but they do so for different tasks, with different inductive biases, and under different notions of similarity.

2. Symbolic-regression setting and motivating problem

LEE is formulated for symbolic regression, where the input is a set of observations

$\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$

and the target is an interpretable closed-form expression

$$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$

The paper characterizes this search space as mixed discrete–continuous: expression structure, operators, variables, and tree shape are combinatorial, whereas embedded constants are real-valued (Chu et al., 26 May 2026).

The stated motivation is the amortization gap in one-shot neural symbolic regression. Standard neural SR methods amortize inference by learning a direct map from observations to decoder parameters or directly to an expression. They are fast, but their single-pass prediction only approximates the true posterior over equations. LEE adapts iterative amortized inference to this setting by replacing explicit gradient-parameter updates with a learned latent update driven by decoded symbolic candidates:

$$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$

This update is the conceptual core of the method. The current latent code is decoded into an expression, that expression is re-encoded jointly with the observations, and the resulting latent is expected to be a better posterior approximation. The framework therefore treats the encoder as a learned inference optimizer rather than merely as an initialization module.

3. Latent-space architecture and functional grounding

LEE consists of three mappings:

$$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$

$\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$

$$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$

Here $f_\theta$0 is a tokenized symbolic expression and $f_\theta$1 is a set of scatter observations (Chu et al., 26 May 2026).

The encoder has two streams. In the symbolic stream, each token $f_\theta$2 is embedded as

$f_\theta$3

In the numeric stream, each observation $f_\theta$4 is embedded by a two-layer MLP into $f_\theta$5. Coordinates and target values are log-compressed for numerical stability via

$f_\theta$6

and non-finite values are handled by special learnable embeddings. The symbolic and numeric sequences are concatenated and processed by a Transformer, then masked mean-pooled:

$f_\theta$7

Two heads then produce Gaussian posterior parameters,

$f_\theta$8

with latent sampling by reparameterization,

$f_\theta$9

The expression decoder is an autoregressive Transformer decoder operating on prefix-notation symbolic tokens. It projects $g_{\text{expr}}$0 into $g_{\text{expr}}$1 memory tokens,

$g_{\text{expr}}$2

and then predicts

$g_{\text{expr}}$3

The evaluation decoder projects the same latent into a second memory bank,

$g_{\text{expr}}$4

and predicts function values at query points by

$g_{\text{expr}}$5

A central property of LEE is that the latent space is functionally grounded. It is trained not only to preserve syntax through $g_{\text{expr}}$6, but also to preserve functional behavior through $g_{\text{expr}}$7. The paper uses the example

$g_{\text{expr}}$8

to argue that syntactically different but functionally equivalent expressions should have nearby latent codes:

$g_{\text{expr}}$9

This suggests that latent search is intended to track functional fit rather than token-level resemblance alone.

Representation design is also tightly specified. Expressions are serialized in prefix (Polish) notation over a vocabulary of 40 tokens: 4 special tokens, 2 structural tokens, 10 variables $g_{\text{eval}}$0, 15 operators, and 14 digit tokens for constants. Constants are encoded at 3 significant figures in scientific notation as 9-token sequences. The operator set includes

$g_{\text{eval}}$1

4. Objective function, refinement mechanism, and inference pipeline

LEE is trained with a five-term objective

$g_{\text{eval}}$2

The expression reconstruction loss is token cross-entropy,

$g_{\text{eval}}$3

and the evaluation loss is a scale-invariant mean absolute error,

$g_{\text{eval}}$4

The KL term regularizes $g_{\text{eval}}$5 toward $g_{\text{eval}}$6.

The cross-modal alignment loss addresses the fact that training can use tokens and observations jointly, whereas inference may begin from observations alone:

$g_{\text{eval}}$7

The initial latent at inference is therefore

$g_{\text{eval}}$8

The most distinctive training term is the iterative refinement loss

$g_{\text{eval}}$9

where $\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$0 is a corrupted version of the ground-truth expression produced by random token drops, swaps, and substitutions. The paper interprets this as training the encoder to be a denoising inference optimizer.

Training proceeds in five phases: Phase 1 uses $\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$1; Phase 2 adds alignment with $\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$2; Phase 3 adds refinement with $\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$3; Phase 4 freezes decoders and emphasizes alignment with $\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$4; Phase 5 unfreezes all components and co-adapts them (Chu et al., 26 May 2026).

At inference, LEE performs pool-based iterative refinement. Starting from $\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$5, it decodes $\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$6 candidates using one greedy decode and multiple temperature-sampled decodes. Candidates are scored by

$\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$7

where $\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$8 is SymPy-simplified node count and $\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,$9. The top $$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$0 candidates form an initial pool $$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$1. A parent expression is sampled from the pool with rank-weighted probability, re-encoded jointly with the observations,

$$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$2

and decoded into new candidates,

$$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$3

Constants in decoded expressions are then refined with L-BFGS-B, and candidates are rescored and merged under diversity constraints.

The paper also adds hybrid gradient refinement because $$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$4 is differentiable in $$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$5:

$$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$6

The proximal term is intended to keep the latent near a decodable region. Every $$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$7 iterative steps, the current best latent is optimized for $$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$8 gradient steps, decoded back into symbolic form, and reinserted into the pool. If the gradient-based candidate harms validation $$e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).$$9, the system falls back to the previous pool champion.

The paper makes the mechanism more explicit with the approximation

$$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$0

arguing that the encoder implicitly computes residual mismatch through cross-attention between symbolic and numeric streams. In this sense, LEE instantiates an error-encoding variant of iterative amortized inference.

5. Empirical profile on SRBench

LEE is evaluated on SRBench, specifically Strogatz (14 ODE systems), Feynman (116 physics equations), and black-box (63 PMLB datasets without known ground truth), using noise levels

$$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$1

for the ground-truth benchmarks and noise-free black-box data (Chu et al., 26 May 2026). Data splitting follows the SRBench 75/25 train/test protocol, with an internal validation split carved from training, giving 60/15/25 train/val/test overall. Accuracy is reported as test $$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$2, and complexity is defined as SymPy-simplified node count.

The paper’s main empirical claim is not maximal raw accuracy but a favorable location on the accuracy–complexity Pareto frontier. Across SRBench, LEE produces expressions with complexity roughly 8–11, while strong accuracy-oriented baselines such as Operon, GP-GOMEA, TPSR, RAG-SR, and GenSR often lie in the 20–90 range.

Benchmark setting	$$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$3	Complexity
Strogatz, $$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$4	0.854	8.1
Strogatz, $$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$5	0.876	8.9
Strogatz, $$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$6	0.880	8.3
Feynman, $$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$7	0.884	9.9
Feynman, $$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$8	0.884	10.1
Feynman, $$\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}$$9	0.824	10.6
Black-box	0.559	9.0

The contrast is especially sharp on Feynman at $$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$0, where the paper reports: Operon at $$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$1, complexity $$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$2; GP-GOMEA at $$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$3, complexity $$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$4; TPSR at $$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$5, complexity $$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$6; RAG-SR at $$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$7, complexity $$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$8; and GenSR at $$\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},$$9, complexity $\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$0. LEE therefore trades $\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$1 for substantially lower complexity (Chu et al., 26 May 2026).

A same-backbone comparison isolates the effect of the refinement mechanism on Strogatz at $\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$2. Using the same pretrained LEE model and the same scoring function, the paper reports: one-shot decode from $\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$3 at $\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$4, complexity $\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$5; CMA-ES in latent space at $\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$6, complexity $\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$7; and full LEE iterative + gradient refinement at $\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$8, complexity $\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),$9. This is presented as evidence that the encoder is not merely a latent initializer but the operative learned optimizer.

The ablation study further separates the contributions of discrete and continuous refinement. On Strogatz, iterative refinement only yields $$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$0 at $$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$1 and $$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$2 at $$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$3; gradient refinement only yields $$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$4 at $$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$5 and $$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$6 at $$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$7; iterative + gradient yields $$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$8 at $$\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).$$9 and $f_\theta$00 at $f_\theta$01. The reported interpretation is that gradient refinement alone can drift off the decodable manifold, whereas iterative refinement alone is less robust under noise.

The appendix provides an additional latent-space ablation: removing the VAE encoder and alignment losses drops Strogatz $f_\theta$02 performance from

$f_\theta$03

to

$f_\theta$04

with some datasets failing to decode a valid expression. This is offered as evidence that regularized latent geometry is necessary for the iterative update to function.

The paper also attributes LEE’s simplicity bias to three compounding mechanisms: the autoregressive decoder’s implicit preference for shorter sequences, the explicit parsimony term in the pool score

$f_\theta$05

and the tendency of the evaluation-grounded latent geometry to allocate larger regions of $f_\theta$06 to simpler functional forms that are more common in the training distribution.

6. Relation to earlier equation-embedding work, misconceptions, and limitations

An earlier and conceptually adjacent line of work is "Equation Embeddings" (Krstovski et al., 2018). That paper addresses a different problem: discovering semantic representations of mathematical equations appearing in scientific articles. Its central idea is to treat an equation as a “singleton word” whose meaning can be inferred from the distributed representations of its surrounding words. EqEmb learns equation-level latent vectors from textual context, and EqEmb-U refines this by decomposing equations into equation units derived from Syntax Layout Trees (SLT). Quantitatively, that paper reports that EqEmb outperforms previous embedding models and EqEmb-U further improves performance on held-out pseudo log-likelihood across arXiv corpora containing about 98.5k equations (Krstovski et al., 2018).

The connection between the two frameworks is therefore conceptual rather than terminological. EqEmb embeds equations as semantic objects in document context; LEE embeds candidate equations in a functionally grounded latent space for symbolic regression. A plausible implication is that both frameworks reject raw string identity as the primary basis for equation meaning, but one grounds meaning in surrounding language and the other in numerical behavior.

A common misconception is to treat LEE as a generic name for any latent representation of equations. The record is more specific. The official model name LEE belongs to the 2026 symbolic-regression framework (Chu et al., 26 May 2026). The 2018 model is officially EqEmb or EqEmb-U, not LEE (Krstovski et al., 2018). Another possible source of confusion is the 2024 acronym LEE for Latent Embedding Exploitation in wearable-sensor few-shot continual learning, which is unrelated to equations (Rafiq et al., 2024).

The 2026 paper is also explicit about limitations. LEE is not accuracy SOTA; it trails top GP systems by about $f_\theta$07–$f_\theta$08 $f_\theta$09 on some clean benchmarks. It assumes a pretrained grammar-derived corpus of synthetic expressions and a learned latent geometry, so insufficient operator vocabulary or training distribution can degrade search quality. Gradient refinement alone can drift off-manifold. The model is about 150M parameters and requires about 200 GH200-GPU-hours of offline pretraining, although this cost is amortized across datasets; inference is reported as tens of seconds per dataset, much faster than many GP baselines (Chu et al., 26 May 2026).

Taken together, these papers situate LEE within a broader research trend toward treating equations as latent objects with nontrivial semantics. In the precise contemporary sense, however, Latent Equation Embedding denotes a functionally grounded latent-search framework for symbolic regression, whose central innovation is to train the encoder not only to represent expressions, but also to repair imperfect decoded expressions when paired with observed data (Chu et al., 26 May 2026).