---
title: Latent Equation Embedding (LEE) Framework
url: https://www.emergentmind.com/topics/latent-equation-embedding-lee
type: topic
---

# Latent Equation Embedding (LEE) Framework

Searching arXiv for the cited papers and closely related terminology.
Latent Equation Embedding (LEE) is a symbolic-regression framework that learns a shared latent space $\mathcal{Z}\subset\mathbb{R}^{d_z}$ in which symbolic expressions and numerical observations are jointly represented and iteratively refined. In its official usage, LEE was introduced for symbolic regression in "Symbolic Regression via Latent Iterative Refinement" [2605.27245]. Its defining components are an encoder $f_\theta$ that jointly embeds symbolic tokens and observed data, an expression decoder $g_{\text{expr}}$ that reconstructs formulas, and an evaluation decoder $g_{\text{eval}}$ that predicts function values, thereby functionally grounding the latent space. The framework is designed to close the amortization gap of one-shot neural symbolic regression by reusing the encoder as a learned inference optimizer through iterative decode–re-encode updates.

## 1. Terminology and scope

The expression **Latent Equation Embedding (LEE)** properly refers to the 2026 symbolic-regression framework described above, not to the earlier 2018 paper "Equation Embeddings" [1803.09123]. The 2018 work introduced **EqEmb** and **EqEmb-U**, which learn semantic representations of equations from surrounding text in scientific corpora, but it does **not** define a model called LEE. A separate 2024 paper uses the acronym **LEE** for **Latent Embedding Exploitation** in wearable gesture recognition; that usage is unrelated to equations [2405.08969].

| Term | Meaning | Source |
|---|---|---|
| **LEE** | Shared latent-space symbolic-regression framework with iterative refinement | [2605.27245] |
| **EqEmb / EqEmb-U** | Context-based semantic representations of equations in scientific text | [1803.09123] |
| **LEE** | Latent Embedding Exploitation for wearable gesture recognition | [2405.08969] |

This distinction matters because the acronym can obscure two substantially different research programs. In the 2026 usage, equations are candidate symbolic programs to be inferred from numerical observations. In the 2018 usage, equations are semantic objects embedded from textual context. A plausible implication is that both lines of work treat equations as latent objects, but they do so for different tasks, with different inductive biases, and under different notions of similarity.

## 2. Symbolic-regression setting and motivating problem

LEE is formulated for symbolic regression, where the input is a set of observations
$$
\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^N,
$$
and the target is an interpretable closed-form expression
$$
e^*=\arg\max_{e\in\mathcal{E}} p(e\mid \mathcal{D}).
$$
The paper characterizes this search space as mixed discrete–continuous: expression structure, operators, variables, and tree shape are combinatorial, whereas embedded constants are real-valued [2605.27245].

The stated motivation is the **amortization gap** in one-shot neural symbolic regression. Standard neural SR methods amortize inference by learning a direct map from observations to decoder parameters or directly to an expression. They are fast, but their single-pass prediction only approximates the true posterior over equations. LEE adapts **iterative amortized inference** to this setting by replacing explicit gradient-parameter updates with a learned latent update driven by decoded symbolic candidates:
$$
\boxed{\mathbf{z}_{t+1} = f_\theta\bigl(\hat{e}_t,\; \mathcal{D}\bigr), \qquad \hat{e}_t = g_\text{expr}(\mathbf{z}_t).}
$$

This update is the conceptual core of the method. The current latent code is decoded into an expression, that expression is re-encoded jointly with the observations, and the resulting latent is expected to be a better posterior approximation. The framework therefore treats the encoder as a learned inference optimizer rather than merely as an initialization module.

## 3. Latent-space architecture and functional grounding

LEE consists of three mappings:
$$
\mathbf{z} = f_\theta(\mathbf{t}, \mathcal{D}) \in \mathbb{R}^{d_z},
$$
$$
\hat{\mathbf{t}} = g_{\text{expr}}(\mathbf{z}),
$$
$$
\hat{y}(\mathbf{x}) = g_{\text{eval}}(\mathbf{z}, \mathbf{x}).
$$
Here $\mathbf{t}\in\mathcal{V}^L$ is a tokenized symbolic expression and $\mathcal{D}$ is a set of scatter observations [2605.27245].

The encoder has two streams. In the **symbolic stream**, each token $t_j$ is embedded as
$$
\mathbf{h}_j^\text{sym} = \mathrm{Embed}(t_j)\in\mathbb{R}^d.
$$
In the **numeric stream**, each observation $(\mathbf{x}_i,y_i)$ is embedded by a two-layer MLP into $\mathbf{h}_i^\text{num}\in\mathbb{R}^d$. Coordinates and target values are log-compressed for numerical stability via
$$
\tilde{u} = \mathrm{sign}(u)\cdot \log(1+|u|),
$$
and non-finite values are handled by special learnable embeddings. The symbolic and numeric sequences are concatenated and processed by a Transformer, then masked mean-pooled:
$$
\mathbf{h}=\mathrm{MeanPool}\bigl(\mathrm{Transformer}([\mathbf{h}^\text{sym};\mathbf{h}^\text{num}])\bigr).
$$
Two heads then produce Gaussian posterior parameters,
$$
\boldsymbol{\mu}=W_\mu \mathbf{h}, \qquad \log \boldsymbol{\sigma}^2 = W_\sigma \mathbf{h},
$$
with latent sampling by reparameterization,
$$
\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma}\odot \boldsymbol{\epsilon}, \qquad \boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}).
$$

The **expression decoder** is an autoregressive Transformer decoder operating on **prefix-notation** symbolic tokens. It projects $\mathbf{z}$ into $K$ memory tokens,
$$
\mathbf{M}_\text{expr} = \mathrm{reshape}(W_m\mathbf{z}) \in \mathbb{R}^{K\times d_\text{expr}},
$$
and then predicts
$$
p_\theta(t_j\mid \mathbf{z}, t_{<j}) = \mathrm{softmax}\bigl(W_o \cdot \mathrm{TransDec}(\mathbf{M}_\text{expr}, t_{<j})\bigr).
$$

The **evaluation decoder** projects the same latent into a second memory bank,
$$
\mathbf{M}_\text{eval} = \mathrm{reshape}(W_e \mathbf{z}) \in \mathbb{R}^{K\times d_\text{eval}},
$$
and predicts function values at query points by
$$
\hat{y}_j = \mathrm{MLP}\bigl(\mathrm{TransDec}(\mathbf{M}_\text{eval}, \mathrm{MLP}(\mathbf{q}_j))\bigr).
$$

A central property of LEE is that the latent space is **functionally grounded**. It is trained not only to preserve syntax through $g_{\text{expr}}$, but also to preserve functional behavior through $g_{\text{eval}}$. The paper uses the example
$$
e_1 = 2\sin(x)\cos(y), \qquad e_2 = \sin(x+y)+\sin(x-y),
$$
to argue that syntactically different but functionally equivalent expressions should have nearby latent codes:
$$
\mathbf{z}_1 \approx \mathbf{z}_2 \quad \text{whenever} \quad e_1(\mathbf{x}) = e_2(\mathbf{x}) \;\; \forall \mathbf{x}.
$$
This suggests that latent search is intended to track functional fit rather than token-level resemblance alone.

Representation design is also tightly specified. Expressions are serialized in **prefix (Polish) notation** over a vocabulary of 40 tokens: 4 special tokens, 2 structural tokens, 10 variables $(x_0,\ldots,x_9)$, 15 operators, and 14 digit tokens for constants. Constants are encoded at 3 significant figures in scientific notation as 9-token sequences. The operator set includes
$$
+, -, \times, \div, \sin, \cos, \tan, \tanh, \exp, \log, \sqrt{\cdot}, x^2, x^3, \text{abs}, \text{neg}.
$$

## 4. Objective function, refinement mechanism, and inference pipeline

LEE is trained with a five-term objective
$$
\mathcal{L} = \lambda_\text{expr}\mathcal{L}_\text{expr} + \lambda_\text{eval}\mathcal{L}_\text{eval} + \lambda_\text{KL}\mathcal{L}_\text{KL} + \lambda_\text{align}\mathcal{L}_\text{align} + \lambda_\text{refine}\mathcal{L}_\text{refine}.
$$
The **expression reconstruction loss** is token cross-entropy,
$$
\mathcal{L}_\text{expr} = -\frac{1}{|\mathcal{T}|}\sum_{j\in\mathcal{T}} \log p_\theta(t_j\mid \mathbf{z}, t_{<j}),
$$
and the **evaluation loss** is a scale-invariant mean absolute error,
$$
\mathcal{L}_\text{eval} = \frac{1}{|\mathcal{V}_\text{fin}|}\sum_{j\in\mathcal{V}_\text{fin}} \frac{|\hat{y}_j-y_j|}{\max(|y_j|,1)}.
$$
The **KL term** regularizes $q(\mathbf{z}\mid \mathbf{t},\mathcal{D})$ toward $\mathcal{N}(\mathbf{0},\mathbf{I})$.

The **cross-modal alignment loss** addresses the fact that training can use tokens and observations jointly, whereas inference may begin from observations alone:
$$
\mathcal{L}_\text{align} = D_\text{KL}\!\left(q(\mathbf{z}\mid \mathbf{t}, \mathcal{D}) \,\big\|\, \mathrm{sg}\!\left[p(\mathbf{z}\mid\mathcal{D})\right]\right).
$$
The initial latent at inference is therefore
$$
\mathbf{z}_0 = f_\theta(\varnothing,\mathcal{D}).
$$

The most distinctive training term is the **iterative refinement loss**
$$
\mathcal{L}_\text{refine} = -\frac{1}{|\mathcal{T}|}\sum_{j\in \mathcal{T}} \log p_\theta\bigl(t_j \mid f_\theta(\tilde{e}, \mathcal{D}),\; t_{<j}\bigr),
$$
where $\tilde e$ is a corrupted version of the ground-truth expression produced by random token drops, swaps, and substitutions. The paper interprets this as training the encoder to be a **denoising inference optimizer**.

Training proceeds in five phases: Phase 1 uses $\lambda_\text{expr}=1.0,\ \lambda_\text{eval}=5.0,\ \lambda_\text{KL}=0.001,\ \lambda_\text{align}=0,\ \lambda_\text{refine}=0$; Phase 2 adds alignment with $\lambda_\text{align}=2.0$; Phase 3 adds refinement with $\lambda_\text{refine}=1.0$; Phase 4 freezes decoders and emphasizes alignment with $\lambda_\text{align}=5.0$; Phase 5 unfreezes all components and co-adapts them [2605.27245].

At inference, LEE performs pool-based iterative refinement. Starting from $\mathbf{z}_0$, it decodes $n_\text{init}$ candidates using one greedy decode and multiple temperature-sampled decodes. Candidates are scored by
$$
R^2_\text{train}-\alpha C(e),
$$
where $C(e)$ is SymPy-simplified node count and $\alpha=0.002$. The top $P$ candidates form an initial pool $\Pi_0$. A parent expression is sampled from the pool with rank-weighted probability, re-encoded jointly with the observations,
$$
\mathbf{z}_{t+1}^{(i)} = f_\theta\bigl(\hat{e}_t^{(i)}, \mathcal{D}\bigr),
$$
and decoded into new candidates,
$$
\hat{e}_{t+1}^{(j)} \sim g_\text{expr}\bigl(\mathbf{z}_{t+1}^{(i)}\bigr), \qquad j=1,\ldots,n_\text{new}.
$$
Constants in decoded expressions are then refined with **L-BFGS-B**, and candidates are rescored and merged under diversity constraints.

The paper also adds **hybrid gradient refinement** because $g_\text{eval}$ is differentiable in $\mathbf{z}$:
$$
\mathbf{z}_{t+1} = \mathbf{z}_t - \eta \nabla_{\mathbf{z}} \Bigl[ \|g_\text{eval}(\mathbf{z}_t,\mathbf{X})-\mathbf{y}\|_2^2 + \lambda_\text{prox}\|\mathbf{z}_t-\mathbf{z}_\text{anchor}\|_2^2 \Bigr].
$$
The proximal term is intended to keep the latent near a decodable region. Every $d$ iterative steps, the current best latent is optimized for $k$ gradient steps, decoded back into symbolic form, and reinserted into the pool. If the gradient-based candidate harms validation $R^2$, the system falls back to the previous pool champion.

The paper makes the mechanism more explicit with the approximation
$$
\mathbf{z}_{t+1} = f_\theta(\hat e_t,\mathcal D) \approx f_\theta(\hat e_t,\hat e_t(\mathbf X)-\mathbf y,\mathbf X),
$$
arguing that the encoder implicitly computes residual mismatch through cross-attention between symbolic and numeric streams. In this sense, LEE instantiates an **error-encoding variant** of iterative amortized inference.

## 5. Empirical profile on SRBench

LEE is evaluated on **SRBench**, specifically **Strogatz** (14 ODE systems), **Feynman** (116 physics equations), and **black-box** (63 PMLB datasets without known ground truth), using noise levels
$$
\epsilon \in \{0, 0.01, 0.1\}
$$
for the ground-truth benchmarks and noise-free black-box data [2605.27245]. Data splitting follows the SRBench 75/25 train/test protocol, with an internal validation split carved from training, giving 60/15/25 train/val/test overall. Accuracy is reported as test $R^2$, and **complexity** is defined as **SymPy-simplified node count**.

The paper’s main empirical claim is not maximal raw accuracy but a favorable location on the **accuracy–complexity Pareto frontier**. Across SRBench, LEE produces expressions with complexity roughly **8–11**, while strong accuracy-oriented baselines such as Operon, GP-GOMEA, TPSR, RAG-SR, and GenSR often lie in the **20–90** range.

| Benchmark setting | $R^2$ | Complexity |
|---|---:|---:|
| Strogatz, $\epsilon=0$ | 0.854 | 8.1 |
| Strogatz, $\epsilon=0.01$ | 0.876 | 8.9 |
| Strogatz, $\epsilon=0.1$ | 0.880 | 8.3 |
| Feynman, $\epsilon=0$ | 0.884 | 9.9 |
| Feynman, $\epsilon=0.01$ | 0.884 | 10.1 |
| Feynman, $\epsilon=0.1$ | 0.824 | 10.6 |
| Black-box | 0.559 | 9.0 |

The contrast is especially sharp on Feynman at $\epsilon=0.1$, where the paper reports: Operon at $R^2=0.985$, complexity $89$; GP-GOMEA at $R^2=0.996$, complexity $46$; TPSR at $R^2=0.984$, complexity $67$; RAG-SR at $R^2=0.985$, complexity $75$; and GenSR at $R^2=0.989$, complexity $24$. LEE therefore trades $R^2$ for substantially lower complexity [2605.27245].

A same-backbone comparison isolates the effect of the refinement mechanism on Strogatz at $\epsilon=0.1$. Using the same pretrained LEE model and the same scoring function, the paper reports: one-shot decode from $\mathbf{z}_0$ at $R^2=0.795\pm0.019$, complexity $13.0\pm1.9$; CMA-ES in latent space at $R^2=0.848\pm0.011$, complexity $13.8\pm0.8$; and full LEE iterative + gradient refinement at $R^2=0.880\pm0.024$, complexity $8.3\pm1.4$. This is presented as evidence that the encoder is not merely a latent initializer but the operative learned optimizer.

The ablation study further separates the contributions of discrete and continuous refinement. On Strogatz, **iterative refinement only** yields $0.872\pm0.012$ at $\epsilon=0$ and $0.850\pm0.024$ at $\epsilon=0.1$; **gradient refinement only** yields $0.742\pm0.027$ at $\epsilon=0$ and $0.744\pm0.045$ at $\epsilon=0.1$; **iterative + gradient** yields $0.854\pm0.005$ at $\epsilon=0$ and $0.880\pm0.024$ at $\epsilon=0.1$. The reported interpretation is that gradient refinement alone can drift off the decodable manifold, whereas iterative refinement alone is less robust under noise.

The appendix provides an additional latent-space ablation: removing the VAE encoder and alignment losses drops Strogatz $\epsilon=0.1$ performance from
$$
0.880 \pm 0.024
$$
to
$$
0.582 \pm 0.084,
$$
with some datasets failing to decode a valid expression. This is offered as evidence that regularized latent geometry is necessary for the iterative update to function.

The paper also attributes LEE’s simplicity bias to three compounding mechanisms: the autoregressive decoder’s implicit preference for shorter sequences, the explicit parsimony term in the pool score
$$
s(e)=R^2(e)-\alpha\cdot C(e),
$$
and the tendency of the evaluation-grounded latent geometry to allocate larger regions of $\mathcal Z$ to simpler functional forms that are more common in the training distribution.

## 6. Relation to earlier equation-embedding work, misconceptions, and limitations

An earlier and conceptually adjacent line of work is "Equation Embeddings" [1803.09123]. That paper addresses a different problem: discovering semantic representations of mathematical equations appearing in scientific articles. Its central idea is to treat an equation as a **“singleton word”** whose meaning can be inferred from the distributed representations of its surrounding words. EqEmb learns equation-level latent vectors from textual context, and EqEmb-U refines this by decomposing equations into **equation units** derived from **Syntax Layout Trees (SLT)**. Quantitatively, that paper reports that **EqEmb outperforms previous embedding models** and **EqEmb-U further improves performance** on held-out pseudo log-likelihood across arXiv corpora containing about **98.5k equations** [1803.09123].

The connection between the two frameworks is therefore conceptual rather than terminological. EqEmb embeds equations as semantic objects in document context; LEE embeds candidate equations in a functionally grounded latent space for symbolic regression. A plausible implication is that both frameworks reject raw string identity as the primary basis for equation meaning, but one grounds meaning in surrounding language and the other in numerical behavior.

A common misconception is to treat LEE as a generic name for any latent representation of equations. The record is more specific. The official model name **LEE** belongs to the 2026 symbolic-regression framework [2605.27245]. The 2018 model is officially **EqEmb** or **EqEmb-U**, not LEE [1803.09123]. Another possible source of confusion is the 2024 acronym **LEE** for **Latent Embedding Exploitation** in wearable-sensor few-shot continual learning, which is unrelated to equations [2405.08969].

The 2026 paper is also explicit about limitations. LEE is **not accuracy SOTA**; it trails top GP systems by about $0.10$–$0.17$ $R^2$ on some clean benchmarks. It assumes a pretrained grammar-derived corpus of synthetic expressions and a learned latent geometry, so insufficient operator vocabulary or training distribution can degrade search quality. Gradient refinement alone can drift off-manifold. The model is about **150M parameters** and requires about **200 GH200-GPU-hours** of offline pretraining, although this cost is amortized across datasets; inference is reported as **tens of seconds per dataset**, much faster than many GP baselines [2605.27245].

Taken together, these papers situate LEE within a broader research trend toward treating equations as latent objects with nontrivial semantics. In the precise contemporary sense, however, **Latent Equation Embedding** denotes a functionally grounded latent-search framework for symbolic regression, whose central innovation is to train the encoder not only to represent expressions, but also to **repair** imperfect decoded expressions when paired with observed data [2605.27245].

Source: https://www.emergentmind.com/topics/latent-equation-embedding-lee