---
title: 'Neural Theorizer (NEO): Explanatory Program Induction'
url: https://www.emergentmind.com/topics/neural-theorizer-neo
type: topic
---

# Neural Theorizer (NEO): Explanatory Program Induction

Searching arXiv for the specified NEO paper and the acronym-overlap paper to ground the article in current arXiv records.
Neural Theorizer (NEO) is a probabilistic neural model introduced within the Learning-to-Theorize (L2T) paradigm for inferring explicit explanatory theories of the world from raw, non-textual observations [2605.03413]. In this formulation, a theory is an executable, compositional program built from learned primitives whose semantics are induced from data rather than specified a priori. NEO operationalizes understanding not as accurate future prediction alone, but as discovering abstract, reusable, compositional mechanisms that explain how observations are generated and transformed, support intervention, and transfer across instances [2605.03413].

## 1. Conceptual basis and relation to world modeling

Learning-to-Theorize reframes “understanding” as inferring explicit explanatory theories of how observations are generated and transformed [2605.03413]. The motivating claim is drawn from developmental “theory-theory”: children build internal theories before mature language, and these theories function as abstract, reusable, compositional mechanisms rather than as instance-specific predictive mappings. In NEO, this perspective is instantiated by treating programs as “mental programs” whose compositionality yields systematic generalization and productivity, including length generalization to longer programs [2605.03413].

This contrasts with standard world models that optimize predictive reconstruction such as next-frame prediction. In the L2T view, conventional predictive objectives often learn entangled, instance-specific mappings or monolithic latent actions, whereas NEO induces explicit programmatic explanations with shared primitives and a shared executor [2605.03413]. The distinction is not merely architectural. NEO explicitly targets transfer of inferred programs across new inputs and unseen compositions, together with length generalization to longer programs than those observed during training [2605.03413].

A common source of confusion is the acronym itself. “NEO” in this context denotes the Neural Theorizer of “Learning to Theorize the World from Observation” [2605.03413], not the unrelated test-time adaptation method “NEO: No-Optimization Test-Time Adaptation through Latent Re-Centering” [2510.05635]. The two works share an acronym but address different problem classes: the former concerns latent program induction and explanation-driven generalization, while the latter concerns optimization-free test-time adaptation through feature re-centering.

## 2. Probabilistic formulation and latent program semantics

NEO models each phenomenon as a source–target pair $(x,y)$ and introduces a latent program $\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$, an ordered sequence over a finite primitive set $\mathcal{Z}=\{z_1,\dots,z_M\}$ [2605.03413]. Latent states are denoted $s=(s_1,\dots,s_{K+1})$, with $s_1 = E_\theta(x)$ and each subsequent state obtained by executing a primitive from the current state [2605.03413]. The shared transition model $T$, implemented as either $p_\theta(s_{k+1}\mid s_k,z)$ or a deterministic counterpart $f_\theta$, assigns operational semantics to primitive symbols [2605.03413].

For observation pairs, the conditional likelihood is marginalized over the latent program and execution trace:
$$
p_\theta(y \mid x) = \int p_\theta(y \mid s_{K+1})\, p_\theta(\tau,s \mid x)\, d\tau\, ds.
$$
The prior over program and execution trace factorizes as
$$
p_\theta(\tau,s \mid x) = p_\theta(s_1 \mid x) \;\prod_{k=1}^{K} p_\theta(z_{i_k} \mid s_k)\, p_\theta(s_{k+1} \mid s_k, z_{i_k}).
$$
Here, $p_\theta(z_{i_k}\mid s_k)$ functions as a state-conditioned program prior, while $p_\theta(y\mid s_{K+1})$ is the observation likelihood via the decoder [2605.03413].

Training uses an amortized posterior $q_\phi(\tau,s \mid x,y)$ and maximizes the ELBO:
$$
\log p_\theta(y \mid x) \;\ge\; \mathbb{E}_{q_\phi(\tau,s \mid x,y)}\!\big[\log p_\theta(y \mid s_{K+1})\big] - \mathrm{KL}\!\big(q_\phi(\tau,s \mid x,y)\,\|\,p_\theta(\tau,s \mid x)\big).
$$
The posterior shares the encoder and executor with the generative model:
$$
q_\phi(\tau,s \mid x,y) = p_\theta(s_1 \mid x)\;\prod_{k=1}^{K} q_\phi(z_{i_k} \mid s_k, y)\; p_\theta(s_{k+1} \mid s_k, z_{i_k}).
$$
In practical implementation, execution is deterministic,
$$
s_{k+1} = f_\theta(s_k, z_{i_k}),\quad \hat{y}_\tau = D_\theta\!\left(f_{\tau,\theta}(E_\theta(x))\right),
$$
with reconstruction loss $\ell(y,\hat{y}_\tau) \equiv -\log p_\theta(y \mid x,\tau)$ [2605.03413].

Program length is selected by a Minimum Description Length criterion:
$$
k^* = \arg\min_{k \in \{1,\dots,K+1\}} \lambda_{\mathrm{MDL}}^k \,\ell\!\left(y,\hat{y}_k\right),\quad \hat{y}_k = D_\theta(s_k).
$$
This penalizes longer explanations while favoring the shortest accurate theory [2605.03413]. A plausible implication is that NEO’s notion of explanation is explicitly complexity-controlled rather than emerging only indirectly from predictive fit.

## 3. Architecture and learned Language of Thought

NEO consists of an observation encoder $E_\theta$, a decoder $D_\theta$, a program induction network $q_\phi$ called the theory programmer, a discrete codebook implemented via VQ-VAE, and a shared executor $f_\theta$ [2605.03413]. The encoder and decoder map raw, non-text inputs to latent states and back; for GridWorld and Image Editing, a pretrained CNN VAE defines the latent space, whereas Arithmetic uses learned embeddings and a linear decoder [2605.03413]. These components are shared across training examples and across the generative and variational paths.

The theory programmer is a goal-conditioned policy over primitive symbols. It takes the current latent state $s_k$ and the encoded target $s_y = E_\theta(y)$ and outputs a categorical distribution over codebook entries [2605.03413]. Discreteness is implemented with a VQ-VAE codebook $\mathcal{E}=\{e_1,\dots,e_{M'}\}$, yielding a learned symbolic vocabulary and inducing programs by iteratively selecting $z_{i_k}$ [2605.03413]. The primitive set $\mathcal{Z}$ is therefore not preset; its symbols are learned as VQ codes, and the grammar is sequence composition $\tau=(z_{i_1},\dots,z_{i_K})$ [2605.03413].

The executor is shared across all phenomena and operationally defines primitive meaning through its effect on latent states [2605.03413]. Intermediate states are decoded both for MDL length selection and for a state grounding regularizer,
$$
\mathcal{L}_{\text{state}} = \sum_{k=1}^{K} \big\| s_k - \mathrm{sg}[E_\theta(D_\theta(s_k))] \big\|^2,
$$
where $\mathrm{sg}$ denotes stop-gradient and the update applies only to the transition model [2605.03413]. This keeps intermediate states on the observation manifold.

The program representation is syntactically simple: a flat sequence of primitive symbols executed by sequential application of the shared transition model [2605.03413]. In the Image Editing example, learned primitive codes correspond to `rot` (90° clockwise), `br_p` (brightness+), and `mask` (gray square mask), so that
$$
\tau = (\mathrm{rot}, \mathrm{br\_p}, \mathrm{mask})
$$
induces the trace
$$
s_1 = E_\theta(x) \rightarrow s_2 = f_\theta(s_1,\mathrm{rot}) \rightarrow s_3 = f_\theta(s_2,\mathrm{br\_p}) \rightarrow s_4 = f_\theta(s_3,\mathrm{mask}),
$$
with $\hat y = D_\theta(s_4)$ [2605.03413]. The same inferred program can then be transferred to a new input $x'$ to obtain $\hat y'$ generated by the same latent program [2605.03413]. This is the core operational meaning of explanation-driven transfer in NEO.

## 4. Training, inference, and test-time scaling

Training initializes pretrained $E_\theta$ and $D_\theta$ together with the theory programmer $q_\phi$, executor $f_\theta$, a maximum step count $K$, and learning rate $\eta$ [2605.03413]. For each minibatch of pairs $(x,y)$, the model computes $s_1 = E_\theta(x)$ and $s_y = E_\theta(y)$, then iterates for $k=1,\dots,K$: sample or select a primitive $z_{i_k} \sim q_\phi(z \mid s_k, s_y)$, execute $s_{k+1} \gets f_\theta(s_k, z_{i_k})$, and optionally decode each $s_k$ to $\hat y_k$ [2605.03413]. MDL selects
$$
k^* = \arg\min_k \lambda_\text{MDL}^k \cdot \ell(y, D_\theta(s_k)),
$$
after which the model applies reconstruction, grounding, and VQ losses, truncated to $k^*$ [2605.03413].

The practical objective is
$$
\mathcal{L}_{\text{NEO}}(\theta,\phi) = \mathbb{E}_{q_{\phi,\theta}(\tau \mid x,y)}\!\big[\ell(y,\hat{y}_\tau)\big] + \lambda_{\text{vq}}\mathcal{L}_{\mathrm{VQ}} + \lambda_{\text{state}}\mathcal{L}_{\text{state}}.
$$
This combines program-conditioned reconstruction with discrete codebook learning and latent-state grounding [2605.03413].

At inference, NEO receives a support pair $(x_s,y_s)$, rolls out the theory programmer greedily with
$$
z_{i_k} = \arg\max q_\phi(z\mid s_k, s_y),
$$
and uses MDL to obtain an inferred program $\tau^*=(z_{i_1},\dots,z_{i_{k^*}})$ [2605.03413]. The model then transfers $\tau^*$ to a query input $x_q$ by applying the same executor sequence to $E_\theta(x_q)$ and decoding the result [2605.03413]. Evaluation distinguishes self-explainability on the support pair and transferability on the query pair.

A further variant, NEO-S, performs test-time scaling by sampling $B$ candidate programs on the support pair using a temperature $T$ over the programmer’s softmax, filtering candidates with MDL, and selecting the most frequent successful program by majority voting [2605.03413]. Larger $B$ and higher temperature improve exploration and length-OOD performance, notably in Arithmetic [2605.03413]. This suggests that, in some regimes, the limiting factor is not primitive discovery but long-horizon search over compositions.

## 5. Benchmark structure and empirical behavior

The Observation-to-Theory Induction Benchmark (OTIB) contains three domains: GridWorld, Arithmetic Factorization Reasoning, and Image Editing [2605.03413]. Each test instance provides a support pair $(x^{(1)},y^{(1)})$ and a query pair $(x^{(2)},y^{(2)})$ sharing the same latent program $\tau$, so the model must infer $\tau$ from the support pair and transfer it to the query input [2605.03413]. Evaluation uses self-explainability and transferability, with splits into in-distribution, compositional OOD, and length OOD regimes [2605.03413].

| Domain | Latent primitives and training lengths | Evaluation |
|---|---|---|
| GridWorld | 10×10 grid; up/down/left/right; train length 1–3; test length 4–8 | Exact match accuracy |
| Arithmetic | Integer pairs with $\times2,\times3,\times5,\times7$; train length 1–3; test length 4–6 | Exact correctness |
| Image Editing | CIFAR-10 with 8 edit primitives; train length 1–2; test length 3–4 | L1 pixel loss |

In GridWorld, at $\alpha=0.33$, NEO achieves transfer 0.933 on compositional OOD and 0.845 on length OOD; NEO-S with $B=64$ improves these to 0.976 and 0.907, while monolithic baselines collapse on OOD, near 0 [2605.03413]. The same study reports robustness across $\alpha=0.66$ and $1.00$, with NEO maintaining high transfer accuracy while baselines fail to generalize [2605.03413].

In Arithmetic Factorization, NEO shows strong compositional OOD behavior, for example 0.573 at $\alpha=0.66$, but greedy inference struggles on long sequences, with length OOD around 0.02–0.04 [2605.03413]. NEO-S raises length OOD to 0.696 at $\alpha=0.66$ and 0.707 at $\alpha=1.00$ using $B=1024$ and temperature, indicating that primitives were learned and that additional test-time search resolves long-horizon recomposition [2605.03413].

In Image Editing, NEO yields the lowest L1 distances across all $\alpha$ splits for compositional and length OOD; the reported compositional OOD range is approximately 0.09–0.12 versus baselines at approximately 0.12–0.18, and length OOD is approximately 0.10–0.13, substantially better than the identity baseline [2605.03413]. Qualitative examples show NEO decomposing novel edit sequences such as brightness+ followed by mask and selecting instance-specific $k^*$ values matching ground-truth step counts under MDL, whereas baselines attempt monolithic mappings and fail [2605.03413].

## 6. Ablations, limitations, and research significance

Ablation analysis attributes a central role to state grounding. Removing $\mathcal{L}_{\text{state}}$ causes catastrophic collapse: primitiveness is approximately 0.002, and both self-explainability and transferability drop to zero [2605.03413]. The reported interpretation is that intermediate states drift off-manifold and reusable building blocks are not discovered [2605.03413]. Code–primitive alignment matrices further show near one-to-one mapping between learned codes and ground-truth primitives, even when many primitives are never seen in isolation, and the primitiveness metric often approaches 1.0, surpassing the fraction directly observable in training [2605.03413].

MDL tuning governs the granularity of learned explanations. Overcomplete codebooks, such as 36 codes in GridWorld or 16 in Arithmetic with many extras, still yield primitive-level codes when MDL is tuned appropriately [2605.03413]. By contrast, too large a $\lambda_{\mathrm{MDL}}$, for example 1.2, incentivizes overly short, entangled explanations and degrades transfer and primitiveness, whereas moderate values in the range 0.8–1.0 recover ground-truth-like explanation lengths and full primitive sets [2605.03413]. This suggests that compactness alone is insufficient; it must be balanced against state-grounded reconstructive adequacy.

The method’s stated limitations are also explicit. Experiments use discrete, small primitive sets and short programs, so scalability to very long horizons or complex continuous dynamics remains open [2605.03413]. Semantics are induced via reconstruction and are not guaranteed to align with human-interpretable or causal variables [2605.03413]. Deterministic execution and reconstruction-threshold stopping may be brittle in noisy or partially observable settings, and no explicit program annotations are provided, so learning can depend on encoder/decoder quality and latent manifold structure [2605.03413]. Computationally, NEO incurs sequential execution of $O(K)$ and test-time sampling of $O(K\times B)$; it trains approximately $2\times$ slower than single-pass baselines, though faster than gradient-based latent optimization at inference [2605.03413].

Within the broader literature, NEO is positioned as related to program induction and synthesis systems such as NTM, NPI, and DreamCoder, but it departs from them by learning a latent Language of Thought from raw observations without DSL supervision [2605.03413]. It also connects to latent action models such as LAPO, Genie, and AdaWorld, and to world models such as RSSM and Dreamer, while differing in its emphasis on transferable explanatory programs rather than monolithic predictive latents [2605.03413]. The paper’s stated contributions are to formulate Learning-to-Theorize, propose NEO with learned primitives, shared executor, MDL-based length selection, and state grounding, introduce OTIB, and demonstrate explanation-driven generalization across transfer, compositional OOD, and length OOD settings [2605.03413].

Future directions identified for the framework include stochastic transition executors, proper Bayesian priors over programs, richer grammars with branching, loops, and hierarchy, object-centric and relational primitives, causal and intervention-aware training signals, and jointly learning encoders and decoders end-to-end [2605.03413]. Proposed application domains include scientific modeling from raw data, robot manipulation or edit pipelines, explainable image or video editing, counterfactual simulation, compositional world modeling in reinforcement learning, and education and cognitive modeling [2605.03413]. A plausible implication is that NEO is best understood not as a finished world model, but as a research program for theory induction in which compositional explanation is the primary target of learning.

Source: https://www.emergentmind.com/topics/neural-theorizer-neo