Neural Theorizer (NEO): Explanatory Program Induction

• Neural Theorizer (NEO) is a probabilistic neural model that inductively infers executable, compositional programs as explicit theories from raw, non-textual data.
• It leverages a shared encoder, decoder, and theory programmer to learn discrete, reusable primitives that support systematic generalization and transfer across contexts.
• NEO employs MDL-based length selection and state grounding to generate concise, explanation-driven representations, outperforming traditional predictive world models.

Searching arXiv for the specified NEO paper and the acronym-overlap paper to ground the article in current arXiv records. Neural Theorizer (NEO) is a probabilistic neural model introduced within the Learning-to-Theorize (L2T) paradigm for inferring explicit explanatory theories of the world from raw, non-textual observations (Baek et al., 5 May 2026). In this formulation, a theory is an executable, compositional program built from learned primitives whose semantics are induced from data rather than specified a priori. NEO operationalizes understanding not as accurate future prediction alone, but as discovering abstract, reusable, compositional mechanisms that explain how observations are generated and transformed, support intervention, and transfer across instances (Baek et al., 5 May 2026).

1. Conceptual basis and relation to world modeling

Learning-to-Theorize reframes “understanding” as inferring explicit explanatory theories of how observations are generated and transformed (Baek et al., 5 May 2026). The motivating claim is drawn from developmental “theory-theory”: children build internal theories before mature language, and these theories function as abstract, reusable, compositional mechanisms rather than as instance-specific predictive mappings. In NEO, this perspective is instantiated by treating programs as “mental programs” whose compositionality yields systematic generalization and productivity, including length generalization to longer programs (Baek et al., 5 May 2026).

This contrasts with standard world models that optimize predictive reconstruction such as next-frame prediction. In the L2T view, conventional predictive objectives often learn entangled, instance-specific mappings or monolithic latent actions, whereas NEO induces explicit programmatic explanations with shared primitives and a shared executor (Baek et al., 5 May 2026). The distinction is not merely architectural. NEO explicitly targets transfer of inferred programs across new inputs and unseen compositions, together with length generalization to longer programs than those observed during training (Baek et al., 5 May 2026).

A common source of confusion is the acronym itself. “NEO” in this context denotes the Neural Theorizer of “Learning to Theorize the World from Observation” (Baek et al., 5 May 2026), not the unrelated test-time adaptation method “NEO: No-Optimization Test-Time Adaptation through Latent Re-Centering” (Murphy et al., 7 Oct 2025). The two works share an acronym but address different problem classes: the former concerns latent program induction and explanation-driven generalization, while the latter concerns optimization-free test-time adaptation through feature re-centering.

2. Probabilistic formulation and latent program semantics

NEO models each phenomenon as a source–target pair $(x,y)$ and introduces a latent program $\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$, an ordered sequence over a finite primitive set $\mathcal{Z}=\{z_1,\dots,z_M\}$ (Baek et al., 5 May 2026). Latent states are denoted $s=(s_1,\dots,s_{K+1})$, with $s_1 = E_\theta(x)$ and each subsequent state obtained by executing a primitive from the current state (Baek et al., 5 May 2026). The shared transition model $T$, implemented as either $p_\theta(s_{k+1}\mid s_k,z)$ or a deterministic counterpart $f_\theta$, assigns operational semantics to primitive symbols (Baek et al., 5 May 2026).

For observation pairs, the conditional likelihood is marginalized over the latent program and execution trace:

$$p_\theta(y \mid x) = \int p_\theta(y \mid s_{K+1})\, p_\theta(\tau,s \mid x)\, d\tau\, ds.$$

The prior over program and execution trace factorizes as

$$p_\theta(\tau,s \mid x) = p_\theta(s_1 \mid x) \;\prod_{k=1}^{K} p_\theta(z_{i_k} \mid s_k)\, p_\theta(s_{k+1} \mid s_k, z_{i_k}).$$

Here, $\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$0 functions as a state-conditioned program prior, while $\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$1 is the observation likelihood via the decoder (Baek et al., 5 May 2026).

Training uses an amortized posterior $\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$2 and maximizes the ELBO:

$\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$3

The posterior shares the encoder and executor with the generative model:

$\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$4

In practical implementation, execution is deterministic,

$\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$5

with reconstruction loss $\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$6 (Baek et al., 5 May 2026).

Program length is selected by a Minimum Description Length criterion:

$\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$7

This penalizes longer explanations while favoring the shortest accurate theory (Baek et al., 5 May 2026). A plausible implication is that NEO’s notion of explanation is explicitly complexity-controlled rather than emerging only indirectly from predictive fit.

3. Architecture and learned Language of Thought

NEO consists of an observation encoder $\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$8, a decoder $\pi \equiv \tau = (z_{i_1}, \dots, z_{i_K})$9, a program induction network $\mathcal{Z}=\{z_1,\dots,z_M\}$0 called the theory programmer, a discrete codebook implemented via VQ-VAE, and a shared executor $\mathcal{Z}=\{z_1,\dots,z_M\}$1 (Baek et al., 5 May 2026). The encoder and decoder map raw, non-text inputs to latent states and back; for GridWorld and Image Editing, a pretrained CNN VAE defines the latent space, whereas Arithmetic uses learned embeddings and a linear decoder (Baek et al., 5 May 2026). These components are shared across training examples and across the generative and variational paths.

The theory programmer is a goal-conditioned policy over primitive symbols. It takes the current latent state $\mathcal{Z}=\{z_1,\dots,z_M\}$2 and the encoded target $\mathcal{Z}=\{z_1,\dots,z_M\}$3 and outputs a categorical distribution over codebook entries (Baek et al., 5 May 2026). Discreteness is implemented with a VQ-VAE codebook $\mathcal{Z}=\{z_1,\dots,z_M\}$4, yielding a learned symbolic vocabulary and inducing programs by iteratively selecting $\mathcal{Z}=\{z_1,\dots,z_M\}$5 (Baek et al., 5 May 2026). The primitive set $\mathcal{Z}=\{z_1,\dots,z_M\}$6 is therefore not preset; its symbols are learned as VQ codes, and the grammar is sequence composition $\mathcal{Z}=\{z_1,\dots,z_M\}$7 (Baek et al., 5 May 2026).

The executor is shared across all phenomena and operationally defines primitive meaning through its effect on latent states (Baek et al., 5 May 2026). Intermediate states are decoded both for MDL length selection and for a state grounding regularizer,

$\mathcal{Z}=\{z_1,\dots,z_M\}$8

where $\mathcal{Z}=\{z_1,\dots,z_M\}$9 denotes stop-gradient and the update applies only to the transition model (Baek et al., 5 May 2026). This keeps intermediate states on the observation manifold.

The program representation is syntactically simple: a flat sequence of primitive symbols executed by sequential application of the shared transition model (Baek et al., 5 May 2026). In the Image Editing example, learned primitive codes correspond to rot (90° clockwise), br_p (brightness+), and mask (gray square mask), so that

$s=(s_1,\dots,s_{K+1})$0

induces the trace

$s=(s_1,\dots,s_{K+1})$1

with $s=(s_1,\dots,s_{K+1})$2 (Baek et al., 5 May 2026). The same inferred program can then be transferred to a new input $s=(s_1,\dots,s_{K+1})$3 to obtain $s=(s_1,\dots,s_{K+1})$4 generated by the same latent program (Baek et al., 5 May 2026). This is the core operational meaning of explanation-driven transfer in NEO.

4. Training, inference, and test-time scaling

Training initializes pretrained $s=(s_1,\dots,s_{K+1})$5 and $s=(s_1,\dots,s_{K+1})$6 together with the theory programmer $s=(s_1,\dots,s_{K+1})$7, executor $s=(s_1,\dots,s_{K+1})$8, a maximum step count $s=(s_1,\dots,s_{K+1})$9, and learning rate $s_1 = E_\theta(x)$0 (Baek et al., 5 May 2026). For each minibatch of pairs $s_1 = E_\theta(x)$1, the model computes $s_1 = E_\theta(x)$2 and $s_1 = E_\theta(x)$3, then iterates for $s_1 = E_\theta(x)$4: sample or select a primitive $s_1 = E_\theta(x)$5, execute $s_1 = E_\theta(x)$6, and optionally decode each $s_1 = E_\theta(x)$7 to $s_1 = E_\theta(x)$8 (Baek et al., 5 May 2026). MDL selects

$s_1 = E_\theta(x)$9

after which the model applies reconstruction, grounding, and VQ losses, truncated to $T$0 (Baek et al., 5 May 2026).

The practical objective is

$T$1

This combines program-conditioned reconstruction with discrete codebook learning and latent-state grounding (Baek et al., 5 May 2026).

At inference, NEO receives a support pair $T$2, rolls out the theory programmer greedily with

$T$3

and uses MDL to obtain an inferred program $T$4 (Baek et al., 5 May 2026). The model then transfers $T$5 to a query input $T$6 by applying the same executor sequence to $T$7 and decoding the result (Baek et al., 5 May 2026). Evaluation distinguishes self-explainability on the support pair and transferability on the query pair.

A further variant, NEO-S, performs test-time scaling by sampling $T$8 candidate programs on the support pair using a temperature $T$9 over the programmer’s softmax, filtering candidates with MDL, and selecting the most frequent successful program by majority voting (Baek et al., 5 May 2026). Larger $p_\theta(s_{k+1}\mid s_k,z)$0 and higher temperature improve exploration and length-OOD performance, notably in Arithmetic (Baek et al., 5 May 2026). This suggests that, in some regimes, the limiting factor is not primitive discovery but long-horizon search over compositions.

5. Benchmark structure and empirical behavior

The Observation-to-Theory Induction Benchmark (OTIB) contains three domains: GridWorld, Arithmetic Factorization Reasoning, and Image Editing (Baek et al., 5 May 2026). Each test instance provides a support pair $p_\theta(s_{k+1}\mid s_k,z)$1 and a query pair $p_\theta(s_{k+1}\mid s_k,z)$2 sharing the same latent program $p_\theta(s_{k+1}\mid s_k,z)$3, so the model must infer $p_\theta(s_{k+1}\mid s_k,z)$4 from the support pair and transfer it to the query input (Baek et al., 5 May 2026). Evaluation uses self-explainability and transferability, with splits into in-distribution, compositional OOD, and length OOD regimes (Baek et al., 5 May 2026).

Domain	Latent primitives and training lengths	Evaluation
GridWorld	10×10 grid; up/down/left/right; train length 1–3; test length 4–8	Exact match accuracy
Arithmetic	Integer pairs with $p_\theta(s_{k+1}\mid s_k,z)$5; train length 1–3; test length 4–6	Exact correctness
Image Editing	CIFAR-10 with 8 edit primitives; train length 1–2; test length 3–4	L1 pixel loss

In GridWorld, at $p_\theta(s_{k+1}\mid s_k,z)$6, NEO achieves transfer 0.933 on compositional OOD and 0.845 on length OOD; NEO-S with $p_\theta(s_{k+1}\mid s_k,z)$7 improves these to 0.976 and 0.907, while monolithic baselines collapse on OOD, near 0 (Baek et al., 5 May 2026). The same study reports robustness across $p_\theta(s_{k+1}\mid s_k,z)$8 and $p_\theta(s_{k+1}\mid s_k,z)$9, with NEO maintaining high transfer accuracy while baselines fail to generalize (Baek et al., 5 May 2026).

In Arithmetic Factorization, NEO shows strong compositional OOD behavior, for example 0.573 at $f_\theta$0, but greedy inference struggles on long sequences, with length OOD around 0.02–0.04 (Baek et al., 5 May 2026). NEO-S raises length OOD to 0.696 at $f_\theta$1 and 0.707 at $f_\theta$2 using $f_\theta$3 and temperature, indicating that primitives were learned and that additional test-time search resolves long-horizon recomposition (Baek et al., 5 May 2026).

In Image Editing, NEO yields the lowest L1 distances across all $f_\theta$4 splits for compositional and length OOD; the reported compositional OOD range is approximately 0.09–0.12 versus baselines at approximately 0.12–0.18, and length OOD is approximately 0.10–0.13, substantially better than the identity baseline (Baek et al., 5 May 2026). Qualitative examples show NEO decomposing novel edit sequences such as brightness+ followed by mask and selecting instance-specific $f_\theta$5 values matching ground-truth step counts under MDL, whereas baselines attempt monolithic mappings and fail (Baek et al., 5 May 2026).

6. Ablations, limitations, and research significance

Ablation analysis attributes a central role to state grounding. Removing $f_\theta$6 causes catastrophic collapse: primitiveness is approximately 0.002, and both self-explainability and transferability drop to zero (Baek et al., 5 May 2026). The reported interpretation is that intermediate states drift off-manifold and reusable building blocks are not discovered (Baek et al., 5 May 2026). Code–primitive alignment matrices further show near one-to-one mapping between learned codes and ground-truth primitives, even when many primitives are never seen in isolation, and the primitiveness metric often approaches 1.0, surpassing the fraction directly observable in training (Baek et al., 5 May 2026).

MDL tuning governs the granularity of learned explanations. Overcomplete codebooks, such as 36 codes in GridWorld or 16 in Arithmetic with many extras, still yield primitive-level codes when MDL is tuned appropriately (Baek et al., 5 May 2026). By contrast, too large a $f_\theta$7, for example 1.2, incentivizes overly short, entangled explanations and degrades transfer and primitiveness, whereas moderate values in the range 0.8–1.0 recover ground-truth-like explanation lengths and full primitive sets (Baek et al., 5 May 2026). This suggests that compactness alone is insufficient; it must be balanced against state-grounded reconstructive adequacy.

The method’s stated limitations are also explicit. Experiments use discrete, small primitive sets and short programs, so scalability to very long horizons or complex continuous dynamics remains open (Baek et al., 5 May 2026). Semantics are induced via reconstruction and are not guaranteed to align with human-interpretable or causal variables (Baek et al., 5 May 2026). Deterministic execution and reconstruction-threshold stopping may be brittle in noisy or partially observable settings, and no explicit program annotations are provided, so learning can depend on encoder/decoder quality and latent manifold structure (Baek et al., 5 May 2026). Computationally, NEO incurs sequential execution of $f_\theta$8 and test-time sampling of $f_\theta$9; it trains approximately $$p_\theta(y \mid x) = \int p_\theta(y \mid s_{K+1})\, p_\theta(\tau,s \mid x)\, d\tau\, ds.$$0 slower than single-pass baselines, though faster than gradient-based latent optimization at inference (Baek et al., 5 May 2026).

Within the broader literature, NEO is positioned as related to program induction and synthesis systems such as NTM, NPI, and DreamCoder, but it departs from them by learning a latent Language of Thought from raw observations without DSL supervision (Baek et al., 5 May 2026). It also connects to latent action models such as LAPO, Genie, and AdaWorld, and to world models such as RSSM and Dreamer, while differing in its emphasis on transferable explanatory programs rather than monolithic predictive latents (Baek et al., 5 May 2026). The paper’s stated contributions are to formulate Learning-to-Theorize, propose NEO with learned primitives, shared executor, MDL-based length selection, and state grounding, introduce OTIB, and demonstrate explanation-driven generalization across transfer, compositional OOD, and length OOD settings (Baek et al., 5 May 2026).

Future directions identified for the framework include stochastic transition executors, proper Bayesian priors over programs, richer grammars with branching, loops, and hierarchy, object-centric and relational primitives, causal and intervention-aware training signals, and jointly learning encoders and decoders end-to-end (Baek et al., 5 May 2026). Proposed application domains include scientific modeling from raw data, robot manipulation or edit pipelines, explainable image or video editing, counterfactual simulation, compositional world modeling in reinforcement learning, and education and cognitive modeling (Baek et al., 5 May 2026). A plausible implication is that NEO is best understood not as a finished world model, but as a research program for theory induction in which compositional explanation is the primary target of learning.