Neural Language of Thought Model (NLoTM)

• NLoTM is a neurosymbolic framework that integrates language modeling, representation learning, and probabilistic reasoning to generate compositional mental structures.
• The model leverages a Semantic Vector-Quantized VAE and an autoregressive transformer to learn discrete object-centric representations and enforce a structured mental grammar.
• Empirical benchmarks on synthetic datasets demonstrate lower FID scores and competitive segmentation accuracy, indicating robust generalization and improved concept formation.

The Neural Language of Thought Model (NLoTM) presents a neurosymbolic framework for structured cognition, unifying advances in language modeling, representation learning, and probabilistic reasoning under a language-of-thought (LoT) paradigm. Rooted in Fodor's hypothesis that cognition operates on a symbolic, combinatorial mental language (“Mentalese”), NLoTM seeks neural mechanisms capable of learning, representing, and generating compositional mental structures from perception and language in an unsupervised fashion (Wu et al., 2024). Core implementations integrate discrete object-centric representation, generative neural architectures, and inference-optimized priors, with significant steps toward aligning artificial systems with human-like concept formation and structured world modeling (Wong et al., 2023, Piccinini, 11 Oct 2025).

1. Theoretical Foundations

The Language of Thought Hypothesis (LoTH), articulated by Fodor (1975), proposes that cognition comprises operations over structured symbolic representations featuring discrete constituents (word-like primitives) and combinatorial rules (sentence-like syntax). While discrete textual tokens provide a natural substrate for neural LLMs, LoTH contends that comparable latent logical structures underlie perception and reason, even in nonlinguistic domains (Wu et al., 2024).

Recent work draws a distinction between “Classical LOT”—requiring digital symbolic encoding and computation along von Neumann principles—and “Nonclassical LOT,” positing that neural computation manifests language-like structure through graded, recurrent states rather than digital symbol tokens (Piccinini, 11 Oct 2025). Empirical evidence and computational tractability strongly favor Nonclassical LOT realizations, instantiated in vectorial neural architectures that capture compositionality and systematicity without discrete digital processors.

2. Object-Centric Discrete Representation: The Semantic Vector-Quantized VAE

At the computational heart of NLoTM is the Semantic Vector-Quantized Variational Autoencoder (SVQ-VAE), an unsupervised object-centric generative model (Wu et al., 2024). SVQ-VAE extends the object slot paradigm by introducing a block-wise vector quantization step that converts continuous object representations into discrete, compositional “concept” tokens.

The SVQ-VAE encoder applies a convolutional backbone to images (e.g., CLEVR, Sprites), flattens features with positional encodings, and passes them through Slot Attention to produce $N$ slot vectors per image. Each slot, with dimension $d_s$, is split into $M$ blocks, each $d_c = d_s/M$ in size. Each block is quantized via separate codebooks ($K=64$ prototypes per block), yielding $N \times M$ discrete code indices per image. Gradient transmission relies on straight-through estimation, and codebook prototypes are updated by EMA with code collapse prevention mechanisms.

By sharing codebooks across all slots, NLoTM encourages semantic factor alignment: particular blocks specialize to latent object properties such as color, size, position, or material. Empirical latent traversals confirm that toggling a single block index for an object-specific slot alters a single interpretable property.

3. Generative Modeling via the Autoregressive LoT Prior

After training the SVQ-VAE, NLoTM fits an Autoregressive LoT Prior (ALP) over the discrete concept sequence $z = (z_1, ..., z_{NM})$ using a decoder-only transformer (Wu et al., 2024). The ALP is trained with cross-entropy over codebook indices, leveraging positional encoding to capture block and slot order. The autoregressive prior enforces a mental “grammar” that governs the probabilistic composition of object attribute tokens and joint distributions over object properties.

Sampling from the prior yields sequences arranging object-wise factor combinations, and qualitative samples demonstrate object-level compositionality—new objects and attribute combinations emerge systematically. The transformer prior consists of 8 layers, 4 heads, with dropout, and is optimized via Adam.

4. Evaluation: Generative Quality, Factorization, and Generalization

NLoTM is empirically benchmarked on 2D Sprites and CLEVR (Easy, Hard, Tex), with metrics including Fréchet Inception Distance (FID), classification accuracy, and ARI-based segmentation scores (Wu et al., 2024). Compared to VQ-VAE+PixelCNN, dVAE+transformer, and continuous-latent GENESIS-V2:

Dataset	FID VQ-VAE	FID dVAE	FID NLoTM	Accuracy (VQ-VAE / dVAE / NLoTM)
3 obj (no bg)	14.81	7.26	6.61	28.9% / 75.8% / 75.0%
4 obj (no bg)	26.35	19.15	17.93	21.9% / 62.5% / 66.4%
4 obj with bg	58.14	66.08	58.50	19.5% / 30.5% / 42.2%

On CLEVR, NLoTM achieves the lowest FID across splits (Easy: 32.50, Hard: 43.12, Tex: 84.52) compared to VQ-VAE (>57), dVAE (>40), and GENESIS-V2 (>93). Segmentation quality (FG-ARI) is competitive with leading slot-based methods: NLoTM achieves 91.4 (Easy), 90.5 (Hard), and 70.9 (Tex).

Downstream and OOD tasks include “Odd-One-Out” inference (99.1% OOD accuracy, outperforming VQ-VAE 55.6% and dVAE 26–29%), and property comparison tasks on CLEVR-Hard, where NLoTM codebook features achieve 75.9% (ID) and 71.2% (OOD), surpassing alternative autoencoder methods and competitive with SysBinder’s continuous slots.

5. Neural Hardware and Theoretical Considerations

NLoTM operationalizes Nonclassical LOT by leveraging architectures aligned with the neurocomputational realities of cortically distributed continuous computation (Piccinini, 11 Oct 2025):

• Representations: High-dimensional neural population vectors encode object properties, with compositional mechanisms emulating linguistic constituency via vector operations.
• Computation: Neural updates follow rate-based or spiking dynamics, not digital symbol manipulation. Compositional operators are realized by trainable matrix transforms or nonlinear gating, e.g., $$v(\alpha \cdot \beta) = \sigma(W [v(\alpha); v(\beta)] + b)$$.
• Learning: Synaptic plasticity, including Hebbian and spike-timing-dependent rules, enables the acquisition of structured dependencies and attribute correlations.
• Architecture: No digital memory–processor separation, no global clock—only distributed, recurrent, and plastic motif networks.

Empirical evidence from fMRI, MEG, and behavioral program-induction tasks favors graded, compositional neural encoding over classical digital symbol manipulation.

6. Probabilistic Language of Thought and Symbolic Integration

A complementary approach instantiates the NLoTM within a probabilistic language of thought (PloT) framework, as shown in language-informed generative modeling (Wong et al., 2023). Here, a “meaning” function, realized by LLMs such as Codex, maps natural language to Church-style probabilistic programs that operate over a symbolic substrate of types, stochastic and deterministic primitives, and compositional queries.

Inference employs Bayesian sampling (rejection, MCMC, SMC), granting systematic generalization and robust updates to beliefs under condition changes. Integrating symbolic modules—including physics engines, renderers, and planners—as first-class primitives enables compositional construction of richly structured world models. Empirical evaluation finds NLoTM matches human judgments within 5–10% across reasoning domains and robustly generalizes to new lexical items and OOD scenarios.

7. Limitations, Open Problems, and Future Directions

Current NLoTM realizations are principally validated on synthetic datasets with discrete factorization. Significant challenges remain:

• Scaling to complex natural scenes, large numbers of objects, and real-world visual factors.
• Foundational limits of discrete tokenization: some perceptual properties (e.g., position, illumination) are intrinsically continuous, motivating hybrid discrete-continuous bottlenecks (Wu et al., 2024).
• Computational efficiency for high-resolution and spatiotemporally extended data demands innovations in attention mechanisms and hierarchical generative modeling.
• Integrating flexible world-model construction, symbolic definition extension, and program induction for broader commonsense reasoning (Wong et al., 2023).
• Expanding empirical evaluation to probe neural correlates of compositional computations in biological systems (Piccinini, 11 Oct 2025).

NLoTM frameworks unify object-centric discrete representation, compositional probabilistic syntax, and neural hardware realism. This synthesis advances the field toward machine cognition characterized by robustness, systematic generalization, and cognitive plausibility, paralleling the symbolically compositional and generative capacities ascribed to the human mind (Wu et al., 2024, Wong et al., 2023, Piccinini, 11 Oct 2025).