Gradient-Based Program Synthesis with Neurally Interpreted Languages

Abstract: A central challenge in program induction has long been the trade-off between symbolic and neural approaches. Symbolic methods offer compositional generalisation and data efficiency, yet their scalability is constrained by formalisms such as domain-specific languages (DSLs), which are labour-intensive to create and may not transfer to new domains. In contrast, neural networks flexibly learn from data but tend to generalise poorly in compositional and out-of-distribution settings. We bridge this divide with an instance of a Latent Adaptation Network architecture named Neural Language Interpreter (NLI), which learns its own discrete, symbolic-like programming language end-to-end. NLI autonomously discovers a vocabulary of primitive operations and uses a novel differentiable neural executor to interpret variable-length sequences of these primitives. This allows NLI to represent programs that are not bound to a constant number of computation steps, enabling it to solve more complex problems than those seen during training. To make these discrete, compositional program structures amenable to gradient-based optimisation, we employ the Gumbel-Softmax relaxation, enabling the entire model to be trained end-to-end. Crucially, this same differentiability enables powerful test-time adaptation. At inference, NLI's program inductor provides an initial program guess. This guess is then refined via gradient descent through the neural executor, enabling efficient search for the neural program that best explains the given data. We demonstrate that NLI outperforms in-context learning, test-time training, and continuous latent program networks on tasks that require combinatorial generalisation and rapid adaptation to unseen tasks. Our results establish a new path toward models that combine the compositionality of discrete languages with the gradient-based search and end-to-end learning of neural networks.

• The paper introduces Neural Language Interpreter (NLI), a novel architecture that learns a discrete, symbolic-like neural language to enable compositional program synthesis.
• It employs an encoder-decoder framework with Gumbel-Softmax relaxation, allowing for differentiable execution and gradient-based test-time adaptation.
• Experimental results demonstrate near-perfect in-distribution accuracy and strong out-of-distribution performance, affirming NLI’s effectiveness in systematic generalisation.

Gradient-Based Program Synthesis with Neurally Interpreted Languages (NLI): An Authoritative Technical Review

Introduction

The paper "Gradient-Based Program Synthesis with Neurally Interpreted Languages" (2604.18907) proposes a novel architecture, Neural Language Interpreter (NLI), addressing the longstanding dichotomy between symbolic and neural approaches in program induction. Traditional symbolic systems offer superior compositional generalisation and can be data-efficient, but their rigidity, reliance on domain-specific languages, and exhaustive combinatorial search restrict scalability and domain transfer. Neural networks, conversely, achieve strong performance on i.i.d data but generalise poorly in compositional, out-of-distribution settings and conflate knowledge within parametric weights.

NLI presents a reconciling paradigm: it learns a domain-specific, symbolic-like neural language and an accompanying interpreter end-to-end. This approach automatically discovers primitive operations, supports variable-length latent program representations, and enables both training and inference via differentiable execution, leveraging Gumbel-Softmax relaxation for discrete sampling. Critically, NLI enables test-time adaptation by refining latent programs using gradient-based search directly through the neural executor.

Figure 1: Overview of NLI's inference, depicting end-to-end latent program token generation, neural program search refinement, and sequential token execution.

Neural Language Interpreter: Architecture and Methodology

NLI employs an encoder-decoder architecture, where the encoder (program inductor) maps input-output example pairs to sequences of discrete program tokens. These tokens encode the neural language, and their distribution is learned via Gumbel-Softmax, allowing for gradient flow and end-to-end differentiability. The decoder (neural interpreter) sequentially executes these tokens, updating an intermediate state at each step.

Distinct from prior architectures such as Latent Program Networks (LPN), NLI represents a program as a variable-length sequence over a learned compact codebook, rather than as a single continuous embedding. This structural shift enables compositional generalisation, as program length is unconstrained and primitive reuse is explicit.

The encoder aggregates input-output pairs by mean-pooling positional token embeddings, yielding permutation invariance and generalisation to test-set specification sizes. Each token position is projected via a feed-forward net to logits over the codebook, subsequently relaxed to discrete samples with Gumbel-Softmax. During training, the temperature is annealed to promote peaked, discrete activations.

Token reuse regularisation is introduced to enforce a compact, compositional latent space, penalising the expected number of distinct tokens per batch and biasing the encoder towards discovering primitives over memorising whole programs.

The decoder processes the latent token sequence recurrently, updating the intermediate state and output distribution at each step, leveraging skip-gating to handle programs shorter than the maximal sequence length. This recurrent execution is imperative for compositional generalisation, as confirmed by ablation studies.

Test-Time Neural Program Search

A salient feature of NLI is efficient search in latent program space at inference. The encoder provides an initial program token sequence, which is refined by gradient ascent in the continuous relaxation of the discrete space. Because the interpreter is differentiable, this refinement is feasible and effective—even for out-of-distribution tasks. The search objective maximises the joint likelihood of the specification over sampled Gumbel noise and relaxed program embeddings, with temperature annealing ensuring convergence to discrete solutions.

Multiple parallel initialisations are employed to escape local optima, and search steps are scalable across hardware. Unlike prior approaches, which rely on combinatorial or beam search, NLI leverages differentiable search, which enables more efficient adaptation and systematic compositional generalisation.

Experimental Results

Compositionality Benchmark

NLI is evaluated on a custom compositionality benchmark with tasks probing length extrapolation (Shift-L), primitive extraction (Shift-P), and novel function composition (Comp-I). Three inference strategies are compared: base encoder, prior search, and gradient search. All methods achieve near-perfect accuracy in-distribution.

For out-of-distribution splits, only NLI with gradient search achieves strong generalisation: 99% accuracy on Shift-L, 100% on Shift-P, and 91% on Comp-I. All other baselines—including in-context learning, LPN/D-LPN (with or without gradient search)—fail on OOD splits.

Analysis of Learned Primitives

The latent codes discovered by NLI display systematic reuse—e.g., a shift of $n$ is executed through repeated or composed tokens for shifts of length 1 and 2 (231 and 476), achieving compression and compositionality. This mechanism supports extrapolation to unseen shift magnitudes and composition tasks.

Ablation Studies

Componentwise ablations confirm essentiality for compositional generalisation: discrete tokenisation in both encoder and interpreter, recurrent interpreter dynamics, and skip tokens are indispensable for OOD extrapolation. Gumbel-Softmax relaxation is the primary driver of discrete, compositional behaviour; without it, generalisation collapses.

Figure 2: Comparison of fully neural baselines and NLI against neuro-symbolic methods on DeepCoder, illustrating competitive performance of NLI in settings without program annotations.

Scaling and Search

Test-time accuracy on Comp-I scales robustly with compute—more gradient steps and initialisations produce consistent accuracy improvements. NLI proves robust to search budget constraints, unlike methods susceptible to overfitting.

DeepCoder Benchmark

On DeepCoder, NLI achieves competitive performance relative to neuro-symbolic baselines (ExeDec, Transformer, Latent Programmer), despite training solely on input-output pairs and not accessing program annotations. NLI generalises more successfully to longer programs and novel concept compositions—critical for scalable program synthesis.

Relation to Prior Work

NLI's approach advances discrete latent representations for program induction, improving upon earlier architectures (e.g., Latent Programmer [Hong et al. 2021], CompILE [Kipf et al. 2019]) by enabling recurrent execution, end-to-end training without program supervision, and differentiable test-time search. Its test-time adaptability parallels models from meta-learning and latent adaptation networks, but its discrete compositional latent space offers unique inductive biases for systematic generalisation.

In contrast to neuro-symbolic methods and beam search approaches, NLI's differentiable search shifts program induction from combinatorial exploration to efficient local refinement in latent space, supporting rapid adaptation and compositionality.

Practical and Theoretical Implications

NLI demonstrates that end-to-end models can autonomously discover interpretable, reusable program primitives and generalise compositionally, without explicit DSLs or ground-truth program annotations. This has implications for scalable program synthesis and few-shot compositional reasoning, reducing engineering and search complexity.

On the theoretical axis, the architecture merges symbolic abstraction (via discrete latent codes) and neural flexibility (via gradient-based optimisation), supporting the conjecture that compositional generalisation hinges on discrete, structured latent spaces together with recurrent, sequential execution.

Challenges remain in scaling test-time search (compute cost, search efficiency), representing parameterised or conditional primitives, and supporting more expressive interpreters. Exploration of evolutionary or local search and architectural modifications are proposed for further research.

Conclusion

The paper introduces Neural Language Interpreter, an architecture fusing the compositional strengths of symbolic systems with the gradient-based learning and search capabilities of neural networks. Empirical results substantiate its systematic and combinatorial generalisation across compositional program synthesis tasks, confirming the necessity of discrete, sequential latent programs with recurrent execution for this performance. Theoretical and practical implications signal promising future directions for scalable, compositional program induction models, toward more general and adaptable AI systems.