LLM-ERM: Efficient Program Learning

• LLM-ERM is a framework for efficient program learning using a propose-and-verify strategy that replaces exhaustive enumeration with LLM-guided candidate generation.
• It leverages small labeled datasets to prompt a pretrained LLM, producing and validating succinct candidate programs on training and validation sets.
• The framework achieves dimension-invariant generalization on tasks like parity, pattern recognition, and primality testing, ensuring interpretable, audit-friendly solutions.

The LLM-ERM framework is a methodology for program learning and algorithmic synthesis that leverages LLMs to achieve sample-efficient and computationally feasible learning of succinct hypotheses. It replaces exhaustive enumeration typical of classical empirical risk minimization (ERM) in finite hypothesis spaces with an LLM-guided propose-and-verify strategy, recovering much of the statistical efficiency of finite-class ERM while controlling the combinatorial explosion in computational cost. The approach is specifically designed for problems where the target function admits a short program representation, such as parity computation, pattern recognition, or primality testing, and addresses the limitations of gradient-based learning in these domains (Singhal et al., 16 Oct 2025).

1. Framework Design: Propose-and-Verify over LLM-Guided Program Space

The core of LLM-ERM is a propose-and-verify loop:

• At each iteration, a prompt constructed from a small labeled subset (e.g., typically 100–200 training and 100 validation examples) is sent to a pretrained, reasoning-augmented LLM.
• The LLM responds with one (or more) candidate programs that are plausible solutions for the task, typically as short, human-readable code (e.g., Python).
• Each candidate program is compiled and tested for correctness first on a training set, then on a held-out validation set.
• The candidate with the lowest validation error is selected and, if the error is below a preset threshold, returned as the learned hypothesis; otherwise the process is iterated.

This design contrasts with traditional ERM, which either enumerates all possible short programs—a process exponential in code length—or relies on gradient-based optimization, which may require an exponentially larger number of samples to generalize on certain program classes.

2. Mathematical Formulation and Sample Complexity

LLM-ERM leverages the ERM generalization bound for finite hypothesis classes: $$\mathrm{err}_D(h) \leq \frac{L \cdot \log |\Sigma| + \log(2 L^2 / \delta)}{m}$$ where $L$ is the length of the target program in symbols, $|\Sigma|$ is the alphabet size, $m$ is the number of examples, and $\delta$ is the failure probability.

Rather than exploring all $|\Sigma|^L$ programs, LLM-ERM uses the LLM to guide exploration toward likely candidates conditioned on the data. This probabilistic focus, combined with the ERM-style selection on a validation set, means that the statistical efficiency is retained—sample complexity thus scales only logarithmically with the size of the actual program search space encountered.

3. Empirical Results: Efficiency vs. Gradient-Based Methods

On tasks including parity (full, half, k-parity), pattern matching, and primality testing, LLM-ERM achieves generalization using as few as 200 examples. For instance, in the parity benchmarks, LLM-ERM frequently synthesizes programs that generalize perfectly beyond the training set—the "dimension-invariant" performance—while stochastic gradient descent (SGD)-trained transformers overfit to the training data and fail to generalize even when provided with up to 100,000 samples.

LLM-ERM requires only a handful of LLM-guided proposals per experiment. In the reported settings, a batch size of $k$ candidates (e.g., $k=1$ or modestly larger) is used at each step, and the reasoning trace for each candidate is fully transparent and auditable.

4. Theoretical Bounds and Complexity Separations

Theoretical analysis in the framework establishes a clear separation in sample complexity and computational tractability between propose-and-verify (LLM-ERM), exhaustive enumeration, and gradient-based optimization:

• Exhaustive enumeration (length-first search) matches the ERM rate but is computationally intractable: $O(m \cdot |\Sigma|^L)$ time.
• SGD or coordinate-wise online mini-batch SGD are viewed as statistical query (SQ) algorithms; on short program families with high SQ dimension (e.g., parity functions on $n$ bits have SQ-dimension $2^n$), even these algorithms require $T = \Omega(d \epsilon^2 / B^{3/2})$ iterations for error below $1/2 - \epsilon$, with $d$ the SQ-dimension and $B$ the batch size.
• LLM-ERM, by leveraging LLMs’ capacity to bias the search toward “data-consistent” short programs, avoids the exponential blowup and simultaneously recovers the finite-class generalization guarantees of ERM.

5. Implementation Workflow

A practical LLM-ERM implementation follows:

1. Construct a prompt containing labeled data (inputs, outputs) and, optionally, context or prior reasoning chains.
2. Query a pretrained, reasoning-augmented LLM for code candidates; the prompt is updated between rounds as needed.
3. For each candidate:
   • Compile and run the code on the training and independent validation set.
   • Record the training and validation error.
   • Optionally, output the LLM's reasoning trace for audit and debugging.
4. Return the candidate with the lowest validation error, contingent on it achieving an error below the specified threshold.
5. If no candidate passes the threshold in $k$ tries, optionally repeat with new prompts or additional data.

For example, for parity on $n$ bits, the LLM may propose bit-wise XOR logic in Python, which is then directly checked for correctness. For pattern recognition or filter inference, the LLM may propose regular expressions or string processing code.

6. Interpretability, Auditing, and Human-in-the-Loop Potential

LLM-ERM produces hypotheses in the form of explicit programs that are both interpretable and auditable, allowing insight into the logic used to capture the true function. The stepwise reasoning traces offer a human-readable record of the proposal pathway, enhancing explainability and enabling detailed scrutiny, which is critical for applications demanding verifiable guarantees in code or domain logic.

Furthermore, each candidate code can be directly inspected, enabling straightforward human-in-the-loop correction or enhancement if the LLM's inductive biases misfire.

7. Position in the Program Synthesis Landscape and Extensions

LLM-ERM bridges the theoretical-statistical gap between classical program enumeration (sample-optimal but computationally prohibitive) and neural gradient-based learning (computationally efficient per step, but susceptible to sample inefficiency and overfitting in algorithmic tasks). The methodology is especially potent in domains characterized by succinct, compositional rules where program families are well-structured but hard for local optimization algorithms to learn efficiently.

A plausible implication is that on tasks with accessible, short, human-understandable representations, ERM augmented with strong, compositional LLM proposal mechanisms will dramatically reduce the effective sample and compute burdens for program learning far beyond those achievable with vanilla deep learning approaches.

Summary Table: LLM-ERM vs. Traditional Learning Approaches

Approach	Sample Efficiency	Computational Cost	Generalization on Short-Program Tasks
Exhaustive enumeration (ERM)	O(L·log	Σ	)
Gradient-based (SGD, transformers)	Requires exponential m	Linear per iteration	Often fails—overfits, poor generalization
LLM-ERM (propose & verify)	O(L·log	Σ	)

In conclusion, the LLM-ERM framework enables sample-efficient program learning with computational tractability by integrating LLM-guided proposal generation with ERM-style validation, achieving generalization and interpretability unattainable by classical learning paradigms on complex algorithmic tasks (Singhal et al., 16 Oct 2025).