Program Synthesis via Test-Time Transduction (2509.17393v2)

Abstract: We introduce transductive program synthesis, a new formulation of the program synthesis task that explicitly leverages test inputs during synthesis. While prior approaches to program synthesis--whether based on natural language descriptions or input-output examples--typically aim to generalize from training examples, they often struggle with robustness, especially in real-world settings where training examples are limited and test inputs involve various edge cases. To address this, we propose a novel framework that improves robustness by treating synthesis as an active learning over a finite hypothesis class defined by programs' outputs. We use an LLM to predict outputs for selected test inputs and eliminate inconsistent hypotheses, where the inputs are chosen via a greedy maximin algorithm to minimize the number of LLM queries required. We evaluate our approach on four benchmarks: Playgol, MBPP+, 1D-ARC, and programmatic world modeling on MiniGrid. We demonstrate that our method significantly improves program synthesis in both accuracy and efficiency. We release our code at https://github.com/klee972/SYNTRA.

• The paper presents SYNTRA, a transductive synthesis framework that uses test-time inputs to eliminate inconsistent candidate programs.
• It leverages LLM-driven candidate generation and a maximin active query selection algorithm that halves the number of required LLM calls.
• Experimental results demonstrate significant scalability and robustness across diverse domains, achieving up to 196% improvement in task accuracy.

Program Synthesis via Test-Time Transduction: A Technical Analysis

Introduction and Motivation

The paper introduces transductive program synthesis, a paradigm that leverages test inputs during synthesis to improve robustness and efficiency in program generation. Traditional inductive program synthesis methods, which generalize from a limited set of input-output examples, often fail to handle edge cases present in real-world applications such as spreadsheet automation and data transformation. The transductive approach, inspired by Vapnik's principle of solving only the problem of interest, focuses on synthesizing programs that are correct for a fixed set of test inputs, rather than aiming for broad generalization.

Figure 1: Spreadsheet auto-completion example illustrating the failure of inductive synthesis on edge cases.

SYNTRA Framework: Algorithmic Formulation

The core contribution is the SYNTRA (SYNthesis-by-TRAnsduction) framework, which formalizes transductive program synthesis as an active learning problem over a finite hypothesis class. The process consists of:

1. Candidate Program Generation: An LLM-based synthesis model generates a diverse set of candidate programs.
2. Hypothesis Class Construction: Programs consistent with training examples are executed on test inputs, and their output tuples are deduplicated to form the hypothesis class $\mathcal{H}$.
3. Active Query Selection: A greedy maximin algorithm selects test inputs that, in the worst case, eliminate the largest number of inconsistent hypotheses.
4. Transductive Prediction: An LLM-based transduction model predicts the most plausible output for the selected test input, given the specification and candidate outputs.
5. Hypothesis Elimination: Hypotheses inconsistent with the transductive prediction are removed, iterating until a single hypothesis remains.

This approach is computationally efficient, requiring $O(\log|\mathcal{H}|)$ queries under reasonable assumptions, and is scalable to large test sets.

Figure 2: Examples from Playgol, MBPP+, 1D-ARC, and MiniGrid domains, highlighting test outputs.

LLM-Based Instantiation and Diversity Enhancement

The program synthesis model is instantiated via LLM prompting, with a focus on maximizing semantic diversity. The AGA (Autoregressively Generated Algorithms) method prompts the LLM to first enumerate distinct algorithms before translating them into executable code, increasing the likelihood of including a correct program in the candidate pool. The transduction model, typically a more capable LLM, is prompted with zero-shot chain-of-thought reasoning and fuzzy string matching to select the most plausible output among candidates.

Experimental Evaluation

The framework is evaluated on four benchmarks: Playgol (string transformation), MBPP+ (Python programming), 1D-ARC (visual reasoning), and MiniGrid (programmatic world modeling). The evaluation protocol filters out trivial tasks and focuses on those where the hypothesis class contains both correct and incorrect candidates.

Key results include:

• Accuracy: SYNTRA achieves up to 196% improvement in task accuracy over baselines.
• Efficiency: The maximin query selection reduces the number of LLM calls by half compared to random selection, approaching the theoretical lower bound.
• Scalability: The number of transduction model calls increases sublinearly with the number of test inputs, demonstrating practical scalability.

  Figure 3: Experimental results showing sublinear scaling of LLM calls and improved generalization to unseen test sets.

Ablation Studies and Model Variations

Ablation studies reveal that diversity-enhancing strategies (AGA, MoC) are critical for robust synthesis, especially in algorithmic domains like MBPP+. SYNTRA can be layered atop advanced synthesis models (e.g., Mixture of Concepts) to further boost performance. Including test inputs directly in the prompt is effective only for small test sets; SYNTRA remains robust and scalable for larger sets.

Application to Programmatic World Modeling

In MiniGrid environments, SYNTRA improves the accuracy of synthesized world models, which is essential for downstream planning and policy learning. The framework is able to resolve ambiguities in state transitions (e.g., coordinate conventions) that are difficult to capture via inductive synthesis alone.

Discussion

Practical Implications

• Online and Human-in-the-Loop Extension: SYNTRA naturally extends to interactive settings, allowing for incremental hypothesis refinement and selective human queries.
• Interpretability and Extensibility: The framework produces executable programs, facilitating inspection, debugging, and integration into larger systems.
• Efficiency Trade-offs: While SYNTRA introduces an initial synthesis cost, its query efficiency and ability to use smaller synthesis models offset the cost for large test sets.

Theoretical Implications

• Probabilistic Extensions: The current framework is agnostic to program probabilities, but can be extended to incorporate uncertainty-based or committee-based query selection if such probabilities are available.
• Limitations: The approach assumes the availability of meaningful test inputs and is less effective when inputs lack semantic structure or when synthesis fails to produce correct candidates.

Future Directions

Potential avenues for future research include:

• Integrating probabilistic reasoning over programs and outputs.
• Extending to domains with latent or generated test inputs.
• Hybridizing with direct LLM transduction for tasks where code synthesis is infeasible.
• Addressing fairness and bias propagation from LLMs in transductive prediction.

Conclusion

Transductive program synthesis, as instantiated by the SYNTRA framework, offers a robust, scalable, and interpretable solution to program synthesis in settings with limited training data and diverse test inputs. By actively leveraging test-time information and combining LLM-based synthesis with transductive reasoning, the approach sets a new standard for accuracy and efficiency in real-world program synthesis tasks. The framework's extensibility and empirical performance suggest broad applicability across domains requiring reliable automation and edge-case robustness.