RefineStat: A Probabilistic Program Synthesis Framework

• RefineStat is a language model–driven framework that enforces semantic constraints to generate syntactically valid probabilistic programs.
• It applies local, diagnostic-aware refinement to adjust prior and likelihood components, ensuring convergence and statistical reliability.
• Empirical evaluations demonstrate that RefineStat nearly doubles reliability and outperforms syntax-only methods in probabilistic program synthesis.

RefineStat is a LLM–driven framework for probabilistic program synthesis that combines semantic constraint enforcement with diagnostic-aware refinement in order to produce programs that are both syntactically sound and statistically reliable. It is designed for statistical model discovery in probabilistic programming, where small LLMs frequently generate outputs with syntactic errors, semantic errors, and statistical or modeling failures such as ill-conditioned or non-convergent models. The framework targets this failure mode directly by constraining generation to valid probabilistic constructs and then revising prior or likelihood components whenever Bayesian workflow diagnostics fail (Kanda et al., 1 Sep 2025).

1. Problem setting and motivation

Probabilistic programming offers a powerful framework for modeling uncertainty, but statistical model discovery in this setting entails navigating an immense search space under strict domain-specific constraints. In this setting, both LLMs and small LLMs can produce code with invalid syntax, misuse of distributions, wrong parameters, or domain- and version-specific bugs. The difficulty is not limited to program execution: a generated model may run yet still be statistically unreliable because inference does not converge, effective sample size is inadequate, divergences occur, or predictive diagnostics are unstable (Kanda et al., 1 Sep 2025).

RefineStat is motivated by probabilistic programmers’ domain expertise and debugging strategies. Its central idea is a two-phase, domain-aware synthesis procedure. First, it enforces semantic constraints during code generation so that synthesized programs contain valid distributions and well-formed parameters. Second, it applies diagnostic-aware refinement by resampling prior or likelihood components whenever reliability checks fail. This design treats statistical reliability as part of synthesis rather than as a downstream afterthought.

A common misconception is that syntactic correctness is sufficient for usable probabilistic programs. RefineStat is explicitly constructed against that view. In the reported formulation, grammar-constrained decoding alone does not eliminate semantic bugs or statistical pathologies; the framework therefore couples semantic validation with a Bayesian workflow–style refinement loop.

2. Two-phase synthesis architecture

The RefineStat pipeline begins by prompting a small LLM with task and data. Program generation is then carried out under semantic constraints, after which the generated program is executed and evaluated with Bayesian workflow diagnostics. If the resulting model fails reliability checks, RefineStat selectively resamples the likelihood block and, if necessary, the prior block, rather than rebuilding the entire program. The final model is selected from the reliable candidates by predictive fit, specifically $\mathrm{ELPD\mbox{-}LOO}$ (Kanda et al., 1 Sep 2025).

This yields a reliability-driven search over probabilistic programs. The framework’s control flow is defined by acceptance, rejection, and local revision decisions rather than by unconstrained full-program sampling. A program is generated as a composition of data, prior, and likelihood fragments; semantically invalid fragments are rejected locally, and statistically unreliable complete programs are refined selectively. This decomposition is significant because it makes the search procedure sensitive to where failures occur.

The selection rule is formalized as

$M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$

where only models that satisfy a reliability criterion are admitted into $\mathcal{M}_\text{valid}$. A plausible implication is that RefineStat separates “can be generated” from “should be selected,” with the latter governed by explicit statistical diagnostics rather than language-model preference alone.

3. Semantic constraint enforcement

Semantic constraint enforcement is defined over a formal grammar for the target probabilistic programming language, such as PyMC or Stan. At each generation step, RefineStat checks three properties for a statement $s$ in context $\Pi$: $$\Phi(s, \Pi) = \phi_1(s, \Pi) \land \phi_2(s, \Pi) \land \phi_3(s, \Pi).$$ Here, $\phi_1$ is parseability, meaning that the statement conforms to the context-free grammar; $\phi_2$ is distribution validity, meaning that all probability distributions referenced exist in the target API or library; and $\phi_3$ is parameter validity, meaning that distribution and method arguments match the domain and specification of the corresponding constructs (Kanda et al., 1 Sep 2025).

This semantic validation goes beyond syntax-only filtering. Distribution validity excludes nonexistent or deprecated distribution calls, while parameter validity excludes argument names or parameterizations that are not appropriate for the current library version or probabilistic construct. The framework also maintains a symbol table mapping for module and alias resolution, so validity is checked in context rather than purely lexically.

When a fragment fails the predicate $\Phi$, RefineStat rejects and resamples only that fragment. This local resampling and backtracking scheme is described as efficient and token-saving. It also prunes the search space because infeasible or incoherent program paths are not allowed to develop into full candidate programs. In this sense, semantic constraints function both as correctness filters and as search-space reductions.

4. Diagnostic-aware refinement and reliability scoring

After a complete program has passed semantic validation, RefineStat runs the code and evaluates the resulting model with seven diagnostics standard in Bayesian modeling. These diagnostics are used to determine whether the model is reliable enough to retain or whether targeted refinement is required (Kanda et al., 1 Sep 2025).

Diagnostic	Criterion
Convergence	max split-$M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$0
Bulk ESS	$M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$1
Tail ESS	$M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$2
Divergences	$M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$3
BFMI	$M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$4
ELPD-LOO	Pointwise predictive density is finite
Pareto $M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$5	at least 80% of points with $M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$6

Each diagnostic contributes an indicator $M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$7, and the aggregate reliability score is

$M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$8

A model is considered valid when $M^* = \arg\max_{M \in \mathcal{M}_\text{valid}} \ \mathrm{ELPD\mbox{-}LOO}(M),$9, with the paper giving $\mathcal{M}_\text{valid}$0 as an example. This converts statistical reliability into an explicit acceptance criterion rather than an informal judgment.

If diagnostics fail, the refinement policy is targeted. RefineStat first resamples the likelihood block $\mathcal{M}_\text{valid}$1, holding fixed the data and prior; if repeated likelihood resampling exhausts its budget, it resamples the prior block $\mathcal{M}_\text{valid}$2. Each replacement block is again generated via constrained decoding. This mechanism is “diagnostic-aware” because it reacts to model failure after inference, and it is “local” because it does not discard the entire program when a single component appears responsible.

5. Empirical evaluation and comparative position

RefineStat is evaluated on five benchmark probabilistic-program synthesis tasks, including Eight Schools, Dugongs, Surgical, Peregrine, and GP, using five open-source small LLMs, including Llama 3 8B, CodeGemma 7B, Qwen2.5, and DeepSeek. The reported results show that the framework produces programs that are both syntactically sound and statistically reliable, often matching or surpassing outputs from closed-source LLMs such as OpenAI o3 (Kanda et al., 1 Sep 2025).

The empirical findings emphasize run rate, reliability, predictive fit, and ablation behavior. RefineStat attains run rates of approximately 50%, compared with approximately 10–20% for unconstrained small-language-model baselines and approximately 20% for syntax-only approaches. In terms of predictive performance, the selected models achieve $\mathcal{M}_\text{valid}$3 values that are equal or higher than those of small-language-model baselines and, in several cases, match or outperform BoxLM and OpenAI o3. Ablation results identify parameter validity checking as especially important, with an approximately 14.5% drop in compilation rate when it is omitted.

The comparative framing is explicit. Syncode is described as enforcing syntax-only constraints, whereas RefineStat additionally enforces semantic and domain-specific correctness. BoxLM is described as using two GPT-4 systems, one for generation and one for critique, making it closed-source and costly. Previous approaches to probabilistic program synthesis are described as often requiring human sketches, hand-designed search, or heavy manual curation. Against this background, RefineStat positions itself as an automated framework that integrates constraint enforcement, Bayesian diagnostics, and targeted revision into one loop.

6. Scope, efficiency, and limitations

RefineStat’s efficiency claim rests on fragment-level correction rather than full-program regeneration. Local resampling and semantic pruning reduce wasted search, though the paper reports that the framework uses, on average, about twice as many tokens as unconstrained decoding. This is presented as a reasonable cost for more than doubling reliability. The refinement budgets are configurable, including the number of rejections, the number of valid programs to collect, and the maximum number of resamples per block (Kanda et al., 1 Sep 2025).

The framework is also evaluated for robustness beyond straightforward memorization effects. The reported tests indicate that outcomes are robust to prompt anonymization and data obfuscation, which suggests that the gains are not merely due to recovering memorized benchmark templates. A plausible implication is that the combination of semantic validation and post-inference diagnostic filtering induces a more procedural form of competence than prompt-only generation.

The remaining failures are characterized concretely. They arise from small-language-model local attractors, overused or deprecated APIs, API mismatches, and numerical instability not captured by the current constraint set. These limitations are important because they define the current boundary of the method: semantic correctness and diagnostic awareness substantially improve synthesis, but they do not eliminate all failure modes in probabilistic programming.

In summary, RefineStat defines probabilistic program synthesis as a constrained and diagnostically mediated search problem. Its distinguishing contribution is the combination of semantic constraint enforcement during decoding with post hoc diagnostic-aware refinement of prior and likelihood components. Within the reported evaluation, this combination makes open, smaller models competitive with larger and more expensive alternatives on probabilistic-programming code-generation tasks (Kanda et al., 1 Sep 2025).