ModelSMC: SMC for LLM Model Discovery

• The paper introduces ModelSMC, a Sequential Monte Carlo–based algorithm that reformulates mechanistic model discovery as probabilistic inference over executable models.
• It employs a particle system with resampling, propagation via LLM-generated proposals, and surrogate likelihood weighting to efficiently explore the model space.
• Empirical evaluations on synthetic and real-world tasks demonstrate that the method outperforms single-particle variants while offering interpretable feedback despite challenges like computational cost and surrogate bias.

Searching arXiv for the target paper and closely related SMC/model-discovery references. ModelSMC is a Sequential Monte Carlo–based algorithm for LLM-driven mechanistic model discovery that recasts model discovery as probabilistic inference over executable mechanistic models (Wahl et al., 20 Feb 2026). In this formulation, a mechanistic model $M$ is treated as a latent object—such as a program or simulator implementation—and discovery consists of approximating the posterior $p(M\mid D)$ from observational data $D$. Candidate models are represented as weighted particles, iteratively proposed and refined by a LLM, and scored using likelihood-based criteria derived from simulator behavior or its surrogate approximation. The framework is presented as a probabilistic alternative to agentic-style heuristic workflows, with the stated aim of unifying model proposal, refinement, and selection within a single inference framework (Wahl et al., 20 Feb 2026).

1. Probabilistic formulation

ModelSMC defines the target of inference as the posterior over mechanistic models

$p(M\mid D)\propto p(D\mid M)\,p(M),$

where $M$ denotes an executable mechanistic model and $D=\{(x_o^j,c_o^j)\}_{j=1}^M$ is the observed dataset of $M$ observations $x_o$ under contexts $c_o$ (Wahl et al., 20 Feb 2026). The prior $p(M)$ is not given in closed form; instead, it is the marginal distribution over programs induced by an LLM under an unconditioned “model-generation” prompt. The likelihood decomposes under an iid assumption,

$p(M\mid D)$0

with each term marginalized over latent simulator parameters,

$p(M\mid D)$1

Here $p(M\mid D)$2 is a known prior over simulator parameters, and the simulator likelihood is typically intractable, so it is approximated using a surrogate density estimator $p(M\mid D)$3 via Neural Likelihood Estimation (Wahl et al., 20 Feb 2026).

The explicit assumptions are that data are exchangeable and conditionally independent given $p(M\mid D)$4, that the surrogate $p(M\mid D)$5 approximates $p(M\mid D)$6 sufficiently well for weighting purposes, and that the LLM prior $p(M\mid D)$7 has support over all scientifically plausible programs (Wahl et al., 20 Feb 2026). These assumptions are central because they connect an LLM-driven code-generation loop to a Bayesian posterior over mechanistic hypotheses rather than to a purely heuristic search trajectory.

This probabilistic view situates ModelSMC within the broader SMC tradition in which inference proceeds through weighted particle populations over a sequence of approximations. In classical SMC, particles are propagated, weighted, and resampled as targets evolve; the same pattern appears here, except that the particles are programs rather than state vectors or parameter points. This suggests an analogy with SMC samplers for graphical models, where a sequence of auxiliary targets on growing spaces is approximated by weighted particles (Naesseth et al., 2014), and with adaptive SMC methods for model comparison, where normalizing-constant estimation and ESS-based control govern transitions between targets (Zhou et al., 2013).

2. Particle representation and sequential update mechanics

ModelSMC approximates $p(M\mid D)$8 by a population of $p(M\mid D)$9 weighted particles

$D$0

over $D$1 iterations, where each particle carries a current model $D$2, a normalized weight $D$3, and auxiliary context $D$4 such as performance-feedback strings (Wahl et al., 20 Feb 2026). The iteration structure is composed of resampling, propagation, weighting, and feedback generation.

Resampling forms ancestor indices

$D$5

that is, sampling with replacement according to the previous particle weights (Wahl et al., 20 Feb 2026). Propagation then uses a mixture kernel controlled by a cloning probability $D$6. For each particle, a uniform draw $D$7 determines whether the particle is simply cloned from its ancestor or replaced by a new LLM-generated proposal conditioned on the ancestor model, its context, and a task prompt. The clone branch is described as balancing exploitation, while new LLM proposals implement exploration (Wahl et al., 20 Feb 2026).

Weighting is based on the marginal likelihood of the full dataset under each proposed model. Let $D$8 denote the surrogate marginalization

$D$9

approximated by Monte Carlo. The raw weight is the geometric mean over observations,

$p(M\mid D)\propto p(D\mid M)\,p(M),$0

A temperature $p(M\mid D)\propto p(D\mid M)\,p(M),$1 may optionally be applied for numerical stability through

$p(M\mid D)\propto p(D\mid M)\,p(M),$2

followed by normalization (Wahl et al., 20 Feb 2026).

The paper further states that under the prior-matching proposal assumption $p(M\mid D)\propto p(D\mid M)\,p(M),$3, the SMC weight update reduces to

$p(M\mid D)\propto p(D\mid M)\,p(M),$4

because the transition term cancels (Wahl et al., 20 Feb 2026). This cancellation is an important structural point: it gives a direct inferential interpretation to the weights and distinguishes the framework from ad hoc ranking heuristics.

An optional effective sample size is defined as

$p(M\mid D)\propto p(D\mid M)\,p(M),$5

Although the method can resample only when $p(M\mid D)\propto p(D\mid M)\,p(M),$6 falls below a threshold such as $p(M\mid D)\propto p(D\mid M)\,p(M),$7, the implementation reported in the paper resamples every iteration for simplicity (Wahl et al., 20 Feb 2026). The use of ESS as a degeneracy diagnostic mirrors standard SMC practice in state-space models and tempering-based samplers [(Chopin et al., 2011); (Zhou et al., 2013)].

3. Prompting, refinement, and LLM-mediated proposal dynamics

The LLM is used in ModelSMC both to generate new model proposals and to produce structured feedback for subsequent revisions (Wahl et al., 20 Feb 2026). The task prompt $p(M\mid D)\propto p(D\mid M)\,p(M),$8 is specified as a fixed text block containing a system description, a signature description, and a task description formulated as “Revise the simulator to better match the data; preserve interpretability.” The system description includes the scientific domain, data description, and evaluation metrics; the signature description specifies a function signature or code skeleton and the required language (Wahl et al., 20 Feb 2026).

The propagation prompt template is few-shot in style. It includes ancestor model code, a performance summary such as “MSE=0.12, failure on negative inputs,” and instructions to produce revised code implementing listed feedback points and return valid Python code (Wahl et al., 20 Feb 2026). This makes the propagation step neither unconditional generation nor free-form code completion; it is an explicitly context-conditioned proposal kernel over programs.

Feedback generation is two-stage. After simulating a model, the system computes numeric metrics such as a log-likelihood estimate or runtime errors. Stage 1 asks the LLM to diagnose structural issues from the simulation results, while Stage 2 requests JSON with fields $p(M\mid D)\propto p(D\mid M)\,p(M),$9 to guide the next proposal (Wahl et al., 20 Feb 2026). Auxiliary context strings $M$0 therefore function as a memory of failure modes and quality signals, and are passed forward through the particle lineage.

A notable feature is that this prompt-and-feedback loop is embedded inside a Monte Carlo sampler rather than operating as an independent optimization routine. This suggests that prompt engineering is not external to the inference procedure but part of the proposal mechanism. A plausible implication is that changes in prompt design affect proposal quality in much the same way that proposal kernels affect variance and mixing in conventional SMC samplers. The paper makes this point indirectly in its limitations section, where proposal quality is tied to prompt design and retrieval of similar scientific code (Wahl et al., 20 Feb 2026).

4. Empirical evaluations

The reported empirical evaluation compares ModelSMC with $M$1 against two baselines: FunSearch+, described as “an LLM-evolutionary method extended with likelihood-based scoring,” and ModelSMC $M$2, a single-particle variant (Wahl et al., 20 Feb 2026). Each method is run with ten random seeds. The reported performance metric is the median negative average log-likelihood per observation across the best particle per run (Wahl et al., 20 Feb 2026).

Task	ModelSMC	FunSearch+
SIR (synthetic)	–503.4	–500.0
Kidney (QSP)	43.6	N/A
Hodgkin–Huxley	25.2	28.9

Task	ModelSMC $M$3
SIR (synthetic)	–464.2
Kidney (QSP)	46.3
Hodgkin–Huxley	25.9

On the synthetic SIR task, ModelSMC is reported to “rapidly concentrate posterior mass on the ground-truth dynamics,” to match FunSearch+, and to outperform the single-particle variant (Wahl et al., 20 Feb 2026). On the kidney task, starting from an R-implemented aldosterone regulation model with 40 data points, ModelSMC recovers feedback mechanisms close to the original model, whereas the single-particle version lags behind (Wahl et al., 20 Feb 2026). On Hodgkin–Huxley neurons, the method identifies extensions such as slow $M$4 or persistent $M$5 currents that improve fit to real Allen-DB recordings and provides a posterior over structural variants (Wahl et al., 20 Feb 2026).

The abstract further states that experiments on real-world scientific systems show that the formulation discovers models with interpretable mechanisms and improves posterior predictive checks (Wahl et al., 20 Feb 2026). Since the article’s core evidence centers on these three tasks, the empirical characterization of ModelSMC is primarily as a posterior-search mechanism over simulator structure rather than a purely discriminative model-selection tool.

These experiments also reveal a contrast with the single-particle regime. ModelSMC $M$6 keeps the same LLM-based proposal machinery but removes the particle-population aspect. The reported degradation suggests that population-based exploration and resampling matter in this setting, especially when the proposal distribution is itself highly stochastic and structurally heterogeneous.

5. Relation to Sequential Monte Carlo methodology

ModelSMC belongs to the family of SMC samplers, but its object of inference is a space of executable mechanistic models rather than state trajectories, continuous parameters, or graph configurations (Wahl et al., 20 Feb 2026). This is methodologically significant because SMC has historically been used to approximate evolving targets over latent states or parameters, often with intractable likelihood increments handled by nested Monte Carlo methods (Chopin et al., 2011). In ModelSMC, the latent variable is a program, and the proposal kernel is partly instantiated by an LLM.

The closest general analogy is to SMC samplers that target a sequence of distributions over expanding or transformed spaces, with resampling and mutation steps designed to control degeneracy. In graphical-model SMC, one constructs a sequential decomposition and approximates each auxiliary target with particles, proposals, and weight corrections (Naesseth et al., 2014). In adaptive SMC for model comparison, particles traverse tempered distributions while ESS or CESS governs transition schedules and normalizing-constant estimation supports evidence evaluation (Zhou et al., 2013). ModelSMC preserves the particle-weight-resample pattern but departs from these formulations by allowing highly nonparametric, language-model-conditioned proposals.

The paper explicitly states that under idealized assumptions—exact likelihoods, prior-matching proposal $M$7, support coverage, and bounded weights—ModelSMC is a consistent SMC sampler targeting $M$8, with error $M$9 (Wahl et al., 20 Feb 2026). This imports a standard asymptotic claim from SMC theory into the model-discovery setting. However, the same section emphasizes that in practice the context-conditioned LLM proposals and surrogate likelihoods introduce approximation (Wahl et al., 20 Feb 2026). The resulting status is therefore not exact posterior inference in the strict sense of SMC$D=\{(x_o^j,c_o^j)\}_{j=1}^M$0, where unbiased nested likelihood estimates and PMCMC rejuvenation preserve the correct target (Chopin et al., 2011), but an approximate SMC framework whose probabilistic objective remains explicit.

This distinction matters for interpretation. ModelSMC is probabilistic in formulation, but its practical correctness depends on two approximation layers: the LLM-induced proposal law over programs and the surrogate density used to approximate simulator likelihoods. A plausible implication is that the framework provides a principled scaffold for future exactness-improving components, such as uncertainty-aware surrogates or more controlled rejuvenation kernels, rather than claiming exact Bayesian computation under present implementations.

6. Limitations, computational profile, and extensions

The paper identifies computational cost as a primary limitation: simulating each candidate model and fitting or evaluating surrogates is expensive (Wahl et al., 20 Feb 2026). The cloning branch with $D=\{(x_o^j,c_o^j)\}_{j=1}^M$1 is introduced partly to save compute by reusing previous simulations, and parallelization through systems such as Ray is cited as a mitigation for wall-clock cost (Wahl et al., 20 Feb 2026). This places ModelSMC in a regime similar to other nested or simulator-based SMC methods, where per-particle cost can dominate runtime.

A second limitation is surrogate bias. Because weighting relies on NLE or NLE-PFN approximations, models may be mis-weighted (Wahl et al., 20 Feb 2026). The paper proposes uncertainty-aware likelihoods or adaptive surrogate refinement as future work. This is a central issue because the particle system is only as reliable as the relative weights assigned to competing structural hypotheses.

A third limitation concerns the implicit prior. Since $D=\{(x_o^j,c_o^j)\}_{j=1}^M$2 is the LLM’s inductive bias, blind spots in the LLM may exclude certain hypotheses (Wahl et al., 20 Feb 2026). The paper notes that retrieval-augmented generation or domain-specific pretraining may broaden support. This is not a minor implementation detail: the support condition is one of the explicit assumptions of the probabilistic formulation, so prior blind spots directly affect posterior reachability.

Proposal quality is also singled out. Better prompt design or retrieval of similar scientific code may improve coverage and reduce variance (Wahl et al., 20 Feb 2026). In conventional SMC language, this can be read as an effort to improve proposal adaptation and reduce weight degeneracy. Unlike classical samplers, however, the proposal family here is expressed in natural language and code exemplars rather than in parametric kernels.

The extensions proposed in the paper are explicitly SMC-theoretic. These include annealed SMC with a temperature schedule $D=\{(x_o^j,c_o^j)\}_{j=1}^M$3 so that $D=\{(x_o^j,c_o^j)\}_{j=1}^M$4, particle rejuvenation through local edits or Metropolis–Hastings moves, clustering the final population to summarize posterior modes and provide model-space geometry, and multi-fidelity screening using coarser simulators before full evaluation (Wahl et al., 20 Feb 2026). Each of these extensions has a direct analogue in established SMC practice, whether in tempering strategies for posterior bridging (Zhou et al., 2013), PMCMC-style rejuvenation (Chopin et al., 2011), or multistage filtering and selection in expensive simulators.

This suggests that ModelSMC is less a single fixed algorithm than a template for embedding LLM-based scientific programming inside mature Monte Carlo machinery. Its distinctiveness lies in treating mechanistic code proposals as particles in an inferential system, thereby reframing iterative LLM-assisted model revision as approximate posterior sampling over simulator structure (Wahl et al., 20 Feb 2026).