PatchWorld: Executable World Modeling

• PatchWorld is a gradient-free framework for inducing executable world models from offline text-agent trajectories in partially observable environments.
• It leverages symbolic belief-state programs with inspectable update functions to enable replay fidelity and multi-step planning without LLM inference at runtime.
• The framework employs a counterexample-guided repair process to iteratively refine Python models, balancing observation fidelity with planning utility.

PatchWorld is a gradient-free framework for inducing executable Python world models from offline trajectories in text-agent environments modeled as partially observable Markov decision processes (POMDPs). It replaces black-box next-observation prediction with symbolic belief-state programs whose action updates can be inspected, replayed, and locally patched. The framework is introduced in “PatchWorld: Gradient-Free Optimization of Executable World Models” (Bai et al., 29 May 2026), which studies whether executable code can be induced to serve as a world model for prediction and planning under partial observability.

1. Problem formulation and representational target

PatchWorld models each text-agent environment as a POMDP $(\mathcal{S},\mathcal{A},\mathcal{T},\Omega)$ in which the latent state is $s_t \in \mathcal{S}$, the agent emits a text action $a_t \in \mathcal{A}$, the environment transitions according to $\mathcal{T}(s_{t+1}\mid s_t,a_t)$, and emits a text observation $o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$ (Bai et al., 29 May 2026). The learner observes only offline trajectories

$\mathcal{H}=\{(o_0,a_0,o_1,a_1,\dots,o_T)\},$

and never observes $s_t$ or $\mathcal{T}$.

The target of induction is a code-based world model $c$, implemented as an executable Python module that maintains a symbolic belief state $\hat s_t$ and supports both next-observation prediction for replay fidelity and multi-step simulation for planning. Each induced world model implements four functions over beliefs and text: init_belief, predict_belief, readout_observation, and correct_belief. In the formulation given in the paper,

$s_t \in \mathcal{S}$0

The induction objective is to minimize replay loss over the log:

$s_t \in \mathcal{S}$1

where $s_t \in \mathcal{S}$2 is, for example, normalized edit distance or token-level BLEU/F1. Because $s_t \in \mathcal{S}$3 is a Python program and therefore discrete, PatchWorld performs gradient-free search over programs rather than gradient descent (Bai et al., 29 May 2026).

This formulation makes the belief state central. The code is intended not merely to render text outputs but to encode latent variables such as agent position in Maze, container contents in AlfWorld, or inventory and recipes in TextCraft. A plausible implication is that PatchWorld treats executable state update logic as the primary modeling object, with surface rendering delegated to a separate readout step.

2. Induction pipeline and counterexample-guided repair

PatchWorld proceeds in three phases: evidence selection via contrastive mining, initial synthesis of a Python skeleton with an LLM, and counterexample-guided repair in CEGIS style (Bai et al., 29 May 2026).

In evidence selection, transitions $s_t \in \mathcal{S}$4 from $s_t \in \mathcal{S}$5 are grouped by an action signature $s_t \in \mathcal{S}$6 and an outcome signature $s_t \in \mathcal{S}$7, such as success, failure, or informational. The method samples up to $s_t \in \mathcal{S}$8 examples per bucket, round-robin up to $s_t \in \mathcal{S}$9 total, so that both success and failure cases are shown. This contrastive set $a_t \in \mathcal{A}$0 is used in the synthesis prompt, while all transitions remain available for validation.

Initial synthesis provides the LLM with the environment description $a_t \in \mathcal{A}$1, the BaseWorldModel interface, and the contrastive examples $a_t \in \mathcal{A}$2. The LLM, exemplified by Qwen3-Coder-480B, emits a standalone Python class of the form class MyWorldModel(BaseWorldModel): ... (Bai et al., 29 May 2026).

After synthesis, PatchWorld validates the candidate program on the replay set $a_t \in \mathcal{A}$3. For each $a_t \in \mathcal{A}$4, it executes the four-step loop and compares the predicted observation $a_t \in \mathcal{A}$5 against the true $a_t \in \mathcal{A}$6, collecting failures into a set of counterexamples $a_t \in \mathcal{A}$7. Recurring patterns are then diagnosed, and the top-$a_t \in \mathcal{A}$8 failures are prioritized by severity and frequency. A repair prompt is formed from the current code, a natural-language summary $a_t \in \mathcal{A}$9, the top-16 counterexamples, and interface guidance.

The repair stage asks the LLM to emit $\mathcal{T}(s_{t+1}\mid s_t,a_t)$0 candidate patches $\mathcal{T}(s_{t+1}\mid s_t,a_t)$1. Each candidate is re-validated using

$\mathcal{T}(s_{t+1}\mid s_t,a_t)$2

where $\mathcal{T}(s_{t+1}\mid s_t,a_t)$3 is, for example, normalized edit distance plus any parsed-state checks. PatchWorld also defines a lexicographic score $\mathcal{T}(s_{t+1}\mid s_t,a_t)$4 that orders models first by aggregate failure severity, then by number of remaining counterexamples, then by $\mathcal{T}(s_{t+1}\mid s_t,a_t)$5. The framework accepts a patch only if its score is strictly better than that of the current program.

Because every patch is verified on the full replay set, the paper states that regressions are prevented (Bai et al., 29 May 2026). This is a distinctive aspect of the method: LLMs act as proposal engines, while acceptance is controlled by executable validation rather than by trust in the proposal itself.

3. Executable world-model interface and inference behavior

The induced world models are subclasses of BaseWorldModel and implement parse_observation(self, obs:str)\to dict of facts, init_belief(self, obs_0)\to State, correct_belief(self, state, obs)\to State, predict_belief(self, state, action)\to State, readout_observation(self, state, action)\to str, and extract_valid_action_forms(self)\to List[str] (Bai et al., 29 May 2026).

The belief state is a pure Python data structure, such as a dictionary or custom class, encoding latent variables. At inference time, no LLM calls occur. The paper gives a typical one-step call flow:

1. belief ← correct_belief(belief, last_obs)
2. belief' ← predict_belief(belief, candidate_action)
3. pred_obs ← readout_observation(belief', candidate_action)

This architecture separates belief correction, latent transition, and textual readout. That separation matters for the paper’s empirical findings. PatchWorld-Simple may miss some template details in the rendered observation while still learning correct symbolic updates; conversely, better surface rendering does not necessarily imply stronger planning behavior (Bai et al., 29 May 2026).

The emphasis on executable modules also makes the induced models inspectable, replayable, and locally patchable. This suggests a different failure mode taxonomy than in black-box neural world models: errors can be localized to belief parsing, transition logic, or readout templates rather than treated as undifferentiated prediction error.

4. Residual-memory bias and the fidelity–utility tradeoff

PatchWorld-Residual augments the symbolic readout with a train-only key-value cache $\mathcal{T}(s_{t+1}\mid s_t,a_t)$6 designed to capture recurring textual details that are easy to memorize but hard to form symbolically (Bai et al., 29 May 2026). Each train transition $\mathcal{T}(s_{t+1}\mid s_t,a_t)$7 is normalized via a signature function $\mathcal{T}(s_{t+1}\mid s_t,a_t)$8 that lower-cases, strips whitespace, and abstracts instance names to type placeholders. If a signature has a unique majority next observation with confidence at least $\mathcal{T}(s_{t+1}\mid s_t,a_t)$9—typically $o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$0—the cache stores

$o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$1

At readout time,

$o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$2

The paper states that, because retrieval is pure string lookup, this variant can reach near-perfect template fidelity where cache coverage is high, with AlfWorld, TextCraft, and Wordle given as examples (Bai et al., 29 May 2026). At the same time, the symbolic belief update $o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$3 and correction $o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$4 are still executed, so planning utility remains grounded in the same latent state.

A central finding is that the residual-memory bias improves surface observation fidelity but weakens decision utility. The paper characterizes this as a tradeoff between observation fidelity and action-discriminative dynamics. In its broader discussion, it links the result to value-equivalence: small readout differences may not support action discrimination, and conversely a model can preserve planning signals even when textual templates are approximate (Bai et al., 29 May 2026).

This is also the main point at which PatchWorld departs from a common assumption that better one-step textual prediction should monotonically improve downstream planning. The reported results do not support that assumption.

5. Experimental setting and quantitative results

PatchWorld is evaluated on seven AgentGym environments spanning four regimes: deterministic structure (Maze, BabyAI, TextCraft), irreducibly stochastic (Wordle, WebShop), and deterministic dynamics with complex rendering (AlfWorld, SciWorld) (Bai et al., 29 May 2026). Data are collected by a Qwen3-Coder-480B ReAct agent and split 60/20/20 by instance.

Observation fidelity

On held-out test transitions, the paper measures Token F1 and BLEU-4 between the true $o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$5 and the model’s $o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$6. No LLM calls occur at inference for any program-based method. The macro averages over seven environments are reported as follows (Bai et al., 29 May 2026):

Method	Token F1	BLEU-4
Word2World (neural SFT)	0.85	0.67
LLM-Direct (ICL next-obs)	0.64	0.41
WorldCoder / PoE-World	0.63	0.48
PatchWorld-Simple	0.57	0.40
PatchWorld-Residual	0.69	0.50

The paper interprets PatchWorld-Simple as already learning correct symbolic updates while missing some template details, and PatchWorld-Residual as closing that gap to become the best code-based predictor (Bai et al., 29 May 2026).

Rollout prediction

For rollout evaluation, each model is unrolled for $o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$7 steps, feeding the model’s own $o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$8 back into correct_belief at each step, and Token F1 is reported at $o_{t+1}\sim\Omega(\cdot\mid s_{t+1})$9. The paper states that the belief state prevents drift and that PatchWorld-Simple outperforms WorldCoder/PoE-World at all horizons (Bai et al., 29 May 2026).

Live one-step lookahead planning

The planning setup integrates each world model into a common one-step lookahead planner with the following steps: candidate actions from ReAct default plus up to 4 extra API candidates; next-observation prediction for each candidate via the world model; re-ranking by a shared LLM selector, specifically Qwen3-Coder-480B; fallback to default if no candidate beats margin; and a 30-step cap with up to 200 test instances per environment (Bai et al., 29 May 2026).

The reported macro episode success rates are:

Method	Macro Succ. (%)	Uses LLM at Inference?
ReAct (no lookahead)	74.4	Yes (for action gen)
LLM-Direct lookahead	75.8	Yes (~63k tokens/task)
Word2World lookahead	63.5	Yes
WorldCoder lookahead	64.4	No
PoE-World lookahead	69.3	No
PatchWorld-Simple	76.4	No (0 tokens/task)
PatchWorld-Residual	72.9	No

PatchWorld-Simple therefore achieves the highest code-based planning score among the evaluated methods, reaching 76.4% macro success in live one-step lookahead while invoking no LLM calls inside the world-model prediction module itself (Bai et al., 29 May 2026).

6. Interpretation, significance, and relation to neighboring approaches

The paper’s most direct conceptual contribution is the demonstration that offline text logs can be converted into executable world models that support both replay fidelity and planning under partial observability (Bai et al., 29 May 2026). Rather than training a black-box predictor, PatchWorld induces symbolic belief-state programs and improves them through discrete, counterexample-guided repair.

The comparison with Word2World, WorldCoder, PoE-World, and LLM-Direct is framed empirically rather than taxonomically in the provided material. The paper’s Pareto analysis plots one-step Token F1 against lookahead success and reports that Word2World has very high fidelity (0.85) but low utility (63.5%), PatchWorld-Residual has the highest code-fidelity (0.69) but weaker utility (72.9%), and PatchWorld-Simple has the best code-utility (76.4%) but lower fidelity (0.57), with WorldCoder and PoE-World lying strictly below this frontier (Bai et al., 29 May 2026).

This suggests that executable world modeling should not be evaluated solely by reconstruction metrics. Under partial observability, action-discriminative dynamics may be more consequential for planning than exact token-level reproduction of observations. The residual cache result sharpens this point: memorized surface forms can improve BLEU-4 and Token F1 without improving, and in this case while weakening, one-step lookahead decisions.

The method also bears a methodological resemblance to classical CEGIS, but with LLMs as the synthesizer and executable validation as the acceptance criterion. That characterization appears explicitly in the paper’s discussion of “Gradient-Free Symbolic Optimization” (Bai et al., 29 May 2026). A plausible implication is that PatchWorld occupies a hybrid position between program synthesis and model-based reinforcement learning: it uses LLMs to propose symbolic programs, but relies on replay-based verification to control program search.

7. Limitations and proposed extensions

The paper leaves richer objective design to future work. It explicitly sketches a hybrid loss

$\mathcal{H}=\{(o_0,a_0,o_1,a_1,\dots,o_T)\},$0

where ObsError is replay loss and DecError measures action-contrastive misrankings, but states that PatchWorld itself uses replay loss (Bai et al., 29 May 2026). This indicates that the reported framework does not directly optimize decision utility, even though utility is a central evaluation target.

The listed extensions are richer hybrid objectives combining replay and value-based losses, template parametrization so that rendering is learned while rules remain symbolic, transfer across environments by reusing dominant code fragments, and automatic invariants or lightweight formal specifications to guide patch proposals (Bai et al., 29 May 2026). These are presented as future directions rather than implemented components.

A common misconception would be to read PatchWorld primarily as a high-fidelity observation generator. The evidence presented does not support that reduction. PatchWorld-Simple is not the strongest one-step predictor by Token F1 or BLEU-4, yet it attains the strongest code-based planning performance. Conversely, PatchWorld-Residual improves fidelity while reducing planning success. The framework is therefore best understood as an approach to executable world modeling in which symbolic belief updates, counterexample-guided repair, and planning utility are at least as important as textual surface accuracy (Bai et al., 29 May 2026).