PatchWorld: Gradient-Free Optimization of Executable World Models

Abstract: Text-agent environments are typically modeled as partially observable Markov decision processes (POMDPs), assuming that the simulator's latent state and transition dynamics are hidden from the agent. Yet little work has examined whether executable code can be induced to serve as a world model for prediction and planning under partial observability. We introduce PatchWorld, a gradient-free framework that turns offline trajectories into executable Python world models through counterexample-guided code repair. Instead of predicting the next observation with a black-box model, PatchWorld induces symbolic belief-state programs whose action updates can be inspected, replayed, and locally patched. Across seven AgentGym environments, PatchWorld-Simple achieves the highest code-based planning score among evaluated methods, reaching 76.4\% macro success in live one-step lookahead while invoking no LLM calls inside the world-model prediction module itself. We further find that a human-specified residual-memory bias improves surface observation fidelity but weakens decision utility. This exposes a tradeoff in executable world models, since improving observation fidelity can come at the expense of action-discriminative dynamics, and vice versa. Code is available at https://github.com/HKBU-KnowComp/PatchWorld.

• The paper introduces a novel gradient-free process that synthesizes symbolic, executable Python world models from offline trajectories in text-based environments.
• It employs counterexample-guided induction and modular program repairs to balance observation fidelity with planning utility, achieving high prediction accuracy and planning success.
• The approach outperforms LLM-based methods by reducing online inference costs and enhancing interpretability through standalone, debuggable code modules.

PatchWorld: Gradient-Free Optimization of Executable World Models

Introduction and Motivation

PatchWorld introduces a framework for constructing executable, interpretable world models for text-agent environments directly from offline trajectories, with no online gradient-based learning or dependence on neural inference during prediction or planning. The central technical objective is to induce symbolic, executable Python programs that approximate the underlying dynamics of partially observable Markov decision processes (POMDPs) in text environments. These programs capture the agent's belief state, transition logic, and rendering, supporting cycle-consistent prediction and tractable programmatic repair informed by counterexample-driven diagnostics.

PatchWorld is motivated by a gap in current practice: While recent world modeling work for text agents emphasizes neural or LLM-based next-observation predictors, these black-box models lack persistent, interpretable belief state and are not easily subject to targeted, offline correction. Symbolic code-based models are attractive due to interpretability, exact executability, and local debuggability, but partial observability and limited offline logs introduce underspecification—many candidate programs can replay the observed transitions, ranging from pure memorization to generalized latent-state abstractions.

Framework Overview

The PatchWorld pipeline consists of a generative optimization cycle, where an LLM first synthesizes an initial Python world model from contrastive offline trajectories. This executable model implements a modular interface, separating parsing, belief-state update, transition, and readout logic. Prediction is performed in cycles: belief is initialized and iteratively propagated forward, with observation prediction and correction at each step. Validation against held-out trajectories surfaces prediction failures, which are transformed into synthesized counterexamples. An LLM is prompted with these diagnostics and emits targeted program patches. Only patches that lexicographically improve formal replay error on the validation set are accepted, in a hill-climbing search loop analogous to counterexample-guided inductive synthesis (CEGIS).

Figure 1: PatchWorld's architecture for generative world-model induction, code repair, and evaluation, emphasizing offline, gradient-free program search.

This approach ensures that world-model improvement is grounded in discrete search over the program space, guided by executable validation, rather than unconstrained text-based self-correction or gradient descent.

Tradeoff Between Fidelity and Utility

A crucial empirical and conceptual insight is the explicit tradeoff between observation fidelity (matching surface text) and planning utility (action-contrastive signal). Under partial observability, different candidate models can attain identical replay (next-observation) accuracy but diverge substantially in their action-discriminative consequences. For example, "symbolic" program variants may preserve critical latent structure for planning without producing canonical simulator text, while "residual-memory" variants that bias toward high-confidence surface form may maximize textual fidelity but attenuate action-relevant differences.

This Pareto frontier is empirically validated: PatchWorld's residual-memory variant provides the highest code-based next-observation accuracy, while the purely symbolic variant yields the highest downstream planning success rates under shared one-step lookahead evaluation.

Figure 3: Empirical Pareto frontier between next-observation Token F1 (fidelity) and live lookahead planning success (utility) across world model variants.

Methodological Contributions

The key methodological innovations in PatchWorld include:

• Counterexample-Guided Induction and Repair: Model synthesis and repair are governed by replay on contrastively mined trajectory subsets. Diagnostics summarize failure patterns for LLM-guided patching, with a rigorous acceptance gate that strictly improves validation performance.
• Contrastive Mining for Prompt Evidence: Transition selection explicitly seeks both success-like and failure-like examples per action-outcome type, maximizing the conditional rule discovery and preventing the LLM from overfitting to uninformative memorization.
• Residual Memory Bias: The residual-memory extension captures high-confidence, recurring textual details as a constrained cache, augmenting the symbolic belief state when surface precision is valuable without degrading the integrity of programmatic state propagation.
• Component Modularity and Validation: All induced programs are standalone Python modules implementing a well-defined interface. Planning and evaluation use these programs directly, with no LLM calls at inference—a substantial reduction in planning-time token usage compared to neural baselines.

Empirical Evaluation

PatchWorld is extensively evaluated across seven AgentGym environments, spanning navigation, manipulation, crafting, shopping, and language-based puzzles. The framework is compared to multiple baselines, including fine-tuned LLM observation predictors, in-context learning, and prior code-induction techniques (WorldCoder, PoE-World).

Observation Prediction: The residual-memory variant of PatchWorld achieves the strongest macro Token F1 and BLEU-4 scores among all code-based methods (up to 0.70 Token F1, exceeding baselines by over 0.06 points), with variance smaller than the observed effect size.

Rollout Consistency: PatchWorld reduces prediction drift under multi-step rollout by leveraging persistent belief-state correction; symbolic models constrain error accumulation better than purely autoregressive text predictors.

Planning Utility: In live one-step lookahead reranking (identical planning stack except for the lookahead predictor), PatchWorld's symbolic variant achieves the highest code-based macro episode success rate (76.4%), matched only by LLM-Direct. Notably, this is accomplished with zero LLM inference calls for lookahead prediction versus over 60,000 tokens/task for LLM-Direct.

Strong Claims:

• PatchWorld's symbolic world models match or outperform neural counterparts in planning utility while being fully executable and interpretable.
• The empirical frontier demonstrates that maximizing observational fidelity can degrade planning utility—an explicit dissociation between metrics.

Practical and Theoretical Implications

Practically, PatchWorld enables the deployment of lightweight, testable, and locally patchable world models in language-agent environments without dependence on large-scale online inference or retraining. Pertinent scenarios include offline reinforcement learning from logged data and simulation-based agent debugging.

Theoretically, the framework supports the emerging view that optimal decision support in partially observable environments does not coincide with maximizing likelihood of observed future text. Instead, structural alignment between model updates and latent-domain consequences is crucial—a direct instantiation of value equivalence. The repair loop also illustrates that LLMs can be used as generalized program optimizers rather than end-to-end predictors, opening a pathway to neuro-symbolic systems where executability and interpretability are first-class requirements.

Future Directions

Potential extensions include:

• Generalization to broader classes of environments beyond text-based tasks, such as multimodal or embodied settings.
• Automated induction of modular world models for new tasks, facilitating transfer and scalability.
• Integration with agent-specific learning algorithms for joint optimization of world model and agent policy.
• Development of richer diagnostic criteria for world-model validation, including safety and invariance properties.

Conclusion

PatchWorld establishes a robust, gradient-free framework for inducing executable world models from offline data, leveraging LLMs for code synthesis and repair. It exposes and characterizes a fundamental tradeoff between surface fidelity and decision-oriented utility, providing a foundation for more interpretable, efficient, and verifiable agent modeling in partially observable domains. The open-source release of code and induced models further enables reproducibility and extension in future research.