LLM-Only PDDL Domain Repair with Open-Weight Models

Abstract: AI planning is concerned with finding a sequence of actions that achieves a specified goal. It relies on explicit models of the world, commonly represented in the Planning Domain Definition Language (PDDL). An active line of research investigates how errors in such models can be detected and repaired. For example, users may provide positive test plans that are solutions, and negative test plans that fail during execution. Automated repair methods then modify the PDDL model to satisfy these constraints. In this paper, we evaluate the ability of recent open-weight LLMs to perform this repair task using an LLM-only approach. Our experiments show that the symbolic baseline achieves an $F_1$ score of $.49$, while the best-performing LLM reaches $.87$ with high reasoning effort, an absolute improvement of $.38$. However, that setting has a mean test pass rate of only $.82$, falling to $.06$ on the Thoughtful domain; even the best setting that includes the test traces reaches only $.92$. Thus, current open-weight models cannot guarantee satisfaction of the test constraints required for reliable automated model repair.

• The paper evaluates whether recent open-weight large language models can solve the PDDL domain repair problem without any intricate symbolism, achieving significantly higher F1 scores, about 38% better, with top open-weight models such as GLM 5.2 and Qwen 3 achieving F1 scores of 85 and increasingly higher test pass rates, up to 82% and beyond.
• The better open-weight models manage to grasp semantic cues and therefore understand the core of the problem - justifying collapses and giving plausible repairs ontologically affiliated to the known ground truths - while causing detrimental effects by outputting context limit errors and unparsable outputs on the Tochautful setting, achieving a lower test pass rate of 6% in all configurations which suggests shortcomings in the LLM-Optimized repair.
• Despite overwhelming findings, the methods demonstrate the insufficiency behind the technicalities of the accomplishments, emphasizing the need for continuous improvement with hybrid solutions between LLMs and more advanced symbolic reasoners.

Problem setting and motivation

This paper evaluates whether recent open-weight LLMs can solve the PDDL domain repair problem without any symbolic post-processing. The repair problem, following Lin et al., takes a flawed planning domain together with positive (whitelist) traces that must remain valid plans and negative (blacklist) traces whose action at a designated index must become inapplicable; a solution is a set of atomic repairs—adding or removing positive/negative preconditions or effects of action schemas—that satisfies all tests. The authors frame each repair as a contrastive explanation in Miller's sense: it answers why a trace fails and how the domain could have behaved differently.

The symbolic baseline is a sound conditional hitting-set algorithm that finds cardinality-minimal repair sets by iteratively executing all test plans and encoding failure diagnoses as hitting-set constraints. Its weakness is semantic blindness: it optimizes only cardinality, ignores predicate and action names, and may miss ground-truth repairs that are not minimal. The LLM-only approach is intended to exploit exactly those semantic cues.

Experimental design

The evaluation uses the error-injected IPC benchmark from prior work, with 12 domains (LOGISTICS98, WOODWORKING08, and MPRIME are excluded—the latter because its deliberately misleading names would confound a semantics-based method). Metrics are precision, recall, $F_1$ against known ground-truth repairs, plus the test pass rate (TR), the fraction of tests actually satisfied. TR is decisive: since a solution must satisfy every test, high $F_1$ with imperfect TR does not constitute a correct solver output.

Two prompt variants are compared. NoTrace supplies only the corrupted domain and asks for semantically plausible single-edit repairs inferred from names; it addresses only a relaxed variant of the problem since no test constraints are visible. WithTrace additionally provides all whitelist and blacklist traces and requires their satisfaction. Fourteen open-weight models spanning roughly 14B to 1.6T parameters (e.g., Phi-4, Qwen3, GPT-OSS, Mistral variants, DeepSeek V4 Flash/Pro, GLM 5.2, Nemotron 3) are run under default and high reasoning effort via OpenRouter, with up to three retries on failed calls; unparsable outputs count as empty predictions. The stochastic symbolic baseline is represented by the published five-run average rather than rerun.

Results

The headline finding is a large gap in repair quality but an equally large gap in correctness:

Setting	Best model	Mean $F_1$	Mean TR
Symbolic baseline	—	.49	1 (by construction)
NoTrace, default effort	GLM 5.2	.85	.85
NoTrace, high effort	GLM 5.2	.87	.82
WithTrace, default effort	GLM 5.2	.78	.82
WithTrace, high effort	DeepSeek V4 Pro / GLM 5.2	.82	.92

The best LLM result ($F_1 = .87$) improves on the symbolic baseline's $.49$ by an absolute $.38$, indicating that modern open-weight models recover ground-truth repairs far more faithfully than cardinality-minimal symbolic search. The data also show a sharp generational effect: the best 2025 model reaches only $F_1 = .53$ while the best 2026 model reaches $.85$. Reasoning effort helps most models but unevenly—GLM 5.2 gains modestly (.85 to .87), whereas Mistral Small 4 nearly triples its $F_1$ (.13 to .41).

Three results qualify the optimistic reading. First, supplying test traces does not improve the best $F_1$: GLM 5.2 drops from .85 to .78 at default effort, and the best WithTrace result at high effort (.82) remains below the best NoTrace result (.85). The authors attribute this to the combinatorial burden of jointly reasoning over many long plans and to context-window overflow, noting that WithTrace runs exhibit context-limit errors and unparsable outputs absent from NoTrace runs. Second, and most consequentially, no LLM configuration achieves complete test satisfaction: the highest mean TR is .92 (DeepSeek V4 Pro, WithTrace, high effort), and every configuration collapses to TR = .06 on Thoughtful. High overlap with the ground-truth repair therefore does not imply a valid solution to the repair problem. Third, the one case where traces demonstrably help is DeepSeek V4 Pro, where higher reasoning effort raises both TR (.82 to .92) and $F_1$0 (.74 to .82)—suggesting that trace-aware reasoning is feasible but not yet reliable.

Limitations

The authors identify two substantive caveats. The benchmark derives from public IPC domains and is available online, so models may have memorized the domains or their fixes during training; this possible contamination means the reported $F_1$1 and TR gains may not generalize to genuinely unseen repair problems, and validation on novel unpublished domains is required. Methodologically, each domain receives a single experimental run per model, so per-domain scores carry unquantified sampling variance beyond the cross-domain standard deviations, and missing values (context-limit errors, API failures, unparsable outputs) are scored as zero, which penalizes weaker models in the WithTrace condition in ways that conflate capability with infrastructure robustness. The paper also leaves open which component of the shortened NoTrace prompt (length, absence of explicit reasoning requests, removal of the one-shot example) accounts for the drop in GPT-4o's $F_1$2 from .46 to .38 relative to earlier work.

Conclusion

The paper establishes two claims. Recent open-weight LLMs exploit semantic cues to produce repairs substantially closer to ground truth than a sound symbolic optimizer ($F_1$3 .87 vs. .49), with clear generational improvement across model releases. However, LLM-only repair cannot guarantee correctness: even the best configurations leave a nontrivial fraction of tests unsatisfied (mean TR at most .92, and .06 on Thoughtful throughout), so they do not solve the repair problem as formally defined. The authors argue for hybrid architectures in which an LLM ranks or filters semantically plausible candidate repairs while a symbolic reasoner enforces test-satisfaction guarantees—an integration proposed but not yet realized—and call for evaluation on uncontaminated domains to confirm that the observed gains transfer.