Generalization to Real-World Planning Scenarios

Establish whether the documentation-retrieval-integrated PDDL generation pipelines—including Modular w/ Specific Doc, Once w/ Whole Doc, and Refinement w/ Code-Retrieved Doc—generalize beyond the evaluated benchmarks (Blocks World, Logistics, Barman, and Mystery Blocks World) to more diverse or real-world planning scenarios.

Background

The paper proposes lightweight pipelines that integrate documentation retrieval with modular PDDL code generation and iterative error refinement to improve planning language generation by open-source LLMs. Experiments across Blocks World, Logistics, Barman, and Mystery Blocks World show substantial gains in syntactic accuracy and more limited improvements in semantic accuracy, highlighting current models’ reasoning constraints.

A stated limitation is that the evaluation is confined to a small set of benchmark domains. The authors explicitly note that whether these methods and improvements generalize to more diverse or real-world planning scenarios remains unverified, leaving this as an open question for future investigation.

References

Lastly, our evaluation is confined to a few benchmark domains; generalization to more diverse or real-world planning scenarios remains to be verified.

— Documentation Retrieval Improves Planning Language Generation  (2509.19931 - Wang et al., 24 Sep 2025) in Section 6 (Limitations)

Finally, because the evaluated IPC-derived benchmark is public, possible training-data recall limits the generality of our results; evaluation on novel, unpublished domains is needed to establish whether these gains transfer to genuinely unseen repair problems.

— LLM-Only PDDL Domain Repair with Open-Weight Models  (2608.17341 - Bavandpour et al., 18 Aug 2026) in Conclusion/Future Work section