Generalization to Real-World Planning Scenarios

Establish whether the documentation-retrieval-integrated PDDL generation pipelines—including Modular w/ Specific Doc, Once w/ Whole Doc, and Refinement w/ Code-Retrieved Doc—generalize beyond the evaluated benchmarks (Blocks World, Logistics, Barman, and Mystery Blocks World) to more diverse or real-world planning scenarios.

Background

The paper proposes lightweight pipelines that integrate documentation retrieval with modular PDDL code generation and iterative error refinement to improve planning language generation by open-source LLMs. Experiments across Blocks World, Logistics, Barman, and Mystery Blocks World show substantial gains in syntactic accuracy and more limited improvements in semantic accuracy, highlighting current models’ reasoning constraints.

A stated limitation is that the evaluation is confined to a small set of benchmark domains. The authors explicitly note that whether these methods and improvements generalize to more diverse or real-world planning scenarios remains unverified, leaving this as an open question for future investigation.