World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning (2503.10480v1)

Abstract: Recent advances in large vision-LLMs (LVLMs) have shown promise for embodied task planning, yet they struggle with fundamental challenges like dependency constraints and efficiency. Existing approaches either solely optimize action selection or leverage world models during inference, overlooking the benefits of learning to model the world as a way to enhance planning capabilities. We propose Dual Preference Optimization (D$^2$PO), a new learning framework that jointly optimizes state prediction and action selection through preference learning, enabling LVLMs to understand environment dynamics for better planning. To automatically collect trajectories and stepwise preference data without human annotation, we introduce a tree search mechanism for extensive exploration via trial-and-error. Extensive experiments on VoTa-Bench demonstrate that our D$^2$PO-based method significantly outperforms existing methods and GPT-4o when applied to Qwen2-VL (7B), LLaVA-1.6 (7B), and LLaMA-3.2 (11B), achieving superior task success rates with more efficient execution paths.

• The paper introduces Dual Preference Optimization (D²PO), a method improving embodied AI planning by jointly optimizing action selection and predicting future world states for better understanding of action consequences.
• The method utilizes a tree search mechanism for data collection, exploring action paths via trial and error to gather detailed information and learn successful strategies.
• Experiments show D²PO improves task success rates and planning efficiency on benchmarks like VoTa-Bench, though the sim-to-real gap and computational cost are noted limitations.

The paper "World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning" (2503.10480) addresses how to improve the ability of AI systems to plan and execute real-world tasks. The key idea is to enable an AI model not only to choose what action to take next but also to predict how the environment will change as a result of that action. This combined approach is called Dual Preference Optimization (D²PO).

Key Contributions:

• Joint Optimization: Instead of focusing only on selecting the best actions, D²PO trains the model to both choose actions and predict future states of the world. This dual focus allows the model to better understand the consequences of its actions.
• Tree Search Mechanism: The paper introduces a tree search method that explores possible action paths and gathers detailed data about what works and what does not. This exploration, done through trial and error, helps the AI learn from its own experiences without needing constant human input.
• Strong Experimental Results: Using benchmark tests, the method showed significant improvements in success rates and planning efficiency compared to other approaches. For example, a smaller model managed to outperform a larger, established model by effectively integrating process guidance with environmental feedback.

Method Details:

1. Task Formulation: The embodied task planning problem is modeled like a partially observable scenario in which the AI does not have full information about the environment. This requires the model to make the best possible decisions with limited data.
2. Tree Search Data Collection:
   • The method samples several possible actions and uses a scoring system to decide which actions are promising.
   • When a successful pathway is found, the sequence of actions is recorded, and preference pairs are created to help train the model to repeat successful strategies.
3. Dual Preference Optimization:
   • Two objectives are optimized simultaneously: one for correctly selecting actions that lead to success and another for accurately predicting the resulting state of the environment.
   • This dual training helps the AI model learn the dynamics of its surroundings, making it better at planning efficient paths.

Results and Limitations:

• Improvements in Success and Efficiency: Experiments on benchmark tests (like VoTa-Bench) showed clear improvements in task success rates and planning efficiency, demonstrating that the model not only makes better decisions but also plans more effectively.
• Sim-to-Real Gap: A noted limitation is that most of the training and evaluation happens in simulated environments, which might not capture all the unpredictability of the real world.
• Computational Expense: The method relies on additional computational resources for evaluating actions, which can be demanding.

In summary, the paper introduces an approach that helps AI systems plan tasks more effectively by teaching them both to choose the right actions and predict how those actions will affect their environment. This dual approach leads to more efficient and successful planning, though challenges remain in applying these methods in real-world settings.