Can preference-based feedback recover correct plans from models that initially cannot plan?

Ascertain whether, and under what conditions, preference-based feedback (such as reinforcement learning from human feedback) can enable a model that initially assigns near-zero probability to correct plans to discover and adopt those plans, despite the limitations of teacher-forcing-based gradients.

Background

The paper discusses reinforcement learning and preference-based methods as potential alternatives to plain teacher-forcing. It notes that, in practice, gradients in such methods still reduce to teacher-forcing on model-generated answers, raising doubts about their ability to overcome planning failures when the model initially lacks the correct plan.

The authors explicitly question whether preference-based feedback alone suffices to move a model from a state of near-zero probability on the correct plan to discovering that plan, especially in exponentially large solution spaces typical of planning problems.