APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts

Abstract: Long-horizon robot planning requires jointly reasoning over semantic task structure and geometric feasibility. To successfully execute a task, a robot must decompose goals, select task-relevant objects, and sequence actions, while ensuring that plans satisfy spatial constraints such as limited free space and object collisions. In this work, we propose APIVOT, a VLM-based planner that adaptively interleaves language and visual thoughts for long-horizon planning. APIVOT learns to leverage language for semantic reasoning, while using visual thoughts as imagined future states for internal verification of geometric feasibility. On long-horizon kitchen tasks, APIVOT outperforms general-purpose VLMs and prior planning frameworks, achieving the largest gains in spatially constrained settings. We find that APIVOT learns meaningful modality selection behavior, demonstrating that adaptive interleaving of vision-language thoughts improves both planning success and reasoning efficiency.

• The paper introduces APIVOT, which interleaves visual thoughts with language planning to jointly address semantic task decomposition and geometric feasibility.
• The model is trained through a three-stage curriculum that enhances visual thought generation and adaptive modality selection, leading to significant performance gains.
• The experimental results show that APIVOT outperforms baselines with an 8–9 percentage point increase in task success and a 39% token reduction for efficiency.

APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts

Motivation and Problem Setting

Long-horizon robot planning demands the joint resolution of symbolic task structure and geometric feasibility constraints. Tasks such as “store the leftovers in the fridge” require a planner to decompose goals, sequence actions, select relevant objects, and satisfy physical constraints like spatial limitations and collisions. Existing LLM- and VLM-based task planners excel at semantic decomposition but struggle to represent and reason about geometry, often producing plans that are plausible in language but infeasible in the physical world (Huang et al., 2022, Ahn et al., 2022). Methods that couple these models with external geometric reasoners introduce geometric awareness only at plan correction—geometry remains detached from the forward reasoning process.

APIVOT addresses this fundamental limitation by integrating vision and language modalities within its planning trace, enabling explicit internal reasoning about both high-level semantics and geometric constraints. Unlike prior vision-language-action (VLA) models, which may uniformly invoke visual reasoning regardless of task phase, APIVOT adaptively selects between modality streams in a step-wise manner, leveraging vision for spatially critical stages and language for abstract task decomposition.

Figure 1: APIVOT’s interleaved language and visual reasoning enables it to anticipate and verify geometric constraints, such as collisions, during the planning process, unlike standard VLM-only planners.

APIVOT Architecture and Training Pipeline

APIVOT is initialized from Qwen3-VL-8B-Instruct, a vision-language large model, and finetuned using LoRA on both attention and MLP projection layers. Planning is realized as an autoregressive multimodal trace, where “visual thoughts”—latent-state renderings represented as token spans—can be injected at any reasoning step. Each visual thought induces a latent in the model’s embedding space, supervised to align with the actual future state of the environment given proposed actions.

Training proceeds via a three-stage supervised curriculum:

1. Visual Thought Comprehension: The model is taught to parse and reason given fixed, externally provided visual thoughts in planning traces.
2. Visual Thought Generation: APIVOT learns to autoregressively generate visual thoughts, conditioned on language, for geometry-sensitive transitions.
3. Adaptive Modality Selection: Traces are heuristically annotated with constraint types; the model is trained to select the optimal modality for each step, creating efficiency by avoiding redundant visual token generation for semantically simple subtasks.

Figure 2: APIVOT adaptively invokes vision during geometry-sensitive phases (collision checks, spatial layouts), while relying on language for symbolic choices (object assignment, subgoal sequencing).

The final model produces planning traces consisting of language segments for task decomposition and visual thought tokens for steps requiring geometric verification.

Experimental Evaluation

APIVOT is benchmarked on KitchenWorlds, a simulation suite built for ecological long-horizon tasks. Three families are considered: Containment, Sorting, and Storing Leftovers, each escalating in spatial and semantic complexity.

• Containment: Place a set of objects within a confined region (e.g., fridge shelf), addressing accessibility, obstruction, and free-space placement.
• Sorting: Assign objects to containers subject to type and geometric constraints, where assignments can be infeasible due to container capacity.
• Storing Leftovers: Compose sorting followed by containment, requiring compositional generalization.

Figure 3: KitchenWorlds task suite: Containment (top), Sorting (middle), and Storing Leftovers (bottom) demonstrate increasing demands for both semantic and geometric reasoning.

APIVOT is compared to state-of-the-art general VLMs (Gemini-ER-1.5/3.1-Pro, Qwen3-VL-8B), planning frameworks (VLM-TAMP (Yang et al., 2024), Reflect-VLM (Feng et al., 23 Feb 2025)), and the symbolic planner FastDownward (Helmert, 2011). Each baseline is provided with complete state/mapping information for fairness.

• Overall Success: APIVOT achieves an average success rate of 0.419, outperforming the best VLM baseline, Gemini-ER-1.5, by 8.1 percentage points and the top planning baseline (VLM-TAMP) by 9.0 points.

Figure 4: Task success rates across all task families show substantial gains for APIVOT over baselines.

• Geometric Complexity: The performance gap in favor of APIVOT widens considerably as spatial constraints intensify (occupancy ratio increases), demonstrating robustness under geometric adversities encountered during planning.
• Reasoning Efficiency: APIVOT produces successful plans with a substantially lower token budget. Adaptive modality selection enables the model to retain $91\%$ of the “always-image” upper bound success with a $39\%$ reduction in token usage.
• Ablation Studies: Removing the visual-thought component or disabling adaptive modality selection significantly degrades performance, clarifying that APIVOT’s gains derive from the internal use of visual thoughts and learned adaptivity, not just data domain supervision.

Figure 5: Example execution traces for Containment and Sorting—APIVOT proactively reorders and rearranges objects to assure geometric feasibility, unlike Gemini-ER-1.5.

Figure 6: APIVOT preserves space for future bowl placement in Containment, anticipating constraints that VLM-TAMP does not.

Figure 7: APIVOT’s correct braiser assignment in Sorting; Gemini-ER-1.5 fails on account of capacity misjudgment.

Adaptive Modality Use and Interpretability

APIVOT’s reasoning traces display constraint-aware behavior: visual thoughts are reserved for steps where spatial reasoning is crucial (e.g., anticipating downstream collisions), but suppressed for purely symbolic subtasks or unconstrained placements.

• Visual-thought invocation rate is highest on steps annotated as constrained or feasibility-checking.
• Quantitative tradeoff curves show that APIVOT optimally balances success and computational efficiency, outperforming both text-only and image-every-step approaches.

Figure 8: Sorting—APIVOT selects an assignment affording greater placement flexibility, while VLM-TAMP’s choice leaves no feasible configuration.

Figure 9: On Storing Leftovers, APIVOT successfully composes sorting and spatial storage, handling compound constraints that Gemini-ER-1.5 cannot.

Limitations and Future Directions

Limitations include reliance on simulation (KitchenWorlds) for supervision, which may not fully capture real-world visual/geometric variability. Latent visual representations, although effective, are supervised solely on within-distribution samples. Direct optimization of modality selection for downstream task completion, potentially via RL, remains unexploited. Additionally, APIVOT reasons in language and full latent images; extension to more abstract and efficient representations (points/boxes) could yield further improvements in both performance and cost.

Scaling to more complex, visually diverse, and real-world tasks remains an open avenue, with the anticipation that combining foundation model pretraining (text and vision), fine-grained geometry supervision, and adaptive abstraction learning could yield planners with superior physical reliability and generalization.

Conclusion

APIVOT demonstrates that integrating adaptively interleaved language and visual reasoning inside the planner’s own trace can significantly improve long-horizon robotic planning. Task success gains are largest in environments where geometric factors critically impact feasibility. Beyond raw accuracy, APIVOT achieves these improvements efficiently, leveraging learned structure to invoke visual computation only when warranted by the task. This work substantiates the case for explicit, internally multimodal planning as a path toward robust and scalable embodied AI systems.