APIVOT: Adaptive Vision-Language Robot Planning

• APIVOT is a vision-language model that integrates semantic task reasoning with internal geometric verification for long-horizon robot manipulation.
• It interleaves language thoughts for subgoal decomposition with visual thoughts that imagine future scene states to ensure spatial feasibility.
• Empirical results in KitchenWorlds show that APIVOT outperforms traditional VLMs and planners, particularly under spatially constrained and complex tasks.

APIVOT, short for Adaptive Planning with Interleaved Vision-Language Thoughts, is a vision-language-model-based planner for long-horizon robot planning that is designed to couple semantic task reasoning with internal checks of geometric feasibility. Its central premise is that successful execution in manipulation tasks requires not only decomposing goals, selecting relevant objects, and sequencing actions, but also verifying spatial constraints such as free space, fit, and collisions. APIVOT addresses this by generating an interleaved reasoning trace composed of language thoughts for semantic reasoning and visual thoughts representing imagined future scene states for internal verification of geometric feasibility. In the reported KitchenWorlds experiments, it outperforms general-purpose VLMs and prior planning frameworks, with the largest gains appearing in spatially constrained settings (Jin et al., 9 Jul 2026).

1. Problem formulation and motivating setting

APIVOT is formulated for long-horizon kitchen manipulation tasks in which semantic task structure and geometry must be planned jointly. The paper centers on tasks such as Containment, Sorting, and Storing Leftovers, where a robot must identify goal-relevant objects, satisfy semantic preconditions, and choose an action order that remains feasible under clutter, capacity limits, and downstream placement constraints (Jin et al., 9 Jul 2026).

The motivating argument is that a semantically plausible plan can still fail because it ignores geometry. A robot may choose containers that do not fit the objects, place items in ways that block later steps, or leave insufficient free space for remaining placements. The paper therefore positions APIVOT against approaches that keep geometry outside the planner’s own reasoning process, for example through external motion planners, feasibility checkers, learned dynamics models, or reflection and replanning modules. APIVOT instead places geometric verification inside the reasoning trace by allowing the model to imagine future states during planning (Jin et al., 9 Jul 2026).

This design reflects a specific division of labor across modalities. Language thoughts are used for subgoal decomposition, semantic grouping, precondition analysis, and ordering constraints. Visual thoughts are used for spatial layout, free-space reasoning, collision checking, and future-feasibility verification. The method’s distinctive claim is not merely that both modalities matter, but that the planner should learn when to invoke each one (Jin et al., 9 Jul 2026).

2. Interleaved reasoning trace and planning representation

APIVOT takes as input a task instruction, a current RGB image observation of the scene, a list of scene objects and their image-space locations, and the available action schema and task constraints. It produces a structured reasoning trace in a > ... section and a formal plan in an <answer>...</answer> section (Jin et al., 9 Jul 2026).

The planner predicts actions in a high-level action space $\mathcal{A}_{\mathrm{plan}}$, consisting of:

• open(obj)
• pick(obj)
• place(obj, target, u, v)

where $(u,v)$ is a normalized image-space placement coordinate in $[0,1000]$. The execution environment is described as $(\mathcal{S}, \mathcal{A}, \mathcal{T})$, with $$\mathcal{T}: \mathcal{S} \times \mathcal{A}_{\mathrm{sim}} \rightarrow \mathcal{S}$$ the simulator transition function. APIVOT does not directly output simulator actions in $\mathcal{A}_{\mathrm{sim}}$; instead, it outputs $\mathcal{A}_{\mathrm{plan}}$, and predicted $(u,v)$ are later lifted into world-frame 3D placements by unprojection and raycasting against simulator geometry (Jin et al., 9 Jul 2026).

Within the reasoning trace, each step includes four elements: Planning intention, Key constraint, Modality decision, and Justification. If a step requires visual verification, APIVOT emits a visual-thought span immediately after the textual reasoning step. A visual thought is represented as

$$<|image\_start|>\;<|image\_pad|>^{K}\;<|image\_end|>$$

with fixed $K=16$. These tokens are not displayed RGB images at test time; they correspond to a latent visual state generated inside the model (Jin et al., 9 Jul 2026).

A crucial property of these visual thoughts is that they represent future post-subgoal states, not the current observation. During supervision, each visual thought is paired with a rendered RGB image after the corresponding subgoal has been achieved. This enables what the paper calls internal verification of geometric feasibility: after proposing a placement, the model can imagine the resulting scene and reason about whether later objects will still fit or whether obstacles must first be cleared (Jin et al., 9 Jul 2026).

3. Adaptive modality selection and geometric supervision

The “adaptive” aspect of APIVOT refers to learned per-step modality selection. The model is trained to decide whether a planning step should remain text-only or should invoke a visual thought. This decision is supervised during dataset construction rather than determined by a separate analytic controller at inference time (Jin et al., 9 Jul 2026).

During trace construction, each subgoal is annotated with a purpose, a constraint type, and a modality label. Constraint types include:

• symbolic precondition
• current geometric constraint
• future geometric feasibility

For place subgoals, the data pipeline computes simulator-derived geometric metadata: available free space on the target surface, free-space margin for the current object, remaining free space after placement, and whether remaining objects’ footprints fit in the leftover area. These are computed from projected object bounding boxes in the $(u,v)$0–$(u,v)$1 plane. Modality labels are then assigned heuristically: symbolic precondition steps are labeled text-only, whereas geometric, current-feasibility, and future-feasibility steps are labeled image if the geometric heuristic exceeds a predefined threshold (Jin et al., 9 Jul 2026).

At inference time, APIVOT does not use a separate classifier. The choice is made implicitly by autoregressive generation under the structured prompt: the model learns when to emit only text reasoning and when to emit text followed by a visual-thought span (Jin et al., 9 Jul 2026).

The paper presents this as a way to balance reasoning fidelity and budget. Some decisions, such as opening a fridge door or grouping foods by type, are primarily semantic and can be handled in language. Others, such as checking collision-free placement or preserving room for future objects, are geometry-sensitive and are better handled through visual imagination. This suggests a planning regime in which modality use is itself part of the learned policy (Jin et al., 9 Jul 2026).

4. Architecture, latent visual states, and training curriculum

APIVOT is initialized from Qwen3-VL-8B-Instruct and fine-tuned with LoRA. LoRA is applied to attention and MLP projection modules with rank $(u,v)$2 and scaling $(u,v)$3. Since Qwen3-VL-8B has hidden dimension $(u,v)$4, each visual thought corresponds to a latent tensor

$(u,v)$5

The start and end tokens supervise the structure, while the 16 image-pad positions carry the latent imagined visual state (Jin et al., 9 Jul 2026).

Training proceeds through a three-stage supervised fine-tuning curriculum:

1. Comprehension: the model learns to reason over provided visual thoughts and use them for planning. Training weights are

$(u,v)$6

2. Generation: the model learns to autoregressively generate visual thoughts itself and optimize them for planning. Training weights are

$(u,v)$7

3. Adaptation: the model learns adaptive modality selection, generating visual thoughts only when useful. Training weights are

$(u,v)$8

The overall training objective is described as a combination of a cross-entropy loss over the output trace and final plan, together with a visual alignment loss:

$(u,v)$9

The paper states that $[0,1000]$0 aligns generated latent visual thoughts with encoded ground-truth images of intermediate simulator states, but it does not provide the explicit algebraic form of this loss (Jin et al., 9 Jul 2026).

5. Data generation, simulator setting, and task suite

All training and evaluation are conducted in KitchenWorlds, a PyBullet-based simulator. Training uses 2,000 examples per training task family for Containment and Sorting, while evaluation uses 100 examples per dataset for held-out Containment, held-out Sorting, and held-out Storing Leftovers. Additional out-of-distribution datasets increase horizon length and task complexity through tighter free-space constraints and more prerequisite actions (Jin et al., 9 Jul 2026).

The task families are defined as follows:

Task family	Goal	Characteristic difficulty
Containment	Place target objects into a constrained storage region	Door state, clutter, obstacle removal, free-space reasoning
Sorting	Sort food objects by type into separate containers	Semantic grouping plus capacity-aware assignment
Storing Leftovers	Sort foods into containers, then place the filled containers in the fridge	Composes sorting and containment; held out from training

Training supervision is synthesized through simulator-generated expert traces. The pipeline samples kitchen scenes with varied object counts, clutter, distractors, door states, and free space; generates successful reference trajectories with PDDLStream and FastDownward; converts them to APIVOT’s high-level action format; decomposes them into subgoals; annotates purpose, constraint type, and modality; computes geometric heuristics from simulator replay; and finally uses Gemini-3-flash-preview to expand the structured skeleton into natural-language planner-style reasoning with visual thought placeholders. Each image-labeled step is paired with the rendered post-subgoal RGB image (Jin et al., 9 Jul 2026).

The evaluation compares APIVOT against three baseline families: general-purpose VLMs (Gemini-3.1-Pro, Gemini-ER-1.5, Qwen3-VL-8B-Instruct, Qwen3-VL-8B-Thinking), VLM planning frameworks (Reflect-VLM, VLM-TAMP), and a symbolic planner (FastDownward) whose symbolic states are inferred from image and language using BLADE (Jin et al., 9 Jul 2026).

6. Empirical performance, ablations, and modality behavior

The primary evaluation metric is task success rate under a common closed-loop receding-horizon execution setup. For VLM-based methods, the paper also reports token usage, including internal reasoning tokens for extended-thinking models. A separate analysis stratifies performance by occupancy ratio, defined as total area of task-relevant objects divided by available target area, as a measure of geometric difficulty (Jin et al., 9 Jul 2026).

The main results report the following success rates:

Model	Avg	Contain / Sort / Store
Gemini-ER-1.5	0.338	0.364 / 0.339 / 0.311
VLM-TAMP	0.329	0.370 / 0.311 / 0.307
APIVOT	0.419	0.472 / 0.421 / 0.365

APIVOT exceeds the strongest VLM baseline, Gemini-ER-1.5, by 8.1 percentage points, and the strongest planning baseline, VLM-TAMP, by 9.0 points. Task-wise, compared with VLM-TAMP, APIVOT improves Containment from 0.370 to 0.472, Sorting from 0.311 to 0.421, and Storing Leftovers from 0.307 to 0.365 (Jin et al., 9 Jul 2026).

The gains become larger as geometric difficulty increases. Using occupancy ratio, the paper states that APIVOT widens its advantage from 0.07 points at low occupancy to 0.17 points at high occupancy. This is presented as direct evidence that the improvement comes from stronger internal geometric reasoning rather than only stronger semantic planning (Jin et al., 9 Jul 2026).

The ablation studies separate the effects of training with visual thoughts, using them at inference, and learning to invoke them adaptively. A text-only fine-tuned variant reaches 0.244, compared with 0.188 for the base Qwen3-VL-8B-Instruct and 0.419 for APIVOT. A reasoning-modality ablation reports:

• Text / Text: 0.244
• Text+Image / Text: 0.282
• Text+Image / Image (always): 0.459
• Text+Image / Image (adaptive): 0.419
• Text+Image / Image (GT oracle): 0.482

This establishes several points. Multimodal training helps even if test-time inference is text-only; inference-time visual thoughts help substantially; adaptive interleaving retains much of the benefit of always using image thoughts; and a remaining gap to GT oracle visual thoughts indicates that the accuracy of imagined visual states is still a limiting factor (Jin et al., 9 Jul 2026).

A training-stage ablation shows the role of the three-stage curriculum:

• APIVOT: success 0.419, token usage 262
• w/o Stage 1: Comprehension: 0.301, 235
• w/o Stage 2: Generation: 0.345, 206
• w/o Stage 3: Adaptation: 0.457, 498

The interpretation given in the paper is that Stage 1 is crucial for learning to understand visual thoughts, Stage 2 is needed to generate them effectively, and Stage 3 teaches selective usage, trading some success for a large reduction in token usage (Jin et al., 9 Jul 2026).

The paper further reports that APIVOT’s adaptive strategy retains 91% of the performance of the always-image variant while reducing token usage by 39%. Constraint-aware analysis of the generated traces shows that APIVOT uses visual thoughts on 48.4% of steps involving limited free space and on 56.2% of steps involving downstream feasibility constraints. This supports the claim that the model is not using images indiscriminately, but has learned a meaningful modality-selection policy (Jin et al., 9 Jul 2026).

7. Limitations, failure modes, and conceptual significance

The appendix identifies four main failure modes: Premature goal completion, Incorrect action ordering, Missing prerequisite actions, and Geometric grounding failures. The last of these is particularly important: even when the high-level plan is correct, execution can fail because the predicted placement point $[0,1000]$1 is inaccurate in tight scenes (Jin et al., 9 Jul 2026).

The paper also reports out-of-distribution results on longer-horizon and more complex settings. APIVOT reaches 0.338 on the longer-horizon set and 0.336 on the more complex set, remaining competitive or strongest among the compared methods. At the same time, the paper notes that APIVOT exhibits one of the larger in-distribution to out-of-distribution drops, which it attributes to the increased difficulty of encoding denser clutter in the learned latent visual states (Jin et al., 9 Jul 2026).

Several limitations are explicitly acknowledged. Evaluation is limited to KitchenWorlds simulator rather than real robots. The latent visual representations are trained on a relatively narrow task distribution. Adaptive modality selection is learned through supervised fine-tuning, not directly optimized for task success. The framework reasons only over text and latent images, and the authors suggest that richer intermediate abstractions such as points or boxes could improve performance (Jin et al., 9 Jul 2026).

A common source of confusion is nomenclature. APIVOT refers specifically to Adaptive Planning with Interleaved Vision-Language Thoughts (Jin et al., 9 Jul 2026). It is distinct from PIVOT, a separate trajectory-refinement framework for LLM agents (Zhang et al., 11 May 2026). Conceptually, APIVOT’s significance lies in making modality choice part of the planner itself: the planner does not only decide what to do, but also which modality to think in at each step. The paper’s broader implication is that long-horizon robot planning benefits from planners that combine semantic decomposition in language with internal geometric verification through visual imagination, rather than treating those functions as entirely separate subsystems (Jin et al., 9 Jul 2026).