LLaPa: Vision-Language Framework for Procedural Planning

• LLaPa is a counterfactual-aware multimodal planning framework that fuses natural language and visual inputs to generate context-driven, executable action sequences.
• It employs a Task-Environment Reranker (TER) to focus on task-relevant regions and a Counterfactual Activities Retriever (CAR) to isolate visual cues for conditional planning.
• The integrated design improves planning executability and counterfactual accuracy, with performance validated on benchmarks like ActPlan-1K and ALFRED.

Searching arXiv for the LLaPa paper and closely related procedural planning work to ground the article. LLaPa, short for “A Vision-LLM Framework for Counterfactual-Aware Procedural Planning,” is a multimodal procedural planning system for embodied AI that converts a natural-language household instruction and visual observations of the environment into an executable action sequence. It is designed to address a specific limitation of earlier planning systems: strong text-based reasoning without visual grounding, or visual input processing without explicit alignment between task language and the scene elements that matter for execution, especially when instructions contain counterfactual conditions such as “if the microwave contains burnt cookies, discard them first.” In the formulation introduced with the model, textual input and visual input are jointly mapped to a plan that is decomposed into normal and counterfactual-dependent components, with explicit clause-level separation between ordinary instructions and counterfactual clauses (Sun et al., 11 Jul 2025).

1. Problem setting and formalization

LLaPa targets embodied procedural planning in settings where an agent must infer an executable action list from both language and images. The system takes as input a set of task sentences and one or more environment images, and produces actions of the form $\mathrm{action}(\mathrm{object})$ (Sun et al., 11 Jul 2025).

The paper formalizes the textual task description as

$$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$

where $S_{\text{goal}}$ denotes the main goal and the remaining sentences may encode context or conditions. Visual observations are represented as

$\mathbf{I} = \{I_1, I_2, \dots, I_m\}.$

The learning objective is defined as

$$\mathbf{A} = \phi\bigl([\mathbf{S}_{\text{norm}}, \mathbf{S}_{\text{ctrf}}, \mathbf{I}]\bigr),$$

where the output action list $\mathbf{A}=\{A_1,\dots,A_t\}$ is decomposed into a normal plan $\mathbf{A}_{\text{norm}}$ and a counterfactual-dependent plan $\mathbf{A}_{\text{ctrf}}$. A clause-level classifier,

$$\alpha_{\text{cls}}(\mathbf{S}) = [\mathbf{S}_{\text{norm}}, \mathbf{S}_{\text{ctrf}}],$$

separates ordinary instructions from counterfactual clauses.

This formulation is significant because it makes the counterfactual branch an explicit object of modeling rather than an implicit by-product of sequence generation. A plausible implication is that the framework is intended not only to improve plan quality but also to isolate the part of the instruction that conditions replanning under alternative environmental states.

2. Backbone architecture and guided prompting

The backbone is a standard multimodal pipeline in which a ViT-based image encoder maps each image to patch-level embeddings $v \in \mathbb{R}^{m \times P^2 \times C}$, and a text encoder maps the instruction to text embeddings $$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$0. These embeddings are projected into a shared language space through an MLP projector and then passed to the LLM, which generates the action sequence (Sun et al., 11 Jul 2025).

What distinguishes LLaPa from a generic VLM is that visual features are not consumed directly. They are first refined by the Task-Environment Reranker (TER), and counterfactual visual tokens are then extracted by the Counterfactual Activities Retriever (CAR). The inference-time input is constructed as

$$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$1

The paper describes this as a “guided prompt” design that explicitly reminds the LLM to use counterfactual environmental features during planning.

This architecture places LLaPa between two prior model families discussed in the paper. Relative to general-purpose VLMs such as GPT-4o, Gemini-Pro-1.5, LLaVA-OV, VideoLLaMA2, DeepSeek-VL2, Qwen2-VL, and InternVL2, it introduces explicit task-to-region alignment. Relative to text-centric planners such as PlaSma and E2CL, it retains visual grounding. The model is therefore positioned as a planner that treats multimodal grounding and counterfactual adaptation as coupled problems rather than separate capabilities.

3. Task-Environment Reranker (TER)

TER is the first auxiliary module and is inspired by task-oriented segmentation (TOS). Its purpose is to force the visual stream to attend to objects relevant to the task instead of merely salient objects in the scene. The paper argues that ordinary VLMs often over-attend to irrelevant but prominent items, which can lead to hallucinated tools or incorrect environmental understanding (Sun et al., 11 Jul 2025).

TER uses a frozen promptable segmentation model, such as Grounded-SAM, to produce binary masks for task-relevant objects from the text-image pair. The segmentation pipeline includes a vision encoder, a prompt encoder that embeds the text, and a mask decoder that outputs masks

$$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$2

Because the VLM operates on image patches, the pixel-level mask is converted into patch-level weights through adaptive max pooling,

$$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$3

For multiple textual constraints, masks are aggregated with a bitwise OR,

$$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$4

and the masks over all images are concatenated into a global attention mask $$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$5.

TER injects this mask into visual self-attention, producing reranked visual features $$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$6. In effect, it constructs a task-sensitive visual representation that suppresses distractors while preserving the spatial structure of important regions. The paper is explicit that TER is not merely a segmentation add-on; it is a reranking mechanism designed to make visual grounding explicit and compatible with the patch-based VLM backbone. This addresses a common misconception that segmentation alone is the core contribution. In the paper’s formulation, segmentation is only the upstream mechanism through which reranking becomes task-sensitive.

4. Counterfactual Activities Retriever (CAR)

CAR is the second auxiliary module and is intended for counterfactual reasoning. It receives the text embedding $$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$7, the reranked visual features $$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$8, and the spatial masks $$\mathbf{S} = \{S_{\text{goal}}, S_1, S_2, \dots, S_n\},$$9. The instruction is segmented into clauses $S_{\text{goal}}$0, where $S_{\text{goal}}$1 is the main goal and the remaining clauses are candidate constraints. An MLP classifier with sigmoid output estimates whether each clause is counterfactual:

$S_{\text{goal}}$2

Clauses with high probability are collected into the counterfactual index set $S_{\text{goal}}$3 (Sun et al., 11 Jul 2025).

CAR then projects the identified counterfactual conditions back into the image domain using the corresponding mask regions. For each relevant image, a binary counterfactual mask $S_{\text{goal}}$4 is formed, and the reranked features are masked element-wise:

$S_{\text{goal}}$5

To reduce noise, CAR applies conditional pooling over sub-grids. Each patch is divided into $S_{\text{goal}}$6 sub-grids, and the feature of a sub-grid is pooled only if it intersects the counterfactual mask:

$S_{\text{goal}}$7

Here $S_{\text{goal}}$8. Concatenating the sub-grid outputs yields counterfactual feature tokens $S_{\text{goal}}$9.

The central idea is selective preservation rather than global averaging. The paper explicitly contrasts CAR with ordinary average pooling and argues that conditional pooling better retains task-critical context while discarding unrelated clutter. This suggests that the model treats counterfactual evidence as a sparse, spatially localized signal that should be isolated before it is exposed to the LLM.

5. Training procedure, implementation, and benchmarks

LLaPa is trained in two stages. In the first stage, only the counterfactual classifier is trained while the rest of the network is frozen, using binary cross-entropy:

$\mathbf{I} = \{I_1, I_2, \dots, I_m\}.$0

In the second stage, the planning model is fine-tuned end-to-end for action generation using a standard language modeling loss:

$\mathbf{I} = \{I_1, I_2, \dots, I_m\}.$1

The total objective therefore combines counterfactual clause detection with sequence generation (Sun et al., 11 Jul 2025).

Implementation details are unusually concrete. The model is built on InternVL2-8B, with an InternViT visual encoder and patch size $\mathbf{I} = \{I_1, I_2, \dots, I_m\}.$2. TER uses Grounded-SAM, and the reranking self-attention uses 8 heads. CAR uses $\mathbf{I} = \{I_1, I_2, \dots, I_m\}.$3, producing 16 counterfactual-related visual tokens per image. Training uses COPLAN plus ActPlan-1K for the counterfactual classifier and EgoCOT plus ActPlan-1K for the planning stage. The code and models are publicly available at https://github.com/sunshibo1234/LLaPa.

The evaluation uses ActPlan-1K, described as the only multimodal procedural planning benchmark explicitly supporting counterfactual activities, ALFRED as a standard embodied instruction-following benchmark for generalization, and MFE-ETP for broader embodied task planning evaluation. The primary metrics are Executability, LCS, Correctness, and Accuracy on MFE-ETP.

Benchmark	Setting	Reported result
ActPlan-1K	Counterfactual tasks	53.2% executability, 0.57 LCS, 36.1% correctness
ActPlan-1K	Normal tasks	62.9% executability, 0.62 LCS, 45.2% correctness
ALFRED	Standard evaluation	85.3% executability, 0.62 LCS, 48.6% correctness
MFE-ETP	Accuracy	58.6%

These results are presented as evidence that LLaPa improves both ordinary grounded planning and counterfactual planning. The paper states that on the listed evaluations it outperforms the advanced models included in the comparison set.

6. Empirical interpretation, ablations, and limitations

The ablation studies reinforce the modular interpretation of the framework. The best counterfactual-token setting is reported as $\mathbf{I} = \{I_1, I_2, \dots, I_m\}.$4; smaller values reduce granularity, while larger values introduce too much noise. Removing CAR hurts counterfactual correctness noticeably, and removing TER causes a large drop in executability and overall planning quality. Using only standard supervised fine-tuning performs worst. The paper also reports that bitwise OR mask aggregation in TER outperforms summation/normalization, similarity-based weighting, and no aggregation (Sun et al., 11 Jul 2025).

A further result concerns transferability. Integrating TER and CAR into other open-set VLMs such as LLaVA-OV and Qwen2-VL improves performance substantially. This suggests that the two auxiliary modules are not presented as backbone-specific heuristics but as broadly reusable components for multimodal planning.

The qualitative and error analyses are aligned with that interpretation. The paper reports fewer hallucinated tools, fewer environmental misunderstandings, and reduced hallucination of extra tools and incorrect image interpretation on ActPlan-1K. At the same time, it acknowledges two limitations: the system depends heavily on rich textual task descriptions, and it does not yet incorporate dynamic video context. A plausible implication is that the framework is strongest in static or image-based planning settings where the instruction explicitly encodes the relevant contingency structure.

7. Conceptual significance and relation to prior planning systems

LLaPa’s principal innovation is described not simply as stronger generation but as structured visual reranking plus counterfactual-aware retrieval. TER improves object relevance and reduces hallucination by aligning task language with the correct visual regions, while CAR strengthens conditional planning by identifying counterfactual clauses and extracting their visual correlates (Sun et al., 11 Jul 2025).

This positioning clarifies how LLaPa relates to two recurring misconceptions in multimodal planning. The first is that generic multimodal attention is sufficient for embodied procedural planning. The paper argues against this by introducing explicit segmentation-derived reranking and clause-conditioned visual retrieval. The second is that counterfactual replanning can be handled adequately by text-only methods. The paper contrasts LLaPa with text-centric replanning systems, arguing that such methods may detect the need to replan but lack the visual context required to do it well.

Within embodied AI, the framework implies that executable planning benefits from explicit interfaces between language, spatial relevance, and conditional structure. The model’s reported gains on both normal and counterfactual tasks suggest that counterfactual awareness does not function as an isolated add-on; rather, it is integrated into a broader multimodal grounding strategy. In that sense, LLaPa is best understood as a modular VLM planner whose core contribution lies in making visual evidence selection and conditional adaptation first-class components of procedural planning.