Counterfactual Activities Retriever (CAR)

• The paper demonstrates that CAR improves counterfactual task correctness by 5.4% through explicit detection of conditional clauses and extraction of relevant visual tokens.
• CAR is a specialized module that identifies counterfactual clauses from textual inputs and projects them onto visual features to influence the planning process.
• Empirical results reveal that incorporating CAR in LLaPa significantly boosts executability and reduces errors on counterfactual tasks by aligning text and image evidence.

Counterfactual Activities Retriever (CAR) is an auxiliary module in the vision-LLM framework LLaPa for multimodal procedural planning. In that setting, the input consists of a task description and one or more environment images, and the output is an ordered action sequence. CAR is designed to identify textual clauses that introduce potential counterfactual conditions and to project those clauses onto the visual domain so as to extract the corresponding visual tokens. Its role is therefore to make the planner explicitly sensitive to hypothetical or conditional task modifiers such as instructions of the form “if the microwave contains burnt cookies, discard them first.” Although the acronym suggests retrieval in a conventional database or nearest-neighbor sense, the module is more precisely a clause-conditioned visual token selector embedded inside a single forward-pass planning pipeline (Sun et al., 11 Jul 2025).

1. Conceptual scope and acronym disambiguation

Within LLaPa, CAR refers specifically to Counterfactual Activities Retriever, a component for counterfactual-aware procedural planning over text-image inputs (Sun et al., 11 Jul 2025). The module operates on the current multimodal input rather than on an external corpus, memory bank, or vector index. The paper is explicit that it is not a retrieval module in the usual nearest-neighbor / memory-bank / external-database sense; “retriever” denotes the extraction of relevant counterfactual visual tokens from the current input.

This usage is distinct from other contemporaneous systems that employ the same acronym. In tabular fairness, CAR denotes Counterfactual Attention Regularization, a fairness regularizer for the Feature Correlation Transformer rather than a retrieval mechanism (Dong et al., 29 Apr 2026). In retrieval-augmented generation, CAR denotes Confidence-Aware Reranking, a post-hoc reranking framework that uses generator confidence change as a document usefulness signal (Song et al., 6 May 2026). The acronym overlap is therefore terminological rather than conceptual.

The surrounding literature on counterfactual retrieval is broader than the LLaPa module. “Counterfactual Realizability” studies when a counterfactual distribution can be physically sampled from the world through lawful experimentation, rather than merely inferred (Raghavan et al., 14 Mar 2025). “CORE” develops retrieve-then-edit counterfactual generation for text, with a dense retriever trained to retrieve label-flipping excerpts (Dixit et al., 2022). “Generating Counterfactual Explanations Under Temporal Constraints” treats counterfactual process traces as activity sequences that must satisfy temporal background knowledge expressed in LTLp (Buliga et al., 3 Mar 2025). These neighboring lines of work show that “counterfactual retrieval” can refer to very different operations; in LLaPa, it denotes multimodal grounding of hypothetical activity conditions rather than corpus retrieval or causal identification.

2. Position within the LLaPa planning architecture

LLaPa formulates multimodal procedural planning with textual input

$\bm{S} = \{ S_{\text{goal}, S_1, S_2, \dots, S_n \},$

visual input

$\bm{I} = \{ I_1, I_2, \dots, I_m \},$

and an action-sequence output

$\bm{A} = \{ A_1, A_2, \dots, A_t \}.$

To account explicitly for counterfactual effects, the plan is decomposed as

$\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$

with a clause-separation function

$$\alpha_{cls}(\bm{S}) = [\bm{S}_{norm}, \bm{S}_{ctrf}],$$

and the overall planner written as

$$\bm{A}= \phi\bigl([\bm{S}_{norm}, \bm{S}_{ctrf}],\, \bm{I}\bigr).$$

This decomposition treats normal instructions and counterfactual information as distinct but jointly consumed inputs to the planner (Sun et al., 11 Jul 2025).

The architecture has three components: a VLM backbone, a Task-Environment Reranker (TER), and CAR. The VLM backbone consists of an image encoder, a text encoder, an MLP projector, and an LLM decoder. TER first reranks visual features using task-oriented segmentation masks so that the model emphasizes task-relevant objects. CAR then takes the textual embedding, TER’s reranked visual features, and TER’s spatial masks, and isolates the features relevant to counterfactual conditions. The LLM then generates the action sequence from the fused multimodal input.

This placement is central to CAR’s role. TER addresses task-environment alignment by suppressing visually salient but task-irrelevant regions. CAR addresses a different problem: recognition of hypothetical or conditional clauses that may modify the procedural plan. The module therefore sits after TER and before final decoding, converting detected counterfactual language into grounded visual evidence (Sun et al., 11 Jul 2025).

3. Internal mechanism and mathematical formulation

CAR takes three inputs: the textual embedding $s$, the TER-reranked visual features $v^{\text{rerank}}$, and the spatial weight masks $W$. The text embedding is segmented into clause-level representations

$\{s_0, s_1, \dots, s_n\},$

where $\bm{I} = \{ I_1, I_2, \dots, I_m \},$0 is the embedding of the main goal and each $\bm{I} = \{ I_1, I_2, \dots, I_m \},$1 corresponds to an auxiliary clause. For each clause $\bm{I} = \{ I_1, I_2, \dots, I_m \},$2, CAR evaluates whether that clause is counterfactual relative to the goal by applying a classifier to the concatenation $\bm{I} = \{ I_1, I_2, \dots, I_m \},$3:

$\bm{I} = \{ I_1, I_2, \dots, I_m \},$4

Here $\bm{I} = \{ I_1, I_2, \dots, I_m \},$5 is implemented as an MLP followed by sigmoid, and $\bm{I} = \{ I_1, I_2, \dots, I_m \},$6 is the probability that clause $\bm{I} = \{ I_1, I_2, \dots, I_m \},$7 is a counterfactual condition (Sun et al., 11 Jul 2025).

The detected clause indices form the set $\bm{I} = \{ I_1, I_2, \dots, I_m \},$8. For each detected counterfactual clause, CAR selects the corresponding patch-level weights from TER and aggregates them into a per-image binary counterfactual mask

$\bm{I} = \{ I_1, I_2, \dots, I_m \},$9

The mask is then applied to TER’s reranked visual features by element-wise multiplication:

$\bm{A} = \{ A_1, A_2, \dots, A_t \}.$0

This operation retains only those patch features aligned with counterfactual conditions (Sun et al., 11 Jul 2025).

To reduce noise while preserving local structure, CAR subdivides each $\bm{A} = \{ A_1, A_2, \dots, A_t \}.$1 patch block into $\bm{A} = \{ A_1, A_2, \dots, A_t \}.$2 sub-grids and performs conditional pooling inside each sub-grid:

$\bm{A} = \{ A_1, A_2, \dots, A_t \}.$3

with

$\bm{A} = \{ A_1, A_2, \dots, A_t \}.$4

The resulting sub-grid features are concatenated to obtain

$\bm{A} = \{ A_1, A_2, \dots, A_t \}.$5

The module output is therefore a compact set of counterfactual visual tokens that capture the visual regions and textual cues critical for counterfactual reasoning (Sun et al., 11 Jul 2025).

CAR affects generation through direct fusion into the decoder input. The final multimodal input to the LLM is

$\bm{A} = \{ A_1, A_2, \dots, A_t \}.$6

This formulation shows that counterfactual features are not used for post-hoc reranking or iterative replanning; they are injected directly into the context from which the LLM generates the action sequence (Sun et al., 11 Jul 2025).

4. Training regime and supervision

CAR is trained in two stages. In the first stage, only the counterfactual clause classifier is trained while the rest of the network is frozen. Its supervision consists of human-annotated binary clause labels $\bm{A} = \{ A_1, A_2, \dots, A_t \}.$7, and the loss is the binary cross-entropy

$\bm{A} = \{ A_1, A_2, \dots, A_t \}.$8

The paper states that the entire COPLAN dataset and the ActPlan-1K training set are used to train the CAR components in this stage (Sun et al., 11 Jul 2025).

In the second stage, the reranker’s self-attention layers, the MLP projector, and the LLM are fine-tuned with the language-modeling objective

$\bm{A} = \{ A_1, A_2, \dots, A_t \}.$9

This stage uses EgoCOT plus the ActPlan-1K training set. The overall learning setup thus separates explicit counterfactual-clause recognition from end-to-end procedural generation (Sun et al., 11 Jul 2025).

The reported implementation details situate CAR inside a specific VLM stack. The backbone is InternVL2-8B, the visual encoder is InternViT, the patch size is $\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$0, and the segmentation model used by TER is Grounded-Segment Anything. TER uses 8-head self-attention. The CAR token granularity is controlled by $\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$1, and with $\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$2 the module extracts $\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$3 counterfactual-related visual tokens per image (Sun et al., 11 Jul 2025).

Some operational details are left unspecified in the paper text. The threshold used to convert $\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$4 into $\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$5, the exact aggregation rule from $\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$6 to $\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$7, the hidden dimensions of the CAR MLP, and the batching details for clause tokenization are not reported. This limits exact reproducibility, even though the main computational structure is explicit.

5. Empirical behavior and ablation evidence

The main empirical evidence for CAR comes from counterfactual-task performance on ActPlan-1K. On the ActPlan-1K counterfactual split, LLaPa is reported to achieve Executability 53.2%, LCS 0.57, and Correctness 36.1%, outperforming several advanced baselines including GPT-4o, Qwen2-VL, InternVL2, and InternVL2(Plasma) on the reported numbers (Sun et al., 11 Jul 2025). In the component ablation on the same benchmark, the full model is reported at 53.9 Executability, 0.57 LCS, and 36.1 Correctness, while removing CAR yields 47.6, 0.51, and 30.7, respectively. The paper explicitly states that removing CAR causes a significant 5.4% decrease in correctness of counterfactual tasks, indicating that the module contributes specifically to hypothetical-condition handling rather than to generic planning quality.

The specialization of CAR is also visible on normal tasks. In the same ablation, the full model scores 62.7 / 0.62 / 45.2 on normal tasks, whereas the version without CAR scores 63.8 / 0.62 / 45.8. This pattern supports the paper’s interpretation that CAR is primarily beneficial for counterfactual conditions rather than universally advantageous across all task types (Sun et al., 11 Jul 2025).

The ablation over the number of counterfactual tokens shows a non-monotonic dependence on $\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$8. On ActPlan-1K counterfactual tasks, the reported values are:

• $\bm{A} = [\bm{A}_{norm}, \bm{A}_{ctrf}],$9: 49.2 / 0.51 / 32.5
• $$\alpha_{cls}(\bm{S}) = [\bm{S}_{norm}, \bm{S}_{ctrf}],$$0: 52.8 / 0.57 / 36.4
• $$\alpha_{cls}(\bm{S}) = [\bm{S}_{norm}, \bm{S}_{ctrf}],$$1: 41.0 / 0.49 / 26.8
• $$\alpha_{cls}(\bm{S}) = [\bm{S}_{norm}, \bm{S}_{ctrf}],$$2: 33.2 / 0.38 / 21.5

The best setting is therefore $$\alpha_{cls}(\bm{S}) = [\bm{S}_{norm}, \bm{S}_{ctrf}],$$3. The paper attributes this to a tradeoff between insufficient feature granularity when $$\alpha_{cls}(\bm{S}) = [\bm{S}_{norm}, \bm{S}_{ctrf}],$$4 is too small and excessive environmental noise when $$\alpha_{cls}(\bm{S}) = [\bm{S}_{norm}, \bm{S}_{ctrf}],$$5 is too large (Sun et al., 11 Jul 2025).

The reported error analysis also links CAR to improved counterfactual grounding. The approach achieves the lowest error percentages in two environment-related error categories—hallucination of extra tools and incorrect image understanding—with reported reductions of 8% and 4%, respectively. The paper attributes part of this improvement to the explicit emphasis on counterfactual conditions and the extraction of the most pertinent visual tokens (Sun et al., 11 Jul 2025).

6. Interpretation, limitations, and relation to adjacent work

CAR is best understood as a counterfactual clause detector plus visual token extractor rather than as a conventional retriever. It does not define an external retrieval corpus, similarity scores, top-$$\alpha_{cls}(\bm{S}) = [\bm{S}_{norm}, \bm{S}_{ctrf}],$$6 exemplars, or a memory-augmented lookup stage. This point is important because the literature contains several systems whose “retrieval” component does involve dense retrieval over external data. CORE, for example, performs dense retrieval over a task-related text corpus and then uses an editor to generate counterfactuals (Dixit et al., 2022). Causal-Counterfactual RAG performs graph-based retrieval followed by counterfactual re-querying of a causal knowledge graph under hypothetical alternatives (Khadilkar et al., 17 Sep 2025). By contrast, CAR in LLaPa retrieves only from the present multimodal context (Sun et al., 11 Jul 2025).

The module also has explicit limitations. The paper states that LLaPa relies heavily on comprehensive textual task descriptions, which is especially consequential for CAR because its first step is textual clause classification. The current approach also does not support dynamic video context integration, so the module grounds counterfactuals in static images rather than in evolving video streams. In addition, several implementation details are not fully specified, including thresholding and mask aggregation (Sun et al., 11 Jul 2025).

From a broader counterfactual-activity perspective, CAR occupies one point in a larger design space. “Counterfactual Realizability” argues that some Layer 3 counterfactual distributions are physically sampleable and provides CTF-REALIZE as a complete realizability checker (Raghavan et al., 14 Mar 2025). “Generating Counterfactual Explanations Under Temporal Constraints” shows that counterfactual activity traces can be constrained by LTLp and enforced through automata-theoretic operators (Buliga et al., 3 Mar 2025). These results suggest that a fuller theory of “counterfactual activity retrieval” may require distinctions among multimodal grounding, temporal feasibility, and physical realizability. LLaPa’s CAR addresses the first of these with a practical, single-pass VLM mechanism (Sun et al., 11 Jul 2025).

A plausible implication is that CAR should be treated not as a generic retrieval primitive, but as a specialized grounding module for counterfactual-aware planning. Its empirical utility lies in converting detected hypothetical clauses into compact visual evidence that the planner can consume directly. Its conceptual importance lies in showing that counterfactual conditions in procedural instructions can be operationalized as clause-conditioned multimodal feature selection rather than as a separate post-hoc reasoning stage.