Cross-Modal Contextual Instructions

• Cross-modal contextual instructions are distributed signals combining text, images, audio, and more to guide task execution in varied settings.
• Recent studies show that latent task representations in VLMs enable cross-modal transfer, boosting performance in robotics, reinforcement learning, and navigation.
• Benchmark analyses and security evaluations reveal practical gains alongside vulnerabilities, emphasizing the need for robust evaluation and defense strategies.

Cross-modal contextual instructions denote instruction regimes in which task-relevant control is not confined to a single explicit text command, but is distributed across modalities and contextual signals such as text, images, speech, environmental sound, demonstrations, or scene state. In recent work, the phrase appears explicitly in robotic manipulation as a setting “where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands,” and closely related formulations arise in vision-LLMs (VLMs), instruction-guided reinforcement learning, navigation, and robot motion synthesis, where task information can be specified in one channel and executed in another, or inferred from multimodal context rather than from a literal prompt alone (Wang et al., 27 Oct 2025, Luo et al., 2024, Saghafian et al., 2024, Hong et al., 2024, Barron et al., 25 Sep 2025).

1. Conceptual scope and defining properties

Across the cited literature, cross-modal contextual instructions are characterized by three recurring properties. First, the instruction signal is distributed: text, images, sketches, dialogue, sound, gaze, or trajectories all function as carriers of task information. Second, interpretation is context-dependent: the meaning of an instruction depends on scene layout, temporal progress, subtask completion, or background semantics rather than on token identity alone. Third, the resulting control signal is often transferable across modalities: a task specified in text may steer image answering, image demonstrations may guide text output, or a multimodal prompt may induce motion in 3D space (Luo et al., 2024, Saghafian et al., 2024, Barron et al., 25 Sep 2025).

A broad typology emerges. In autoregressive VLMs, contextual instructions are studied as latent task representations induced by text examples, image examples, or instructions. In instruction-guided RL, they are modeled as alignments between language and observation-action trajectories, together with online progress estimates. In robotics, they appear as proactive intent inference from dialogue, sound, and vision, or as mixed-modality instruction tuples composed of images, sketches, and free-form text. In navigation, they take the form of interleaved text-and-image prompts attached to landmark phrases. In multimodal verification and ambiguity resolution, they appear as structured checks over sentiment, narrative, temporal, and logical consistency, or as iterative image-grounded disambiguation (Luo et al., 2024, Saghafian et al., 2024, Wang et al., 27 Oct 2025, Hong et al., 2024, Ma et al., 8 Aug 2025, Ni et al., 2024).

A common misconception is that cross-modal instruction following is simply multimodal input processing. The cited work points to a stricter criterion: the multimodal context must act as an instructional constraint on downstream behavior. In CAREL, trajectory-word similarity changes which subtask remains active during an episode; in CrossInstruct, annotated scene images function as behavioral examples rather than passive observations; in VLN-MP, prompt images are inserted into the instruction stream itself rather than treated as environment observations (Saghafian et al., 2024, Barron et al., 25 Sep 2025, Hong et al., 2024).

2. Latent task representations in vision-LLMs

The clearest mechanistic account comes from the finding that autoregressive VLMs can compress conceptually equivalent task specifications into a shared latent task vector. The formulation is explicit: given in-context examples $S=\{x_i,y_i\}_{i=1}^n$, a model $F$ solves $y_q = F(x_q,S)$ by first inducing a latent vector $z = G(S)$, then generating with $y_q = F(x_q \mid z)$; a stable estimate is $z=\mathbb{E}_S[G(S)]$. In the VLM setting, $G$ is operationalized as the activation at a chosen transformer layer from the final delimiter token of a prompt containing text examples, image examples, or an instruction. This supports the central claim that task representations can be invariant both to modality and to format (Luo et al., 2024).

Functionally, this invariance is tested by activation patching. A task vector extracted from a source prompt in one modality is patched into a target query in another modality. The strongest result is text-to-image transfer. Across LLaVA-v1.5, Mantis-Fuyu, and Idefics2, patched text-derived task vectors consistently outperform same-window text ICL on image queries, with a reported $14\%-33\%$ advantage. The averages are $0.18 \rightarrow 0.32$ for LLaVA-v1.5, $0.09 \rightarrow 0.31$ for Mantis-Fuyu, and $F$0 for Idefics2. On Idefics2, patched text vectors also exceed the stronger image-ICL baseline on average, $F$1 versus $F$2, and instruction-only vectors match the patching performance of a five-example vector; averaging instruction-only and exemplar-based vectors yields an $F$3 improvement over the five-example vector in the low-data regime. Task vectors can also transfer from the base LLM to the fine-tuned VLM, with cosine similarity $F$4 for LLaVA/Vicuna and $F$5 for Idefics2/Mistral, versus a random mismatched-pair baseline of $F$6 (Luo et al., 2024).

Representation analysis suggests why this is possible. Using logit lens on Idefics2, the hidden state of the final token passes through three phases—input, task, answer—regardless of whether the context examples are textual or visual. Middle-layer states decode to task-summary tokens such as “headquarters,” “currency,” or “species,” and image-ICL task vectors decode to similar language-like summaries rather than arbitrary image-specific tokens. t-SNE visualizations cluster by task more than by specification type at the relevant layers, and successful cross-modal patching shows functional interchangeability. A plausible implication is that VLMs map different instruction channels into a common semantic control space, even when the externally visible prompt forms differ (Luo et al., 2024).

The limitations are equally explicit. The most robust effect is one-sided: text-to-image transfer is much stronger than image-to-text transfer on the main benchmark. The task vector depends on selecting a favorable layer by validation-set grid search; the best layer is around the middle of the network for late-fusion models and later-middle for Mantis-Fuyu. The benchmark itself contains only six structured mapping tasks plus a few qualitative examples, and patching is an intervention rather than proof that the model naturally uses a modular task vector in exactly the same way without intervention (Luo et al., 2024).

3. Grounding instructions in embodied control and robot behavior

In instruction-guided RL, CAREL treats cross-modal contextual instructions as alignments between language and trajectories rather than between language and single observations. An instruction is encoded into local token representations $F$7, while each observation-action pair $F$8 is encoded into a local representation $F$9. The framework adds an auxiliary retrieval-style loss to a baseline RL objective,

$y_q = F(x_q,S)$0

so that successful trajectories and their instructions become nearby in latent space at multiple granularities: whole episode with whole instruction, episode with word, observation with instruction, and observation with word. CAREL then uses these similarity signals for instruction tracking: as a subtask’s similarity spikes relative to its running history, that subtask is masked from the instruction so the policy can focus on the residual goal. In BabyAI and SHELM, vanilla CAREL improves sample efficiency across all BabyAI levels, with especially strong gains on sequential tasks such as GoToSeq and OpenDoorsOrder; adding instruction tracking and action embeddings improves performance further, and tracking is reported as especially useful for more complex RGB tasks (Saghafian et al., 2024).

A more radical formulation appears in proactive robotic manipulation. RoboOmni defines cross-modal contextual instructions as a setting “where auditory (speech and environmental sound) and visual cues are fused to infer latent user intent and verified proactively by interaction, in contrast to conventional setups that assume explicit commands.” The framework is a Perceiver–Thinker–Talker–Executor architecture built on an end-to-end omni-modal LLM. At time step $y_q = F(x_q,S)$1, it computes visual and audio embeddings $y_q = F(x_q,S)$2 and $y_q = F(x_q,S)$3, concatenates them with textual context $y_q = F(x_q,S)$4 into $y_q = F(x_q,S)$5, and autoregressively predicts either text tokens or action tokens. The OmniAction dataset comprises 141,162 multimodal episodes, 5,096 distinct speaker timbres, 2,482 non-verbal sound events, 640 environmental backgrounds, 112 skills, and 748 objects. On OmniAction-LIBERO-TTS, RoboOmni achieves $y_q = F(x_q,S)$6 average success rate, versus $y_q = F(x_q,S)$7 for the strongest baseline, NORA with ASR; on OmniAction-LIBERO-Real, it reaches $y_q = F(x_q,S)$8, outperforming $y_q = F(x_q,S)$9 at $z = G(S)$0, OpenVLA at $z = G(S)$1, NORA at $z = G(S)$2, and OpenVLA-OFT at $z = G(S)$3. Intent recognition reaches $z = G(S)$4, versus $z = G(S)$5 for ASR+GPT-4o (Wang et al., 27 Oct 2025).

CrossInstruct moves from intent inference to motion synthesis. It replaces physical motion demonstrations with an instruction tuple

$z = G(S)$6

where $z = G(S)$7 is a scene image, $z = G(S)$8 is a free-form sketch, and $z = G(S)$9 is optional text. The target is a distribution over robot trajectories,

$y_q = F(x_q \mid z)$0

A large reasoning VLM produces semantic keypoints $y_q = F(x_q \mid z)$1, a smaller fine-tuned pointing model grounds them into image coordinates, the large model synthesizes 2D trajectories over multiple views, and calibrated ray casting fuses them into 3D waypoint Gaussians. The resulting trajectories can be executed directly or used to initialize TD3+BC. On RLBench, CrossInstruct reaches $y_q = F(x_q \mid z)$2 on Basketball-in-Hoop, $y_q = F(x_q \mid z)$3 on Close Drawer, $y_q = F(x_q \mid z)$4 on Slide Block to Target, $y_q = F(x_q \mid z)$5 on Play Jenga, $y_q = F(x_q \mid z)$6 on Lift Numbered Block, $y_q = F(x_q \mid z)$7 on Put Rubbish in Bin, and $y_q = F(x_q \mid z)$8 on Push Button, substantially exceeding a reasoning-only baseline on most tasks; on Jenga, CrossInstruct-initialized RL reaches around $y_q = F(x_q \mid z)$9 success after 40k RL steps, while training from scratch yields no nonzero return under sparse binary reward (Barron et al., 25 Sep 2025).

Navigation work reaches a similar conclusion from a different angle. VLN-MP generalizes text-only VLN by inserting images into landmark phrases, defining multimodal instructions of the form

$z=\mathbb{E}_S[G(S)]$0

The framework supports aligned, related, and terminal prompt settings, and uses MPF to fuse prompt images with instruction text before passing the result into backbones such as HAMT or DUET. On RxR-MP with HAMT, val unseen SR improves from $z=\mathbb{E}_S[G(S)]$1 for text-only to $z=\mathbb{E}_S[G(S)]$2 with a terminal prompt, $z=\mathbb{E}_S[G(S)]$3 with related prompts, $z=\mathbb{E}_S[G(S)]$4 with aligned prompts, and $z=\mathbb{E}_S[G(S)]$5 with aligned prompts from Marky-mT5. In CVDN-MP, image+text outperforms both text-only and image-only for both HAMT and DUET, which is especially notable because the benchmark is designed around highly ambiguous object references (Hong et al., 2024).

4. Ambiguity, referential grounding, and contextual verification

A major line of work treats cross-modal contextual instructions as a disambiguation problem. Visual-O1 formulates multimodal instruction understanding as

$z=\mathbb{E}_S[G(S)]$6

where $z=\mathbb{E}_S[G(S)]$7 is an ambiguous instruction and $z=\mathbb{E}_S[G(S)]$8 is visual context. The paper studies ambiguity categories of ellipsis, colloquialism, subjectivity, relativity, and other, and introduces a multi-modal multi-turn chain-of-thoughts reasoning framework with reasoning, reflection, and response synthesis. For strong models, the system accumulates instance-specific reasoning and reflection history; for weaker models, it distills empirical experience across examples and uses it to rewrite ambiguous instructions into clearer ones before final answering. On ambiguous-instruction splits, LISA improves from $z=\mathbb{E}_S[G(S)]$9 to $G$0 in gIoU/cIoU, SoM from $G$1 to $G$2, LLaVA from $G$3 accuracy and $G$4 BLEU-1 to $G$5 and $G$6, and GPT-4o from $G$7 and $G$8 to $G$9 and $14\%-33\%$0. The same framework also improves general RIS and VQA performance, which suggests that image-grounded disambiguation is useful beyond explicitly ambiguous subsets (Ni et al., 2024).

An earlier robotic grounding formulation makes a complementary point: context can come from gaze and demonstrations as well as from vision and language. “Grounding Symbols in Multi-Modal Instructions” parses instructions into abstract plans of the form $14\%-33\%$1, aligns fixation streams to actions using GLIDE, extracts image patches for attended objects, and learns per-symbol Gaussian feature models with invariant-feature selection. The system is trained on approximately 2000 labeled image patches with an estimated $14\%-33\%$2 correct annotation rate and $14\%-33\%$3 mislabelling, and on a test set of 48 objects it recognizes $14\%-33\%$4 of presented colours and $14\%-33\%$5 of presented shapes. The paper’s central claim is that gaze provides the referential signal needed to ground symbols such as blue or cube in a user-specific way from small, online datasets (Hristov et al., 2017).

A different disambiguation problem arises in multimodal news verification. ContextGuard-LVLM argues that entity consistency is insufficient because visual narrative, emotional tone, background information, temporal or spatial fit, and scene-event logic can all diverge while the entities still match. It introduces Fine-grained Cross-modal Contextual Consistency (FCCC), decomposed into sentiment, narrative, background, temporal/spatial consistency, and logical coherence. From a fused image-text representation $14\%-33\%$6, the model extracts contextual vectors $14\%-33\%$7, fuses them, and predicts both overall consistency and per-context scores. The CTXT results are illustrative: sentiment $14\%-33\%$8 versus $14\%-33\%$9 for InstructBLIP/LLaVA 1.5, narrative $0.18 \rightarrow 0.32$0 versus $0.18 \rightarrow 0.32$1, background $0.18 \rightarrow 0.32$2 versus $0.18 \rightarrow 0.32$3, temporal/spatial $0.18 \rightarrow 0.32$4 versus $0.18 \rightarrow 0.32$5, and logical coherence $0.18 \rightarrow 0.32$6 versus $0.18 \rightarrow 0.32$7. On a subtly perturbed test set, average accuracy drops from $0.18 \rightarrow 0.32$8 to $0.18 \rightarrow 0.32$9 for InstructBLIP and from $0.09 \rightarrow 0.31$0 to $0.09 \rightarrow 0.31$1 for LLaVA 1.5, but only from $0.09 \rightarrow 0.31$2 to $0.09 \rightarrow 0.31$3 for ContextGuard-LVLM (Ma et al., 8 Aug 2025).

These works jointly reject another common misconception: contextual instruction following is not exhausted by entity matching or literal instruction parsing. In the cited formulations, ambiguity often survives after object identity is known; what matters is whether the model can infer latent referents, scene logic, sentiment framing, or implied constraints from multimodal context (Ni et al., 2024, Ma et al., 8 Aug 2025, Hristov et al., 2017).

5. Benchmarks and evaluation regimes

Recent benchmarks make cross-modal contextual instructions measurable rather than anecdotal. MCIF is the first multilingual human-annotated benchmark based on scientific talks designed to evaluate instruction following in crosslingual, multimodal settings over both short- and long-form inputs. It spans speech, vision, and text across English, German, Italian, and Chinese; uses 21 presentations for the main multitask benchmark and 100 samples for summarization; and organizes 13 tasks across recognition, translation, question answering, and summarization. A key design feature is that prompts are written in the target language while evidence remains in English source material, and that question-answer pairs are modality-labeled as A, V, AV, or NA. The benchmark also includes fixed-prompt and mixed-prompt variants, and the results show substantial brittleness under prompt reformulation and long-context multimodal reasoning (Papi et al., 25 Jul 2025).

A complementary line of work addresses data scarcity in multimodal instruction alignment itself. X-InstructBLIP introduces a modular framework that aligns image, audio, video, and 3D point cloud inputs to a frozen LLM using modality-specific Q-Formers or linear projections. To support instruction-aware alignment outside the image domain, it automatically generates 24K QA data for audio and 250K QA data for 3D from caption data and introduces DisCRn, with 9K audio-video QA samples and 28K image-3D QA samples, for discriminative cross-modal reasoning. The strongest result is conceptual rather than architectural: joint and discriminative cross-modal reasoning can emerge even though each modality projector is trained independently and the LLM remains frozen (Panagopoulou et al., 2023).

Navigation benchmarking adds another dimension: contextual prompts can vary in relevance and density rather than being merely present or absent. VLN-MP constructs R2R-MP, RxR-MP, REVERIE-MP, and CVDN-MP with aligned, related, and terminal prompt settings. On R2R-MP, 17,328 of 17,409 original instructions are converted into multimodal instructions, with only $0.09 \rightarrow 0.31$4 failing to detect any landmark image; the average number of landmarks in the aligned setting is $0.09 \rightarrow 0.31$5, with a maximum of $0.09 \rightarrow 0.31$6. On RxR-MP, the aligned setting averages $0.09 \rightarrow 0.31$7 landmarks, with a maximum of $0.09 \rightarrow 0.31$8. These regimes enable evaluation not just of whether models can use images in instructions, but of how they degrade as prompt fidelity or prompt count changes (Hong et al., 2024).

A practical implication of these benchmark designs is that cross-modal contextual instruction following is now evaluated along several orthogonal axes: modality combination, instruction paraphrase sensitivity, context length, prompt relevance, task granularity, and whether the answer requires joint evidence or discriminative comparison. This suggests that future systems will need to be robust not only to more modalities, but also to more varied instruction structures (Papi et al., 25 Jul 2025, Panagopoulou et al., 2023, Hong et al., 2024).

6. Security, limitations, and open questions

Cross-modal contextual instructions also define a new attack surface. CrossInject shows that in multimodal agents the effective prompt is distributed across the system prompt $0.09 \rightarrow 0.31$9, visual input $F$00, external data $F$01, and user command $F$02, formalized as $F$03. Under attack, the output becomes

$F$04

The visual component aligns the perturbed image with a target image generated from a malicious instruction, while the textual component optimizes a malicious command against an inferred system prompt $F$05. On RecipeMaster and PoetryGenius across local-document and online-webpage settings, CrossInject outperforms naive prompting, JIP, and UPI; the paper reports at least a $F$06 increase in attack success rates across diverse tasks, and in a driving-assistant case study it succeeds in 9 of 10 trials versus 4 of 10 for naive malicious text-only injection (Wang et al., 19 Apr 2025).

COMET extends this red-teaming logic to VLMs by distributing harmful meaning across three stages: knowledge-scalable reframing, cross-modal clue entangling, and cross-modal scenario nesting. The final attack prompt concatenates entangled text, an entangled benign-looking image, a rubric image, and a scenario image that frames the payload as a reasoning benchmark. The paper reports over $F$07 ASR across advanced VLMs in aggregate and a $F$08 improvement over the best baseline, and the ablations show that scenario nesting especially increases harmfulness score even when it contributes less to raw jailbreak rate on its own (Yan et al., 9 Feb 2026).

These security results make a broader point. The same properties that make contextual instruction following powerful—distributed semantics, modality transfer, latent control, and context-sensitive interpretation—also make it difficult to defend with unimodal prompt filters or shallow image inspection. Prompt-based defenses fail because neither the malicious text nor the malicious image needs to contain the full harmful instruction in isolation; the harmful intent emerges from multimodal reconstruction (Wang et al., 19 Apr 2025, Yan et al., 9 Feb 2026).

Across the broader literature, recurring limitations are explicit. VLM task-vector studies rely on narrow task suites and interventionist activation patching; CAREL depends on successful episodes and rule-based subtask parsing; RoboOmni does not formalize an uncertainty-aware ask/act policy; Visual-O1 still uses a high-intelligent model during the reflection stage for general-intelligent models; ContextGuard-LVLM leaves several symbols and training details underspecified; CrossInstruct depends on calibrated cameras and omits a formal end-to-end training objective (Luo et al., 2024, Saghafian et al., 2024, Wang et al., 27 Oct 2025, Ni et al., 2024, Ma et al., 8 Aug 2025, Barron et al., 25 Sep 2025).

A plausible synthesis is that the field is converging on a common view: contextual instructions are best treated not as literal prompt tokens but as structured, multimodal control signals. The unresolved questions are therefore less about whether such signals exist than about how they are represented, how they should be benchmarked, when they should be trusted, and how to prevent them from becoming vectors for misalignment or attack.