Learning from Cross-Modal Instructions

• Learning from Cross-Modal Instructions is a framework where models combine diverse modalities to map instructions into actionable outputs such as generation, perception, navigation, or control.
• The approach integrates methods like cross-modal few-shot learning, reinforcement learning, and explicit grounding to improve task performance across varied environments.
• Recent advances leverage frozen backbone models with instruction-aware projections and local-global grounding mechanisms to achieve robust cross-modal reasoning and measurable performance gains.

Learning from cross-modal instructions denotes a family of methods in which supervision is conveyed through coordinated signals from more than one modality—such as text with images, language with visual observations, sketches with scene images, or language with gaze traces—and the model learns to map those signals to generation, perception, navigation, or control outputs. Across the literature, the problem appears in at least three closely related forms: symbol grounding from multimodal demonstrations, instruction-conditioned decision making in embodied environments, and multimodal generation or adaptation with foundation models. The common technical theme is that instruction meaning is not treated as purely linguistic; it is inferred through alignment, attention, or grounding operations that couple language to perception, spatial layout, action history, or a shared embedding space (Hristov et al., 2017, Wang et al., 2018, Zhang et al., 2020, Lin et al., 2023, Panagopoulou et al., 2023, Kumar et al., 2024, Saghafian et al., 2024, Barron et al., 25 Sep 2025).

1. Conceptual scope and development

Early work framed the problem as symbol grounding under severe data constraints. In a table-top manipulation scenario, a raw stream of cross-modal input—linguistic instructions, visual perception of a scene, and a concurrent trace of 3D eye tracking fixations—was used to produce segmentation of objects with a correspondent association to high-level concepts. The system parsed narration into abstract instruction tuples $(\text{action}, \text{target}, \text{location})$, used GLIDE to segment fixation streams and associate them with actions, and then learned per-symbol probabilistic models from automatically labeled image patches. This formulation emphasized online learning, small numbers of demonstrations, and generalization to novel combinations of words such as color and shape terms (Hristov et al., 2017).

Vision-and-language navigation later recast learning from cross-modal instructions as sequential decision making in photo-realistic 3D environments. In this setting, an embodied agent must attend to the meaningful paragraphs of a language instruction while processing dynamically varying visual scenes. Reinforced Cross-Modal Matching introduced local cross-modal reasoning plus a matching critic that supplied an intrinsic reward based on trajectory–instruction reconstruction, while subsequent work added a cross-modal grounding module with two complementary attention mechanisms and alternated imitation and exploration under adversarial regularization to reduce the train–inference discrepancy (Wang et al., 2018, Zhang et al., 2020).

Recent work broadened the paradigm beyond navigation. Cross-modal few-shot learning treated examples from different modalities as additional support samples in a shared representation space, effectively turning $n$-shot into $(n+1)$-shot classification when class names or audio examples are available (Lin et al., 2023). X-InstructBLIP aligned images, video, audio, and 3D point clouds to a frozen LLM through instruction-aware projections and reported emergent cross-modal reasoning when multiple modalities were concatenated at inference time, despite no joint fine-tuning on paired multi-modality data (Panagopoulou et al., 2023). MGCC addressed multimodal prompt sequences for image generation by combining a frozen CLIP ViT-L encoder, a frozen OPT-6.7B backbone, a Cross-Modal Refinement Module, and a contextual object grounding module (Kumar et al., 2024). CAREL adapted the problem to instruction-guided reinforcement learning with cross-modal auxiliary objectives and instruction tracking (Saghafian et al., 2024). CrossInstruct extended the idea to robot motion generation, replacing physical motion demonstrations with rough annotations consisting of an RGB image, free-form sketches, and optional textual labels or constraints (Barron et al., 25 Sep 2025).

2. Representation spaces and alignment mechanisms

A central design choice is whether cross-modal instructions are fused through explicit shared latent spaces or through stepwise grounding modules layered on top of frozen backbones. In cross-modal few-shot adaptation, the foundation assumption is that models such as CLIP map different modalities into the same $\mathbb{R}^N$. Frozen encoders $f_v$, $f_t$, and $f_a$ produce $L_2$-normalized features, and a single linear classifier $W \in \mathbb{R}^{N \times C}$ is trained on a support set augmented with text prompts or audio clips. Because no manual pairing is required between the image and text or audio samples beyond a shared class label, the auxiliary modality is treated as an additional shot rather than as a separate task (Lin et al., 2023).

Instruction-aware alignment to frozen LLMs is developed further in X-InstructBLIP. The framework trains one projection module per modality on top of frozen encoders—EVA-CLIP-ViT-G/14 for image and video frames, BEATs$_{\text{iter3+}}$ for audio, and ULIP-2 with a PointBERT backbone for 3D point clouds—and feeds the projected tokens, modality cues, and textual instruction into a frozen Vicuna 7B or 13B model. Two projection mechanisms are explored: a Q-Former with hidden dimension $n$0 and $n$1 learnable queries, and a lighter linear projection baseline. The Q-Former conditions the projection on instruction text, so the representation is not merely modality-aligned but instruction-aware (Panagopoulou et al., 2023).

MGCC uses a different alignment route for multimodal generation. A frozen CLIP ViT-L encoder $n$2 maps each image $n$3 to a $n$4-dimensional feature vector, and a learned linear layer $n$5 splits that feature into $n$6 vectors of dimension $n$7, producing LLM-token embeddings. The LLM, OPT-6.7B, is frozen except for new image tokens, the Cross-Modal Refinement Module, and a 4-layer encoder–decoder transformer $n$8. After refinement, the aligned conditioning features are

$n$9

which are then passed to the diffusion model together with grounding information (Kumar et al., 2024).

These mechanisms reflect two distinct but compatible views of learning from cross-modal instructions. One view treats modalities as interchangeable evidence inside a shared feature geometry; the other treats them as structured inputs that require explicit token-level or object-level refinement before a downstream model can use them. This suggests that the choice of alignment mechanism is strongly shaped by the output space: classifier weights, LLM token streams, diffusion conditioning, or action policies.

3. Grounding instructions in perception, space, and action

Instruction learning becomes operational only when the model can ground symbols or sub-instructions in perceptual context. In the symbol-grounding formulation, the English Resource Grammar produced Elementary Dependency Structures, from which an abstract plan $(n+1)$0 and a symbol set $(n+1)$1 were extracted. GLIDE then used monocular SLAM from ORB-SLAM to recover the eye tracker pose, lifted 2D fixation rays into 3D, segmented fixations by Bayesian inference with a Dirichlet prior over change-point locations, and labeled image patches with the subset of symbols appearing in the instruction tuple’s target. This created a pipeline from free-form English to grounded symbol hypotheses without hand-labeled visual data (Hristov et al., 2017).

In embodied navigation, grounding is handled by attention over both language and visual context. One approach used a reasoning navigator that maintained a history context $(n+1)$2, attended over panoramic view features, then attended over instruction tokens and re-attended locally over the scene before predicting actions. A related model introduced two complementary paths: historical co-grounding, where the previous hidden state queries both modalities, and mutual co-grounding, where global visual and textual summaries interact directly before producing modality-specific attention weights. The purpose in both cases is to ensure that the agent tracks the correspondence between textual instructions and the current scene rather than relying on language or vision alone (Wang et al., 2018, Zhang et al., 2020).

CAREL extends grounding to sub-instruction progression. The framework computes episode–instruction, episode–word, observation–instruction, and observation–word similarities, aggregates them through Attention Over Similarity Matrix, and uses the time-varying similarity scores as a detector of subtask completion. For a subtask $(n+1)$3, completion is declared when

$(n+1)$4

after which the completed subtask is removed from future inputs and optionally masked with probability $(n+1)$5. This mechanism converts cross-modal alignment signals into a curriculum over remaining instruction content (Saghafian et al., 2024).

Grounding can also be explicitly spatial. MGCC’s Contextual Object Grounding Module constructs an in-context prompt of the form “You are a bounding-box generator” and has the same OPT-6.7B model output a list of box coordinates $(n+1)$6, which are concatenated with $(n+1)$7 and passed into the diffusion U-Net. Because COGM is zero-shot at inference, no explicit box-regression loss is used during training (Kumar et al., 2024). CrossInstruct moves from 2D grounding to 3D motion synthesis: a large VLM produces semantic keypoint descriptors $(n+1)$8, a smaller pointing model returns pixel coordinates in each calibrated view, and the reasoning model then sketches two pixel-space trajectories $(n+1)$9 and $\mathbb{R}^N$0. These define Gaussian densities in image space, which are lifted by ray geometry into a distribution over 3D points; the resulting mean positions, along with orientations and gripper commands produced by the reasoning model, form the executable open-loop plan $\mathbb{R}^N$1 (Barron et al., 25 Sep 2025).

4. Learning objectives and supervision regimes

The literature spans a wide range of supervision regimes, from maximum-likelihood density fitting to reinforcement learning with auxiliary objectives. In the symbol-grounding setting, the core learner fits Gaussian models for each symbol over invariant feature subsets. For symbol $\mathbb{R}^N$2 and feature $\mathbb{R}^N$3, the empirical mean and variance are computed, outliers beyond $\mathbb{R}^N$4 are pruned, invariant features are selected by the criterion $\mathbb{R}^N$5, and the final diagonal Gaussian is used to evaluate $\mathbb{R}^N$6 at test time. No explicit loss function is optimized beyond this maximum-likelihood fitting (Hristov et al., 2017).

Cross-modal few-shot adaptation uses a much simpler objective. After augmenting the support set with text prompts $\mathbb{R}^N$7 or audio examples $\mathbb{R}^N$8, a single linear classifier is learned by minimizing cross-entropy over the concatenated batch of image, text, and audio features:

$\mathbb{R}^N$9

This formulation makes the additional modality an ordinary supervised sample rather than a separately aligned regularizer (Lin et al., 2023).

Instruction-aware LLM alignment is typically trained with causal language modeling. X-InstructBLIP keeps the LLM frozen and updates only the Q-Former parameters and linear projection, minimizing a standard cross-entropy loss between generated token distributions and gold targets. MGCC combines three objectives: a language modeling loss, an MSE alignment loss between the mapper output and the frozen CLIP text embedding, and the diffusion denoising loss for U-Net $f_v$0-prediction. Its total objective is

$f_v$1

Only $f_v$2, the new image-token embeddings, the CMRM layers, and the mapper $f_v$3 are learnable during training on Conceptual Captions 3M (Panagopoulou et al., 2023, Kumar et al., 2024).

Instruction-conditioned control problems adopt hybrid RL objectives. CAREL augments the base RL loss with an InfoNCE-style cross-modal contrastive loss computed only on successful episodes, yielding

$f_v$4

The auxiliary term pulls matched episode–instruction pairs together while pushing mismatched pairs apart at multiple granularities (Saghafian et al., 2024). In navigation, alternate adversarial learning combines teacher-forced imitation, student-forced exploration, and an adversarial discriminator over hidden-state trajectories:

$f_v$5

A related navigation framework combines extrinsic reward from progress toward the goal with intrinsic reward from instruction–trajectory reconstruction, and then adds Self-Supervised Imitation Learning by storing and imitating the agent’s own highest-scoring rollouts in unseen environments (Zhang et al., 2020, Wang et al., 2018).

CrossInstruct uses yet another supervision pattern. Its open-loop motion distribution can be executed directly, but it also supplies a demonstration dataset $f_v$6 for a downstream TD3 + Behavior Cloning pipeline. The actor minimizes a weighted sum of a BC term and a critic term, the critic is trained with a standard bootstrapped squared-error loss, and the BC term is retained throughout training to prevent catastrophic drift (Barron et al., 25 Sep 2025).

5. Benchmarks, task families, and reported results

The task family is unusually broad. In multimodal generation, MGCC was evaluated on VIST and VisDial. On the VIST dataset with multimodal inputs, MGCC achieved a CLIP Similarity score of $f_v$7 compared to GILL $f_v$8. On VisDial with lengthy dialogue sequences, MGCC achieved a CLIP score of $f_v$9, compared to $f_t$0 for the existing method. On CC3M, FID improved from $f_t$1 to $f_t$2. Ablations on VIST reported $f_t$3 with COGM only, $f_t$4 with CMRM only, and $f_t$5 with the full model; increasing the number of CMRM layers from $f_t$6 to $f_t$7 changed FID from $f_t$8 to $f_t$9 and CLIP from $f_a$0 to $f_a$1 (Kumar et al., 2024).

In few-shot learning, cross-modal augmentation was reported on 11 vision–language datasets with a ResNet50 backbone. Cross-modal linear probing improved 1-shot performance from $f_a$2 to $f_a$3 and 16-shot performance from $f_a$4 to $f_a$5; partial fine-tuning of the last pooling layer further raised those numbers to $f_a$6 and $f_a$7. On the ImageNet-ESC benchmark, adding one audio shot improved 1-shot image classification from $f_a$8 to $f_a$9 on ESC-19 and from $L_2$0 to $L_2$1 on ESC-27, while adding one image shot improved 1-shot audio classification from $L_2$2 to $L_2$3 on ESC-19 and from $L_2$4 to $L_2$5 on ESC-27 (Lin et al., 2023).

Instruction-aware multimodal LLM alignment was evaluated across 16 benchmarks in X-InstructBLIP. On ModelNet40 zero-shot 3D understanding, X-InstructBLIP$L_2$6(7B) reported $L_2$7 closed-set classification and $L_2$8 open-generation, compared with $L_2$9 and $W \in \mathbb{R}^{N \times C}$0 for the linear projection baseline. On audio tasks, X-InstructBLIP$W \in \mathbb{R}^{N \times C}$1(7B) reported $W \in \mathbb{R}^{N \times C}$2 on ESC50 closed-vocabulary classification and $W \in \mathbb{R}^{N \times C}$3 on open classification; on silent video, it reported $W \in \mathbb{R}^{N \times C}$4 CIDEr on MSVD captioning and $W \in \mathbb{R}^{N \times C}$5 on MSVD QA. On DisCRn, the same model reported $W \in \mathbb{R}^{N \times C}$6 on image–3D and $W \in \mathbb{R}^{N \times C}$7 on audio–video, compared with $W \in \mathbb{R}^{N \times C}$8 and $W \in \mathbb{R}^{N \times C}$9 for the linear projection baseline (Panagopoulou et al., 2023).

For embodied instruction following, Reinforced Cross-Modal Matching reported on the R2R test set that RCM achieved path length $_{\text{iter3+}}$0, navigation error $_{\text{iter3+}}$1, oracle success rate $_{\text{iter3+}}$2, success rate $_{\text{iter3+}}$3, and SPL $_{\text{iter3+}}$4, while Speaker-Follower reported SPL $_{\text{iter3+}}$5. Adding SIL on the training set kept success rate at $_{\text{iter3+}}$6 while raising SPL to $_{\text{iter3+}}$7 and reducing path length to $_{\text{iter3+}}$8. On unseen validation, using SIL to explore the unseen environment reported success rate $_{\text{iter3+}}$9 and reduced the seen–unseen gap from $n$00 to $n$01 (Wang et al., 2018). In a related R2R study, the full CMG + AAL model achieved on val-unseen navigation error approximately $n$02 m, success rate $n$03, and SPL $n$04, and on test-unseen navigation error approximately $n$05 m, success rate $n$06, and SPL $n$07 (Zhang et al., 2020).

Cross-modal instruction learning also appears in reinforcement learning and robotics. On BabyAI after 1 M frames, CAREL increased GoToSeq success rate from approximately $n$08 to approximately $n$09, OpenDoorsOrder from $n$10 to $n$11, PickupLoc from $n$12 to $n$13, PutNext from $n$14 to $n$15, and Synth from $n$16 to $n$17. On SHELM PutNear, baseline performance of $n$18 increased to $n$19 with CAREL and $n$20 with CAREL plus tracking (Saghafian et al., 2024). On RLBench, direct CrossInstruct execution reported success rates of $n$21 on Basketball-in-Hoop, $n$22 on Insert Block on Peg, $n$23 on Close Drawer, $n$24 on Slide Block to Target, $n$25 on Play Jenga, $n$26 on Lift Numbered Block, $n$27 on Put Rubbish in Bin, and $n$28 on Push Button. In downstream RL, CrossInstruct-initialized TD3+BC converged to approximately $n$29 success in 40 k steps on Play Jenga, while TD3 and SAC from scratch never achieved nonzero returns; on Insert Peg on Block, CrossInstruct+TD3+BC reached $n$30 success by 400 k steps, while SAC and TD3 from scratch remained near $n$31 (Barron et al., 25 Sep 2025).

The small-data symbol-grounding setting remains important as a counterpoint to foundation-model approaches. On a test set of 48 new physical objects, the reported color recognition accuracy was $n$32, whereas shape recognition was $n$33, with the paper attributing the weaker shape result to the fact that the single feature “pixel area” was insufficient to distinguish several object categories (Hristov et al., 2017).

6. Common misconceptions, limitations, and future directions

A frequent misconception is that cross-modal instruction learning necessarily requires densely paired multimodal supervision or joint multimodal fine-tuning. Several results contradict this directly. Cross-modal few-shot adaptation adds class names or audio clips as extra shots without manual pairing beyond class identity (Lin et al., 2023). X-InstructBLIP trains each Q-Former in isolation on single-modality instruction-tuning data and combines modalities only at inference time, yet reports emergent cross-modal reasoning (Panagopoulou et al., 2023). CrossInstruct explicitly replaces expensive kinesthetic or tele-operated demonstrations with sketches and optional text annotations (Barron et al., 25 Sep 2025).

The main limitations are heterogeneous and domain-specific. In symbol grounding, shape grounding suffered because only a single area feature was used; Gaussian models also assume unimodal feature distributions and independence across features, and no lifelong update mechanism was evaluated (Hristov et al., 2017). In MGCC, the object grounding module is zero-shot and purely prompt-based at inference, so no explicit box-regression loss is used and no back-propagation passes through COGM (Kumar et al., 2024). CAREL relies on successful episodes for the contrastive loss, which means early training may have few positives, and its instruction parsing is rule-based (Saghafian et al., 2024). In cross-modal few-shot learning, benefits depend on good alignment in pretraining; when alignment is weak, gains are smaller (Lin et al., 2023). In X-InstructBLIP, the Q-Former projection shows lower generalization capabilities than linear projection in contexts where task-modality data are limited (Panagopoulou et al., 2023). In navigation, some objects remain ungrounded when they fall outside ImageNet pretraining, and error accumulation can derail the remainder of the instruction after an early misstep (Wang et al., 2018). Alternate adversarial learning characterizes the trade-off directly: pure imitation converges to short but brittle paths, whereas pure RL yields higher success but meandering trajectories (Zhang et al., 2020).

The cumulative picture is not that one architecture has solved the problem, but that a set of reusable mechanisms has emerged: shared embedding spaces, instruction-aware projections, local and global cross-modal grounding, in-context prompting for latent structure such as boxes or keypoints, auxiliary contrastive objectives, and self-generated demonstrations. A plausible implication is that future systems will continue to combine stronger grounding with lighter supervision: richer visual features and online continual updating for symbol learning, learned instruction chunking for RL, stronger object detectors and semantic maps for navigation, and broader open-vocabulary multimodal reasoning over video, depth, or other sensor streams (Hristov et al., 2017, Saghafian et al., 2024, Wang et al., 2018, Lin et al., 2023).