Compositional Multimodal Instruction (CMI)

• CMI is a framework that structures multimodal instructions by interleaving text, image placeholders, and other modalities to guide task execution.
• It enables cross-domain applications such as image editing, visual reasoning, music reward modeling, and composed image retrieval through factorized sub-instructions.
• CMI leverages decomposition and routing mechanisms, integrating modules like U-Nets and CRFs, while addressing challenges in segmentation, safety, and instruction compliance.

Searching arXiv for the cited papers to ground the article in current records. I’m grounding the terminology and examples in the cited arXiv papers listed in the source block. Compositional Multimodal Instruction (CMI) denotes a class of instruction-following formulations in which the conditioning signal is both multimodal and compositionally structured. In recent literature, CMI appears as interleavings of text tokens with one or more 〈imagehere〉 placeholders for subject-driven generation and instruction-based editing, tuples $\mathcal{P}=(t,l,a_{\mathrm{ref}})$ that combine text, lyrics, and reference audio for music reward modeling, pairs $(i,t)$ in which a textual modifier is applied to a visual reference for composed image retrieval, and decompositions of a complex question or instruction into factors or sub-instructions that are later recomposed or executed in sequence (Tian et al., 28 Feb 2025, Ma et al., 28 Feb 2026, Zhong et al., 2024, Gu et al., 16 Oct 2025, Bai et al., 8 Feb 2026). A precursor to this line of work appears in modular embodied instruction following, where natural-language commands are segmented into typed subgoals and routed through specialized modules (Corona et al., 2020).

1. Conceptual scope across domains

Across the cited works, CMI is not tied to a single modality pair or a single downstream task. It is used in embodied instruction following, chart and webpage reasoning, diffusion-based image generation and editing, composed image retrieval, music reward modeling, and multimodal jailbreak construction. This suggests that CMI functions as a reusable abstraction for tasks in which an instruction must be interpreted jointly with heterogeneous evidence and then composed into a coherent action, generation, retrieval, or evaluation process (Corona et al., 2020, Gu et al., 16 Oct 2025, Tian et al., 28 Feb 2025, Bai et al., 8 Feb 2026, Zhong et al., 2024, Ma et al., 28 Feb 2026, Xu et al., 18 May 2026).

Work	Domain	CMI form
MIGE	Image generation and editing	Text interleaved with 〈imagehere〉
COGS	Visual reasoning	Factors, subquestions, intermediate answers
MCIE-E1	Complex image editing	Sub-instructions with bounding boxes
InstructCIR	Composed image retrieval	Pair $(i,t)$
CMI-RewardBench	Music reward modeling	Tuple $(t,l,a_{\mathrm{ref}})$
DMN	Multimodal jailbreak	Distributed instruction, evidence, number-chain images

A common misconception is that CMI is restricted to interleaving text with image placeholders. That representation is explicit in MIGE, but other works define CMI through factorized subquestions, prompt tuples spanning text, lyrics, and audio, or spatially grounded decompositions of a single complex instruction (Tian et al., 28 Feb 2025, Gu et al., 16 Oct 2025, Ma et al., 28 Feb 2026, Bai et al., 8 Feb 2026).

2. Formal representations and notation

One formalization appears in MIGE, where a multimodal instruction is “nothing more than a sequence of text tokens interleaved with one or more 〈imagehere〉 placeholders,” and each placeholder is replaced at run time by the visual embedding of a reference image. Subject-driven generation is treated as creation on a blank canvas, while instruction-based editing uses the VAE encoding of a source image as the conditional input to the same denoising process. The unifying idea is that both tasks preserve some part of an image while allowing free-form modifications under natural language guidance (Tian et al., 28 Feb 2025).

A second formalization appears in music reward modeling. CMI-RewardBench denotes a CMI prompt as

$\mathcal{P}=(t,l,a_{\mathrm{ref}}),$

where $t$ is an optional free-form textual description, $l$ is an optional sequence of lyrics, and $a_{\mathrm{ref}}$ is an optional reference audio clip. Given $\mathcal{P}$ and a candidate generated clip $a_{\mathrm{eval}}$, a CMI reward model outputs two real-valued scores, $(i,t)$0 for musicality and $(i,t)$1 for alignment (Ma et al., 28 Feb 2026).

A third formulation appears in composed image retrieval. In InstructCIR, a CMI query is a pair $(i,t)$2, where $(i,t)$3 is a source image and $(i,t)$4 is an instruction such as “make the shirt red” or “move the dog into a running pose.” The task is to retrieve a target image $(i,t)$5 whose embedding matches the composed embedding of $(i,t)$6, rather than to treat the image and text as independent retrieval cues (Zhong et al., 2024).

Decomposition-based formulations are equally central. COGS assumes that a complex seed question $(i,t)$7 over an image $(i,t)$8 can be factorized as

$(i,t)$9

where each factor $(i,t)$0 is a primitive perception or reasoning step with a category, a subquestion, and a ground-truth partial answer. MCIE-E1 uses an MLLM to decompose a complex editing instruction $(i,t)$1 into spatially grounded pairs

$(i,t)$2

The earlier modular instruction-following architecture on ALFRED similarly assumes that a full instruction decomposes into $(i,t)$3 contiguous segments, each corresponding to a subgoal type drawn from a small inventory $(i,t)$4 (Gu et al., 16 Oct 2025, Bai et al., 8 Feb 2026, Corona et al., 2020).

3. Decomposition, routing, and composition mechanisms

The modular ALFRED model provides a canonical decomposition-and-routing pipeline. A bidirectional LSTM encodes the instruction, and a linear-chain CRF predicts a joint BIO+type tagging over word-level labels. Viterbi decoding yields both subgoal boundaries and subgoal labels, and the controller achieves $(i,t)$5 exact match on held-out subgoal sequences. For each subgoal type $(i,t)$6, the architecture allocates an independent seq2seq subgoal module $(i,t)$7. At test time the controller predicts $(i,t)$8, executes modules in order, and passes the final hidden state of one module into the initial hidden state of the next. No explicit recursion or branching is used; control passes through a learned linear chain (Corona et al., 2020).

COGS extends the same decomposition logic to multimodal reasoning rather than embodied action. Its pipeline has three stages: seed question decomposition, question generation via factor recomposition, and reinforcement learning fine-tuning. The synthetic dataset contains not only the final answer but also subquestions and intermediate answers, which permits factor-level process rewards. The paper compares StandardRM, ProcessRM-sum, and ProcessRM-max, and reports that ProcessRM-max best preserves the correct policy ordering when sub-rewards are noisy (Gu et al., 16 Oct 2025).

MCIE-E1 applies decomposition to image editing with explicit spatial grounding. A pretrained multimodal LLM decomposes a single complex instruction into $(i,t)$9 sub-instructions, each paired with a bounding box. Each sub-instruction is encoded via CLIP, and the boxes are mapped into continuous spatial embeddings through a small Fourier feature network. This places decomposition not only at the semantic level but also at the geometric level, making spatial guidance part of the instruction representation itself (Bai et al., 8 Feb 2026).

Taken together, these systems treat compositionality as an operational property: the model does not merely receive a long prompt, but rather receives or infers a structured sequence of factors, subgoals, or spatial regions that govern downstream computation.

4. CMI in generation and editing systems

MIGE makes CMI the central interface for a unified diffusion model. Each reference image is processed in two parallel streams: a pretrained VAE encoder produces dense visual embeddings $(t,l,a_{\mathrm{ref}})$0, and a frozen EVA-CLIP ViT followed by a small Q-Former produces semantic tokens $(t,l,a_{\mathrm{ref}})$1. These are fused by a lightweight cross-attention + MLP block,

$(t,l,a_{\mathrm{ref}})$2

and then interleaved with text embeddings before entering a frozen Flan-T5-XXL. The resulting multimodal condition vectors are used as cross-attention keys and values in each U-Net block. Training uses a single denoising objective for both generation and editing,

$(t,l,a_{\mathrm{ref}})$3

with $(t,l,a_{\mathrm{ref}})$4 defined as either a real image latent for editing or an all-zero tensor for generation (Tian et al., 28 Feb 2025).

MCIE-E1 focuses on the specific difficulties of complex instruction image editing: insufficient instruction compliance and background inconsistency. Its diffusion U-Net uses two parallel cross-attention modules. Spatial-Aware Cross-Attention (SACA) aligns each sub-instruction with its spatial region through masked soft-attention and timestep-aware masks, while Background-Consistent Cross-Attention (BCCA) preserves unedited regions by attending to source-image features outside the union of edit masks. The two streams are fused as

$(t,l,a_{\mathrm{ref}})$5

and training uses the standard diffusion $(t,l,a_{\mathrm{ref}})$6 denoising loss on a Stable Diffusion 1.5 backbone (Bai et al., 8 Feb 2026).

These architectures treat compositionality as a property of conditioning and not only of outputs. In MIGE, a single instruction may describe arbitrary compositions of subjects, scenes, and operations by concatenating image snippets and free-form text. In MCIE-E1, a complex edit is decomposed into region-specific operations whose execution is modulated throughout denoising. A plausible implication is that CMI becomes most useful when instruction adherence and preservation constraints must be optimized simultaneously.

5. Evaluation protocols and reported empirical behavior

In COGS, the primary empirical claim is improved generalization to unseen questions and compositional structures. On ChartQAPro, the base model Qwen2.5-VL-7B reaches $(t,l,a_{\mathrm{ref}})$7 accuracy, while COGS with ProcessRM-max reaches $(t,l,a_{\mathrm{ref}})$8, a gain of $(t,l,a_{\mathrm{ref}})$9 points. The largest gains are reported on multi-hop questions with at least 3 factors and on complex factor types such as Extrapolation $\mathcal{P}=(t,l,a_{\mathrm{ref}}),$0 and Comparison $\mathcal{P}=(t,l,a_{\mathrm{ref}}),$1. Under factor-level mixing of ChartQAPro and MMC, the model reaches $\mathcal{P}=(t,l,a_{\mathrm{ref}}),$2 on ChartQAPro and $\mathcal{P}=(t,l,a_{\mathrm{ref}}),$3 on MMC, and on VisualWebBench COGS improves Qwen2.5-VL-7B from $\mathcal{P}=(t,l,a_{\mathrm{ref}}),$4 to $\mathcal{P}=(t,l,a_{\mathrm{ref}}),$5 (Gu et al., 16 Oct 2025).

MIGE reports cross-task gains for both generation and editing. On DreamBench, it achieves subject fidelity $\mathcal{P}=(t,l,a_{\mathrm{ref}}),$6 in DINO and instruction adherence $\mathcal{P}=(t,l,a_{\mathrm{ref}}),$7 in CLIP-T. On MagicBrush editing test, it reports CLIP-I $\mathcal{P}=(t,l,a_{\mathrm{ref}}),$8 and CLIP-T $\mathcal{P}=(t,l,a_{\mathrm{ref}}),$9. On MIGEBench for instruction-based subject-driven editing, Subject Replacement reaches DINO $t$0, CLIP-I $t$1, and CLIP-T $t$2, while Subject Addition reaches DINO $t$3, CLIP-I $t$4, and CLIP-T $t$5. MCIE-E1, evaluated on CIE-Bench, reports CLIP-I $t$6, DINO-I $t$7, $t$8, $t$9, Instruction Compliance $l$0, and Background Consistency $l$1, with a $l$2 relative gain in IC over the best prior model (Tian et al., 28 Feb 2025, Bai et al., 8 Feb 2026).

InstructCIR frames retrieval as instruction following and reports substantial gains over prior zero-shot CIR baselines. CIRR Recall@1 reaches $l$3, CIRCO mAP@5 reaches $l$4, FashionIQ average R@10 reaches $l$5, and GeneCIS average R@1 reaches $l$6. CMI-RewardBench, by contrast, evaluates models rather than generators. It introduces CMI-Pref-Pseudo with $l$7 pseudo-labeled preference pairs, CMI-Pref with $l$8 expert-annotated pairs, and CMI-RewardBench with five evaluation tasks across four datasets. The associated CMI-RM architecture has approximately $l$9 M trainable parameters, uses frozen encoders, and is trained by a “distill $a_{\mathrm{ref}}$0 fine-tune” strategy. Reported results include SRCC $a_{\mathrm{ref}}$1 on MusicEval, musicality accuracy of $a_{\mathrm{ref}}$2 on the CMI-Pref test split, and $a_{\mathrm{ref}}$3 preference accuracy on full CMI with text+lyrics+audio, compared with $a_{\mathrm{ref}}$4 for Gemini3-Pro (Zhong et al., 2024, Ma et al., 28 Feb 2026).

6. Limitations, misconceptions, and adversarial implications

The literature repeatedly identifies supervision, decomposition quality, and preservation constraints as limiting factors. The ALFRED modular architecture requires supervised subgoal segmentations and labels at training time. COGS notes that subquestion decomposition via zero-shot prompting can be noisy and may miss rare factors, and that reinforcement learning with factor-level rewards demands substantial compute and careful KL-regularization to avoid overfitting to synthetic artifacts. MCIE-E1 reports failure modes on extremely small target objects $a_{\mathrm{ref}}$5 and on highly semantically overlapping sub-instructions. InstructCIR depends on high-quality instruction/caption pairs, and MIGE identifies integration with advanced object segmentation or pre-trained vision-LLMs as a possible extension (Corona et al., 2020, Gu et al., 16 Oct 2025, Bai et al., 8 Feb 2026, Zhong et al., 2024, Tian et al., 28 Feb 2025).

A further misconception is that compositional multimodality is beneficial only for capability improvement. DMN shows the same principle in an adversarial setting. It composes a jailbreak across three image sub-modules—Distributed Instruction, Multimodal Evidence, and Number Chain frames—while using one overarching text prompt. The target MLLM receives a sequence of instruction-split images, evidence images, and randomized number-chain frames, all fused with the text prompt. On SafeBench, HADES, and MM-SafetyBench, DMN reports attack success rates of over $a_{\mathrm{ref}}$6, over $a_{\mathrm{ref}}$7, and over $a_{\mathrm{ref}}$8, respectively; ablations report that Distributed Instruction alone raises ASR from $a_{\mathrm{ref}}$9 to $\mathcal{P}$0, adding Multimodal Evidence raises it to $\mathcal{P}$1, and interleaving Number Chains raises it to $\mathcal{P}$2 (Xu et al., 18 May 2026).

CMI therefore has a dual status in current research. On one side, it enables structured conditioning, cross-task transfer, factor-level supervision, and improved generalization on tasks involving generation, editing, retrieval, reasoning, and reward modeling. On the other side, the same compositional decomposition of intent across modalities can weaken safety mechanisms when the model must aggregate dispersed cues before recognizing the underlying objective.