Modality-switching Instruction Tuning (MosIT)

• MosIT is a training paradigm for multimodal LLMs that integrates explicit projections and adapters to align heterogeneous modalities.
• It enables any-to-any mapping, allowing unified processing of text, images, audio, video, and more under natural language instructions.
• It employs composite losses—including cross-entropy and diffusion reconstruction—to achieve robust cross-modal dialogue and positive transfer.

Modality-switching Instruction Tuning (MosIT) designates a training paradigm for multimodal LLMs (MLLMs) in which a single model is explicitly optimized to process instructions involving heterogeneous and dynamically varying input and output modalities. The aim is to endow a unified system with the ability to parse, reason over, and generate content conditioned on arbitrary combinations of modalities—including but not limited to text, image, audio, video, 3D point clouds—according to natural language instructions, and to generalize its instruction-following and cross-modal capabilities under high modality-switching pressure (Wu et al., 2023, Han et al., 2023, Zheng et al., 2024).

1. Formal Definition and Motivation

MosIT is characterized by training procedures in which every instruction-tuning instance comprises: (a) a free-form natural language instruction; (b) an arbitrarily chosen subset of modalities as input signals; and (c) target outputs that may be of any supported modality or mixture thereof. The theoretical motivation is to mimic human-like seamless cross-modal discourse—such as asking for an image from text, summarizing audio input as video, or reasoning across text, vision, and sound in a single dialogue.

While conventional MLLMs target multi-modal understanding (e.g., VQA) or conditional generation (e.g., image captioning), such models generally lack the architecture and data curation to support any-to-any (arbitrary input/output modality mapping) and multi-turn, modality-switching dialogue. Cascaded toolchains are insufficient, as they accumulate errors and lack holistic joint learning (Wu et al., 2023). MosIT instead addresses this by integrating explicit projections and adapters for modality alignment, and curates instruction-tuning corpora with fine-grained modality-switch instructions and outputs (Wu et al., 2023, Han et al., 2023).

2. Architectural Integration

MosIT is implemented atop MLLMs with modular fusion of pretrained modality-specific encoders, an LLM backbone, and (optionally) modality-specific decoders. The architectural core involves:

• Encoding Projections: Each modality encoder (e.g., ImageBind (Han et al., 2023), CLIP-ViT, Q-Former) provides a vectorial embedding which is mapped into the LLM input space via a learnable linear or MLP-based projection (e.g., $h_{\mathrm{m}} = W_p^{\mathrm{m}} x_{\mathrm{m}}$, where $x_{\mathrm{m}}$ is the modality embedding and $W_p^{\mathrm{m}}$ is the learned projection).
• Decoding Projections: For generative modalities (such as image, video, or audio outputs), the LLM emits special signal tokens (e.g., $<$IMG$>$, $<$VID$>$, $<$AUD$>$); their hidden representations are mapped via decoding projections ($W_o^m$) into the conditional embedding spaces required by diffusion-based decoders (e.g., Stable Diffusion, AudioLDM).
• Instruction Injection: Visual or generalized modality information is injected into the LLM at each transformer layer, typically via residual addition (e.g., $h_j' = h_j + g_l \cdot T_m$ in ImageBind-LLM, where $g_l$ is a trainable gate) (Han et al., 2023).
• Adapters and LoRA: Parameter-efficient fine-tuning is achieved by updating only the projection layers and lightweight adapter/LoRA subspaces, with all modality encoders and LLM backbone either frozen or sparingly updated (Wu et al., 2023).

This architecture allows for direct, joint learning of mappings across combinations of input and output modalities, and seamless switching at inference.

3. Training Objectives and Losses

MosIT employs a composite objective that simultaneously supervises text generation, modality-specific generation (e.g., through diffusion reconstruction), and instruction-to-signaling alignment. For a given instance, with trainable parameters $\theta_p$, $\theta_o$, and $\theta_{\mathrm{LoRA}}$:

• Text Cross-Entropy Loss: For text outputs—$$\mathcal{L}_{\mathrm{CE}} = - \sum_{t=1}^T \log p_\theta(y_t | y_{<t}, h_{\mathrm{src}}, \mathrm{instr})$$.
• Diffusion Reconstruction Losses: For each non-text modality $m$ required on output (e.g., image, video, audio), $$L_{\text{diff}}^m = \mathbb{E}_{t, \epsilon} \|\epsilon - \epsilon_\theta( z_t^m, c^m )\|^2$$, where $c^m = W_o^m z^m$ serves as the condition embedding for the respective decoder (Wu et al., 2023).
• Instruction-Signal Alignment: $L_{\text{align}}^m = \|W_o^m z^m - e^m_\star\|^2$, matching the projection of the token’s hidden state to the gold caption’s condition embedding (Wu et al., 2023).
• The total loss is a weighted sum:

$$\mathcal{L}_{\mathrm{MosIT}} = L_{\mathrm{CE}} + \sum_m \left( \lambda^m_{\text{diff}} L^m_{\text{diff}} + \lambda^m_{\text{align}} L^m_{\text{align}} \right)$$

with task-specific $\lambda$ coefficients balancing generation quality and multimodal alignment.

No hard-negative or contrastive objectives are needed at this stage for modalities, relying fully on supervised, alignment-driven transfer (Han et al., 2023).

4. Instruction-Tuning Datasets and Data Curation

Standard multimodal datasets lack coverage of arbitrary modality conversion and multi-turn, context-rich dialogues mixing text, vision, audio, and video. For MosIT, tailored corpora are assembled with the following properties (Wu et al., 2023):

• Any-to-Any Instruction Patterns: Manual design of diverse templates (e.g., "Describe this image then generate a matching melody") for cross-modal conversion in both input and output directions.
• Multi-turn Dialogue: Expansion of each template into synthetic dialogues comprising 3–7 turns, varying modality context and targets within each interaction.
• Exemplar Retrieval and Filtration: Retrieval of real content (images, audio, video) to match instructions, with manual vetting for relevance and quality.
• Dataset Scale: NExT-GPT’s MosIT corpus comprises approximately 5,000 dialogues, each averaging 4.8 turns, and spanning thousands of unique content items per modality (Wu et al., 2023).

ImageBind-LLM leverages large-scale image-text pairs (≈940M) for stage 1, and a mixture of image-instruction and language-instruction datasets for stage 2, including synthetic GPT-4 generated visual instructions (≈150K) and human-checked fine-tuning examples (3.5K) (Han et al., 2023).

5. Specialized Algorithmic Approaches for Modality Discrepancy and Continual Adaptation

A marked challenge in MosIT is distribution shift and knowledge interference across modalities and task types:

• Visual Cache Model: In ImageBind-LLM, a precomputed cache of image embeddings is used to anchor embeddings from modalities such as audio or video during inference, mitigating distribution discrepancy caused by purely image-driven training. This cache employs nearest-neighbor retrieval to blend new modality embeddings with image-derived embeddings (Han et al., 2023).
• Soft Prompt Pooling with Gradient Projection: Continual MosIT—where tasks or modalities are learned in sequence—is subject to catastrophic forgetting and negative forward transfer. Singular Value Decomposition (SVD) is applied to the embedding matrices at each task to define "core" and "residual" subspaces. Updates to soft prompts are projected into the residual (to avoid forgetting) and pretrained core (to promote positive forward transfer), achieving state-of-the-art performance, lowest forgetting, and positive transfer metrics on benchmarks (Zheng et al., 2024).
• Minimal Parameter Update: Only small projection layers, adapters, or prompts are tuned for MosIT, enabling low-cost adaptation and ease of extension to new modalities, with all foundation encoders/decoders remaining frozen (Wu et al., 2023, Han et al., 2023, Zheng et al., 2024).

6. Evaluation, Results, and Empirical Insights

MosIT-trained models demonstrate significant improvements in seamless cross-modal dialogue and generation:

• Any-to-Any Conversion: NExT-GPT achieves robust performance transferring text to image, image to audio, or video+text to multi-modal outputs in a single unified system. After MosIT, NExT-GPT improves COCO caption FID from 11.28 to 10.95, and human relevance ratings from 6.8 to 8.2 (out of 10) (Wu et al., 2023).
• Instruction Transfer Efficiency: Text-centric MosIT (as in MLAN) enables substantial zero-shot generalization to vision tasks; models trained with as little as 12.5% vision-language data achieve parity with vision-heavy tuning, and pure language-phase tuning already lifts vision held-out by 10.4% (Tu et al., 2024).
• Continual MosIT: Fwd-Prompt reduces forgetting and achieves positive forward transfer, with final multi-task InstructBLIP accuracy at 77.14% (Δ+4.17 over baselines), and forgetting score of 4.8 (compared to 9.3–21.0 for alternatives) (Zheng et al., 2024).

Limitations include lingering modality-specific artifacts (e.g., single-token conditioning’s inability to capture complex images), rare failure on unseen modality types (such as thermal images), and reliance on the representational overlap of pretrained and incoming modalities (Han et al., 2023, Zheng et al., 2024).

7. Implications and Open Challenges

MosIT establishes a principled paradigm for training and extending MLLMs that more closely approximate human modality fluidity. The paradigm unifies efficient adaptation, robust cross-modal dialogue, and positive transfer in a single architecture. Remaining challenges involve scaling toward more fine-grained modality sets (e.g., new sensor types), fine-tuning hierarchical instruction routing, mechanistically understanding inter-modal representational overlap, and automating SVD/residual hyperparameter choices (Zheng et al., 2024). A plausible implication is that the underlying techniques (projections, caches, gradient projection) will generalize toward future unified, open-world MLLMs.

System	Core MosIT Mechanism	Evaluation Modalities
NExT-GPT (Wu et al., 2023)	Paired encoding/decoding projections, LoRA	Text, Image, Video, Audio
ImageBind-LLM (Han et al., 2023)	Bind MLP, zero-gated residual, cache	Image, Audio, Video, 3D
Fwd-Prompt (Zheng et al., 2024)	Residual gradient projection (SVD)	Vision–language tasks

MosIT, as demonstrated across these frameworks, represents a critical evolution in both the instruction-tuning methodology and deployment of multimodal agents.