---
title: Multimodal-to-Text Generation Models
url: https://www.emergentmind.com/topics/multimodal-to-text-generation-models
type: topic
---

# Multimodal-to-Text Generation Models

Multimodal-to-text generation models are neural architectures that map heterogeneous multimodal signals—such as images, video, audio, structured data, or sequences thereof—to textual outputs drawn from natural language. Unlike unimodal text generation, these models must simultaneously encode, align, and fuse information from two or more distinct input modalities within a unified generative process. Advances in this area are central to tasks such as image-to-text captioning, video-based dialog, music captioning, experience-driven story and lyric generation, multimodal citation-grounded summarization, and more. This article surveys fundamental principles, contemporary architectures, training methodologies, evaluation paradigms, and open challenges for multimodal-to-text generation, referencing recent academic contributions.

## 1. Foundational Architectures and Modal Alignment

Multimodal-to-text generation models can be organized into several major design paradigms depending on how they process and align inputs:

**a. Unified Token Stream Transformers:**  
Transformer-based models increasingly adopt a joint tokenization strategy, where visual, textual, and, more recently, music or motion inputs are quantized into shared discrete token streams for early fusion. For example, ANOLE [2407.06135] and the unified model in [2110.09753] (building on Chameleon and LXMERT/X-LXMERT, respectively) concatenate image and text tokens (with modality markers) into a single autoregressive sequence processed by a standard Transformer decoder. This enables native multimodal-to-text and text-to-image generation via shared embeddings, causal self-attention, and a unified language modeling objective.

**b. Modality-specific Encoding and Prefixing:**  
Other models employ modality-specific encoders (e.g., ResNet for vision, ViT for visual tokens, or audio classifiers), project all encodings to a common language embedding space, and concatenate them as a conditional prefix for a pretrained or jointly-trained text decoder. This conditionality prefix approach exemplified by MAnTiS [2109.01229] and the video-centric Vx2Text [2101.12059], reduces architectural complexity by keeping the downstream decoder unchanged and leveraging language modeling advances directly.

**c. Multi-stream and Experience-aware Architectures:**  
Innovations such as Multi-Modal Experience Inspired AI Creation (MMTG) [2209.02427] extend the paradigm to ordered, sequential experiences: image–text pairs observed over time are processed by multi-stream encoder–decoders using cross-modal, temporal, and attention mechanisms that explicitly model the dependencies between input sequence order and generated narrative. The architectural design fuses each experience’s latent representation over the set of output sentences using a two-dimensional attention with a temporal regularizer, enabling sensitive modeling of narrative flow.

**d. Shared Generative-Embedding Models:**  
Models such as MM-GEM [2405.19333] demonstrate that a single LLM backbone with lightweight projection adapters and a PoolAggregator can simultaneously solve both embedding and generative (captioning) objectives, including at fine-grained region level, without significant conflict or capacity interference between tasks.

## 2. Training Objectives and Multimodal Fusion Methods

Most multimodal-to-text generation models pursue autoregressive maximum-likelihood training over the textual output sequence, often conditioned on learned or projected representations of one or more non-text inputs. The following mechanisms are prevalent:

- **Cross-entropy Language Modeling:**  
The standard loss is the negative log likelihood of the ground-truth tokens given the multimodal context (e.g., Stage 3 GPT-2/StoryGen [2001.10980]; text head in TextHarmony [2407.16364]; Stage 3 T5-style captioning in UniMuMo [2410.04534]).

- **Contrastive and Joint Objectives:**  
Architectures such as MM-GEM [2405.19333] interleave contrastive embedding losses (image–text, text–image) with cross-entropy captioning loss, yielding a single training signal:
  $$
  \mathcal{L}_{\text{MM-GEM}} = \mathcal{L}_{\text{emb}} + \mathcal{L}_{\text{gen}}
  $$
where embedding and generative heads are decoupled with lightweight adapters but share the main LLM.

- **Attention and Fusion Schemes:**  
Unified token streams enable early-fusion cross-modal attention (e.g., ANOLE, [2110.09753]), while more traditional models use cross-modal fusion via prefixing ([2109.01229], MAnTiS), or concatenation after projection (Vx2Text). Two-level granularity—combining dense (continuous) and discrete features—enhances semantic coverage [2110.09753].

- **Dynamic Expert and Modality Selection:**  
TextHarmony [2407.16364] employs Slide-LoRA—multiple parallel low-rank adaptation modules in Transformer layers gated by a dynamically-learned coefficient—to enable parameter-efficient, dynamically adaptable separation of text- and vision-specific generation without sacrificing the benefits of unified modeling.

## 3. Modalities Addressed: Images, Video, Music, and Beyond

As the field expands, modern architectures move well beyond static image captioning:

- **Video + Audio + Text:**  
Vx2Text [2101.12059] processes video frames, raw audio, speech transcripts, and dialog history as streams of tokens for generative video-based dialog and QA. Differentiable tokenization via Gumbel–Softmax enables end-to-end training, with all fusion performed in the language space.

- **Sequential Multimodal Experiences:**  
MMTG [2209.02427] attacks the problem of generating poetry or lyrics based on a temporally ordered sequence of multimodal experiences, explicitly encoding both sequence and modality dependencies, and employs a curriculum negative sampling strategy for efficient optimization.

- **Music and Motion:**  
UniMuMo [2410.04534] generalizes token-based fusion to music and motion. Both music and motion are encoded and quantized via a shared codebook, enabling a single Transformer architecture to process and generate music, motion, and text, including cross-modal captioning (music→text, motion→text).

- **Document/Scientific Multimodal:**  
MAnTiS [2109.01229] and MCiteBench [2503.02589] bring multimodal-to-text generation to complex contexts such as product descriptions (image+title) and scientific discourse (image, table, paragraph, etc. with citation grounding).

## 4. Controllability, Faithfulness, and Attribution

Robust multimodal-to-text generation must provide mechanisms for both controllability and faithfulness:

- **Plug-and-play Multimodal Controllability:**  
ZeroGen [2306.16649] introduces a zero-shot paradigm wherein both token-level (textual oracle, e.g. GloVe similarity) and sentence-level (multimodal oracle, e.g. CLIP image-text alignment) controls are imposed directly at decoding, without additional training. Dynamic control weights at each time step calibrate the influence of content-specific and modality-specific signals, yielding both high-quality captioning and attribute-controlled news generation.

- **Faithful and Salient Generation:**  
A hybrid critic–generator framework [2409.03961] augments LMMs with a vision critic (BLIP-2 + LoRA) that categorizes draft text features as salient, non-salient, or hallucinated, pruning non-grounded content and appending missing salient features in post-editing. This yields significant improvements in BLEU, METEOR, ROUGE, BERTScore, and CLIPScore, and is essential for domains requiring faithful grounded descriptions, such as advertising.

- **Citation and Attribution in Multimodal Scientific QA:**  
MCiteBench [2503.02589] evaluates MLLMs' abilities to generate answer text with fine-grained, per-sentence citations to multimodal evidence (text, figures, tables). Analyses reveal a strong textual modality bias; even large models under-attend to visual content when assigning citations, resulting in lower attribution fidelity for visual versus textual evidence.

## 5. Datasets, Evaluation, and Benchmarking

Model capabilities are measured on diverse datasets and by a range of metrics:

- **Generation Benchmarks:**  
MS-COCO (image captioning, BLEU-4, CIDEr, SPICE) [2110.09753, 2405.19333], Flickr8K/Flickr30k [2001.10980, 2405.19333], MusicQA [2410.04534], HumanML3D (motion captioning) [2410.04534], FACAD (fashion captioning) [2109.01229].

- **Faithfulness and Saliency:**  
CLIPScore (image–text alignment), BERTScore, BLEU, METEOR, ROUGE, custom saliency matching [2409.03961].

- **Citation Quality:**  
MCiteBench measures Citation F₁, Source Reliability (F₁, Exact Match), and LLM-judged answer accuracy.  
Experience-aware lyric/story generation models are further evaluated by metrics capturing output diversity and order-sensitivity (NNR), and by detailed human relevance/coherence/meaning/overall panel ratings [2209.02427].

- **Region-level Fine-grained Evaluation:**  
MM-GEM [2405.19333] supports region-level captioning and retrieval, measured by Recall@1 on DCI and R-Precision (for HumanML3D).

## 6. Practical Considerations, Limitations, and Future Directions

**Parameter and Computational Efficiency:**  
Recent models demonstrate parameter efficiency via modular adapters (Slide-LoRA [2407.16364]), head-only fine-tuning (ANOLE [2407.06135]), and multi-stage minimal retraining for both global and region-level tasks (MM-GEM [2405.19333]).

**Limitations:**
- Modular, multi-stage pipelines (as in StoryGen [2001.10980]) are not end-to-end differentiable, leading to error propagation.
- Token-based models can struggle with fine spatial or temporal alignment (either visual or in music/motion streams [2410.04534]).
- Faithful grounding and citation in the presence of numerous distractors and modality-mismatch remains unsolved (MCiteBench [2503.02589]).
- Visual-text fusion is still subject to textual grounding bias [2503.02589].

**Areas for Advancement:**
- End-to-end unimodal and multimodal fusion; explicit cross-modal attention for deeper joint representations.
- Better alignment of music, motion, and text features (UniMuMo [2410.04534]).
- Unified expert mechanisms to minimize destructive cross-modal interference (TextHarmony [2407.16364]).
- Inclusion of additional modalities (audio, structured data, code [2409.03961, 2503.02589]).
- Direct reinforcement or contrastive objectives for more robust cross-modal grounding [2409.03961, 2503.02589].

## 7. Comparative Summary and Research Outlook

The table below summarizes representative models and distinguishing features:

| Model         | Modalities          | Fusion/Tokenization      | Faithfulness/Control      | Evaluation Highlights          |
|---------------|--------------------|-------------------------|--------------------------|-------------------------------|
| ANOLE [2407.06135]      | Image, Text         | Shared token stream        | Head-only image tuning         | Qualitative, interleaved gen. |
| MM-GEM [2405.19333]     | Image, Text         | PoolAggregator + shared LLM| Embedding+gen dual obj.        | CIDEr 110.9 (COCO), region R@1|
| Vx2Text [2101.12059]    | Video, Audio, Text  | Differentiable tokenizer   | Joint caption/QA/dialog        | SOTA, AVSD/TVC/TVQA           |
| TextHarmony [2407.16364]| Images, Text        | Slide-LoRA, shared LLM     | Dynamic MoE expert gating      | OCRBench, NED, FID, CLIP      |
| ZeroGen [2306.16649]    | Image, Text         | Decoding-time oracles      | Plug-and-play control          | B@4 15.5, CIDEr 55.4           |
| MMTG [2209.02427]       | Seq. img+text       | 2D cross-modal attention   | Curriculum negative sampling   | BLEU-2 0.076, BERTScore 0.595  |
| UniMuMo [2410.04534]    | Music, Motion, Text | Shared codebook+Transformer| Parallel music-motion gen      | BLEU@1 0.261, MMDist 2.958     |
| MCiteBench [2503.02589] | Text, Fig., Table   | Multimodal encoding        | Citation F₁, Source EM         | C-F₁ ∼0.84 (text), EM <0.1 (fig)|

Ongoing research emphasizes the need for unified architectures offering flexible, faithful, and controllable multimodal-to-text generation, robust to modality imbalance and capable of fine-grained grounding with attribution, as well as end-to-end differentiable training and efficient fine-tuning. Emerging benchmarks—such as MCiteBench—will be critical for rigorous, modality-aware evaluation and for driving progress on fine-grained grounding in real-world scientific and technical applications.

Source: https://www.emergentmind.com/topics/multimodal-to-text-generation-models