Multiple Foundation Model Mapper

• The paper demonstrates a novel strategy for video-to-audio generation by fine-tuning a GPT-2-based autoregressive mapper that links multiple frozen foundation models.
• MFM-Mapper reduces training data and computational demands by freezing both visual and audio generators while training only the connector to ensure semantic and temporal alignment.
• The system extends to source-free domain adaptation by aligning prediction spaces among heterogeneous foundation models and task-specific models.

Searching arXiv for the specified papers and closely related work on MFM-Mapper. I found the primary paper "Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper" (Chen et al., 5 Sep 2025) and the related source-free domain adaptation paper "Collaborative Learning with Multiple Foundation Models for Source-Free Domain Adaptation" (Lee et al., 24 Nov 2025), which explicitly frames CoMA as a generalizable multiple-foundation-model mapper. Searching for prior mapper-based video-to-audio work and the AudioLDM-2 lineage referenced by MFM-Mapper. Relevant related work includes prior mapper-based V2A systems that connect a visual encoder to a text-to-audio generator, as well as AudioLDM-2-based conditioning schemes; the MFM-Mapper paper positions itself directly against that mapper lineage while replacing the linear mapper with a GPT-2-based autoregressive mapper and moving from CLAP-space conditioning toward AudioMAE conditioning (Chen et al., 5 Sep 2025). Multiple Foundation Model Mapper (MFM-Mapper) designates a mapper-based strategy for exploiting several pretrained foundation models through a comparatively small trainable connector. In the literature, the term specifically refers to a video-to-audio generation system that links dual frozen visual foundation models to a frozen text-to-audio diffusion model via a fine-tuned GPT-2 mapper, with the aim of producing audio that is semantically consistent with video content and temporally synchronized with visual events while remaining highly data-efficient (Chen et al., 5 Sep 2025). A broader interpretation is also suggested by later work on source-free domain adaptation, where a “multiple foundation model mapper” is realized as prediction-space alignment and knowledge transfer among heterogeneous foundation models and a task-specific model (Lee et al., 24 Nov 2025).

1. Definition and formal setting

In its canonical video-to-audio formulation, MFM-Mapper operates within a standard mapper-based pipeline. Given a silent video $\mathbf{V}$ of duration $T$, frames are sampled at $fr$ FPS to form

$$\mathbf{V_f} \in \mathbb{R}^{T_f \times H \times W \times 3}, \quad T_f = fr \times T.$$

A visual encoder $\mathcal{F}$ extracts embeddings

$$\mathbf{V_e} = \mathcal{F}(\mathbf{V_f}), \quad \mathbf{V_e} \in \mathbb{R}^{T_e \times D_e},$$

which are then mapped by $\mathcal{M}$ to conditioning features

$$\mathbf{C} = \mathcal{M}(\mathbf{V_e}), \quad \mathbf{C} \in \mathbb{R}^{T_c \times D_c},$$

for a pretrained audio generator $\mathcal{G}$, yielding

$\mathbf{X} = \mathcal{G}(\mathbf{C}).$

During training, $T$0 and $T$1 are frozen and only $T$2 is trained (Chen et al., 5 Sep 2025).

The central motivation is that training video-to-audio systems from scratch requires large-scale audio-video datasets and expensive cross-modal pretraining. By freezing foundation models and learning only a connector, the system can drastically reduce training data and compute, inherit rich semantic and temporal knowledge from pretrained models, and adapt quickly to the task. The named MFM-Mapper extends earlier single-encoder mapper designs