---
title: Multiple Foundation Model Mapper
url: https://www.emergentmind.com/topics/multiple-foundation-model-mapper-mfm-mapper
type: topic
---

# Multiple Foundation Model Mapper

Searching arXiv for the specified papers and closely related work on MFM-Mapper.
I found the primary paper "Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper" [2509.04957] and the related source-free domain adaptation paper "Collaborative Learning with Multiple Foundation Models for Source-Free Domain Adaptation" [2511.19147], which explicitly frames CoMA as a generalizable multiple-foundation-model mapper.
Searching for prior mapper-based video-to-audio work and the AudioLDM-2 lineage referenced by MFM-Mapper.
Relevant related work includes prior mapper-based V2A systems that connect a visual encoder to a text-to-audio generator, as well as AudioLDM-2-based conditioning schemes; the MFM-Mapper paper positions itself directly against that mapper lineage while replacing the linear mapper with a GPT-2-based autoregressive mapper and moving from CLAP-space conditioning toward AudioMAE conditioning [2509.04957].
Multiple Foundation Model Mapper (MFM-Mapper) designates a mapper-based strategy for exploiting several pretrained foundation models through a comparatively small trainable connector. In the literature, the term specifically refers to a video-to-audio generation system that links dual frozen visual foundation models to a frozen text-to-audio diffusion model via a fine-tuned GPT-2 mapper, with the aim of producing audio that is semantically consistent with video content and temporally synchronized with visual events while remaining highly data-efficient [2509.04957]. A broader interpretation is also suggested by later work on source-free domain adaptation, where a “multiple foundation model mapper” is realized as prediction-space alignment and knowledge transfer among heterogeneous foundation models and a task-specific model [2511.19147].

## 1. Definition and formal setting

In its canonical video-to-audio formulation, MFM-Mapper operates within a standard mapper-based pipeline. Given a silent video $\mathbf{V}$ of duration $T$, frames are sampled at $fr$ FPS to form
$$
\mathbf{V_f} \in \mathbb{R}^{T_f \times H \times W \times 3}, \quad T_f = fr \times T.
$$
A visual encoder $\mathcal{F}$ extracts embeddings
$$
\mathbf{V_e} = \mathcal{F}(\mathbf{V_f}), \quad \mathbf{V_e} \in \mathbb{R}^{T_e \times D_e},
$$
which are then mapped by $\mathcal{M}$ to conditioning features
$$
\mathbf{C} = \mathcal{M}(\mathbf{V_e}), \quad \mathbf{C} \in \mathbb{R}^{T_c \times D_c},
$$
for a pretrained audio generator $\mathcal{G}$, yielding
$$
\mathbf{X} = \mathcal{G}(\mathbf{C}).
$$
During training, $\mathcal{F}$ and $\mathcal{G}$ are frozen and only $\mathcal{M}$ is trained [2509.04957].

The central motivation is that training video-to-audio systems from scratch requires large-scale audio-video datasets and expensive cross-modal pretraining. By freezing foundation models and learning only a connector, the system can drastically reduce training data and compute, inherit rich semantic and temporal knowledge from pretrained models, and adapt quickly to the task. The named MFM-Mapper extends earlier single-encoder mapper designs

Source: https://www.emergentmind.com/topics/multiple-foundation-model-mapper-mfm-mapper