NaiveMC: Training-Free MLLM Composition

• NaiveMC is a training-free baseline that composes specialized modality encoders with a shared large language model using parameter-wise averaging.
• It avoids the high cost and extensibility issues of joint multimodal training by reusing separate, instruction-tuned branch expertise.
• Empirical evaluations demonstrate that NaiveMC outperforms several contemporaries while highlighting challenges like parameter interference in scaling modalities.

NaiveMC is a training-free model composition baseline for multimodal LLMs (MLLMs) that composes existing single-modality or limited-modality MLLMs into one more versatile MLLM without additional joint multimodal training (Chen et al., 2024). Its core design is to retain the modality-specific encoders and connectors of separately trained source models, attach them to a shared LLM backbone, and, when the source LLMs were fine-tuned, merge the LLM parameters by parameter-wise arithmetic averaging. Within the broader model-composition paradigm introduced in "Model Composition for Multimodal LLMs," NaiveMC is the foundational realization that establishes the feasibility of composition, while DAMC is presented as an interference-aware refinement.

1. Problem setting and motivation

NaiveMC is designed to address the high cost and poor extensibility of standard multimodal LLM construction. The usual pipeline for an MLLM requires first aligning a modality encoder to an LLM with modality-text paired data, and then instruction-tuning with modality-specific or jointly multimodal instruction data. If a single model is expected to handle image, audio, video, and point cloud jointly, the common solution is to train all modality pipelines against the same LLM using paired multimodal instruction data. The paper characterizes this as expensive because synchronized multimodal instruction data are difficult to collect, and because extending the system to a new modality typically forces more training (Chen et al., 2024).

The composition paradigm formalizes the setting as follows. Given MLLMs

$\{M_1, M_2, \ldots, M_n\},$

where each model understands one modality $m_i$ plus text, a composition method $C$ produces

$M_{\text{compose}} = C(M_1, M_2, \ldots, M_n),$

such that the composed model can process inputs from any combination of modalities in

$\mathbf{m} = \{m_1, m_2, \ldots, m_n\}.$

Each source MLLM is assumed to contain modality-specific components—its modality encoder and connector or projector—and a modality-agnostic component, namely the underlying LLM backbone. NaiveMC instantiates the composition principle that all modality-specific components are retained, attached to a shared LLM, and that the LLM is either directly reused or merged, depending on whether it remained frozen during source-model training.

The method is positioned against several alternatives. Methods such as PandaGPT or ImageBind-LLM cover multiple modalities by relying on a unified encoder such as ImageBind, but the paper argues that their performance is limited because they do not use enough modality-specific instruction data. Joint-training methods with multiple encoders can be stronger, but require costly paired multimodal instruction data such as synchronized video-audio chat data. X-InstructBLIP is closer in spirit to composition, but its applicability is restricted to frozen-LLM settings. NaiveMC is motivated by the observation that MLLMs initialized from the same base LLM can be composed zero-shot even when the LLM parameters were fine-tuned during each model’s own training.

2. Composition mechanism and architecture

NaiveMC composes separately trained image, audio, video, and point cloud MLLMs, all based on Vicuna-7B-v1.5. For each source model $M_i$, it reuses the modality encoder, the connector or projector into the LLM space, and the modality-processing interface to the LLM. The implementation examples are explicit: the image MLLM uses a CLIP-ViT-L-336px encoder with an MLP projector; the audio MLLM uses a BEATs-Iter3+ encoder with a Q-Former with 32 query tokens; the video MLLM uses a LanguageBind video encoder with an MLP projector; and the point cloud MLLM uses a point encoder with an MLP projector (Chen et al., 2024).

The merge logic is simple. Parameters unique to a single model are copied directly into the composed model, while parameters common across multiple models are averaged groupwise. In practice, modality-specific parameters are copied, and shared LLM parameters are merged by parameter-wise arithmetic averaging: $$\theta_{\text{merge}}=\frac{1}{N}\sum_{i=1}^{N}\theta_i.$$ Equivalently, the composed model can be summarized as

$$M_{\text{compose}}= \left( \bigcup_i \text{modality encoder}_i,\; \bigcup_i \text{projector}_i,\; \frac{1}{N}\sum_{i=1}^{N}\Theta^{\text{LLM}_i} \right),$$

where $\Theta^{\text{LLM}_i}$ denotes the LLM parameters in source model $M_i$. This is the unweighted special case later generalized by DAMC.

The paper distinguishes two NaiveMC cases. If the source MLLMs were trained with frozen LLMs, no nontrivial merge is required because all LLM backbones remain equal to the common initialization; NaiveMC simply copies the modality-specific encoders and projectors and connects them to the unchanged pretrained LLM. If the source MLLMs used trainable or fine-tuned LLMs, NaiveMC copies all modality branches and averages corresponding LLM parameters across models.

After composition, the architecture contains one input branch per modality. Raw input is encoded by the modality encoder, transformed by the corresponding projector or connector, inserted into the shared LLM’s input sequence as modality tokens or embeddings, and then jointly processed with text tokens by the single shared LLM. The paper emphasizes that NaiveMC introduces no extra routing network, gating module, learned fusion block, or extra post-composition training objective. Its defining characteristic is precisely to retain the source modality branches, merge the LLM, and run inference.

3. Preconditions and source-model preparation

NaiveMC depends on a specific compatibility regime. The most important explicit assumption is common LLM initialization: the source MLLMs must be initialized from the same LLM. Parameter-wise averaging also presupposes the same parameter topology for the LLM, so that corresponding tensors have identical shapes and semantic roles. The design further assumes a compatible hidden interface to the LLM, since each modality connector or projector must output features in the same embedding space expected by the shared LLM, and a compatible tokenizer and text embedding convention, since all source models share the same Vicuna backbone (Chen et al., 2024).

A further implicit requirement is that the encoder-LLM interface be preserved. Each modality-specific connector must already have been trained to inject modality features into the same base LLM. NaiveMC does not relearn this alignment after composition. These constraints explain why the experiments focus on homologous MLLMs built from the same backbone rather than arbitrary multimodal systems.

Although NaiveMC itself is training-free at composition time, its performance depends directly on the preparation of the source models. All source MLLMs use Vicuna-7B-v1.5 and LoRA for efficient LLM adaptation, with LoRA applied to all linear modules in the LLM, rank $m_i$0, and $m_i$1. The source branches are trained separately with modality-specific alignment and instruction data. The paper gives examples: image uses LCS 558K and then LLaVA-mixed 665K; audio uses WaveCaps 400K and then filtered OpenAQA 350K; video reuses Video-LLaVA stage-1 weights and uses Video-ChatGPT 100K plus sampled LLaVA-mixed for stage 2; point cloud uses PointLLM data. The practical implication is that NaiveMC reuses already instruction-tuned modality expertise rather than learning multimodal coordination from a new composition dataset.

4. Empirical evaluation

The evaluation covers a new benchmark, MCUB, and four downstream multimodal understanding tasks. MCUB measures multimodal commonality understanding through 4-choice questions. MCUB-3 uses combinations of three modalities among image, audio, video, and point cloud, while MCUB-4 uses all four modalities. Additional tasks are audio-visual question answering on MUSIC-AVQA and AVQA, and 3D object classification on ModelNet40 and Objaverse. The paper reports percentage accuracy or benchmark scores directly; MCUB is reported as percentage accuracy over multiple-choice questions (Chen et al., 2024).

NaiveMC’s main empirical role is as a proof of concept for training-free composition. On MCUB it substantially outperforms ImageBind-LLM, X-InstructBLIP, and Proj-only, while remaining below DAMC. On downstream tasks it is consistently stronger than Proj-only and typically improves as modalities are added, although DAMC retains an advantage.

Evaluation	NaiveMC	Selected comparison
MCUB-3 / MCUB-4	54.70 / 54.03	ImageBind-LLM 32.95 / 32.93; X-InstructBLIP 29.30 / 27.94; Proj-only 44.15 / 43.00; DAMC 59.80 / 60.08
MUSIC-AVQA, $m_i$2	53.63	Proj-only 50.21; DAMC 57.32
AVQA, $m_i$3	80.70	Proj-only 66.85; DAMC 81.31
Objaverse, Type-I / Type-C	55.00 / 59.50	DAMC 60.50 / 62.00
ModelNet40, Type-I / Type-C	66.00 / 64.59	DAMC 70.02 / 65.24

The detailed MCUB-3 subtask results for NaiveMC are 56.00 for $m_i$4, 51.00 for $m_i$5, 53.00 for $m_i$6, and 58.80 for $m_i$7. On MUSIC-AVQA, NaiveMC scores 49.00 on $m_i$8, 52.52 on $m_i$9, 50.66 on $C$0, and 53.63 on $C$1. On AVQA, it scores 79.37 on $C$2, 79.74 on $C$3, 79.82 on $C$4, and 80.70 on $C$5. The appendix also reports point-cloud-only results, where NaiveMC reaches 56.00 and 54.50 on Objaverse depending on prompt type, and 68.59 and 58.79 on ModelNet40.

These results support the paper’s claim that preserving trainable-LLM modality expertise and merging it is better than merely attaching multiple projectors to a frozen LLM. They also show that training-free composition is already a strong baseline before interference-aware refinements are introduced.

5. Strengths, extensibility, and limitations

NaiveMC’s principal strengths are practical. It requires no additional training after composition, no paired multimodal instruction data, and no composition dataset. It reuses existing specialized MLLMs, is easily extensible to new modalities, and works even when the source LLMs were fine-tuned, unlike methods restricted to frozen-LLM settings (Chen et al., 2024). Within the constraints of the shared-backbone setup, the method offers a data-efficient route to multimodal expansion: to add a new modality, one can train a new modality-specific MLLM on top of the same base LLM, add its encoder and projector branch, and merge the LLM with the existing composite model.

The limitations are equally explicit. The central problem is parameter interference: averaging fine-tuned LLM weights from different modality-specialized models can damage specialized behaviors. The paper also identifies parameter mismatch, since equal averaging treats all source MLLMs as equally reliable even when their data quality, instruction-tuning strategy, and final performance differ. Scaling to more modalities can intensify these issues; the paper notes that NaiveMC tends to degrade on MME as more modalities are merged, which is consistent with interference. Generality is also constrained by engineering compatibility: the models must share the same LLM initialization, tokenizer, architecture, hidden sizes, and aligned projector interface. Finally, the empirical scope is limited to four modalities and 7B-scale backbones.

Ablation results reinforce these limitations. The base no-decoupling, no-adjustment setting—essentially NaiveMC—has an average score of 62.79, while decoupling only yields 65.05, adjustment only yields 63.55, and combining both yields 66.24. A separate ablation shows that simply increasing trainable parameter count does not resolve the shortcoming: DAMC with $C$6 scores 57.32 on MUSIC-AVQA and 81.31 on AVQA, whereas NaiveMC scores 53.63 and 80.70 at $C$7, and 52.79 and 80.81 at $C$8. The paper’s interpretation is that the core problem is indiscriminate merging rather than insufficient parameter count.

6. Relation to DAMC and subsequent significance

NaiveMC is the unweighted, non-decoupled baseline from which DAMC departs. DAMC extends NaiveMC in two directions. First, it introduces parameter decoupling: instead of sharing one set of LLM parameters for both modality and text tokens during source-model training, it separates modality-specific and text-related parameters inside attention and FFN layers, and at composition time merges only the text-related parameters while keeping modality-specific parameters distinct. Second, it introduces adaptive parameter adjustment, replacing uniform averaging with

$C$9

where the coefficients $M_{\text{compose}} = C(M_1, M_2, \ldots, M_n),$0 are selected by search over

$M_{\text{compose}} = C(M_1, M_2, \ldots, M_n),$1

using a validation set or general per-modality performance (Chen et al., 2024).

This contrast clarifies NaiveMC’s significance. Its role is not to solve all composition problems, but to show that model composition for MLLMs works at all. In the paper’s narrative, NaiveMC demonstrates that one can build a unified multimodal LLM without costly joint multimodal instruction tuning by reusing each source model’s modality encoder and connector, sharing a common LLM backbone, and directly averaging fine-tuned LLM parameters when needed. DAMC then addresses the interference and mismatch that NaiveMC exposes.

The broader significance follows from that ordering. NaiveMC establishes a new composition-centered alternative to end-to-end joint multimodal training: a model can inherit modality understanding from separately trained branches and remain operational without composition-time optimization. A plausible implication is that the method reorients multimodal extensibility from synchronized-data collection toward compatibility engineering around a shared backbone. Within the paper, however, the final conclusion remains narrower and more technical: NaiveMC is the foundational training-free baseline, and DAMC is the refinement that makes the same paradigm work better.