---
title: 'MALM: Modular Multi-Information Adapters'
url: https://www.emergentmind.com/topics/multi-information-adapters-malm
type: topic
---

# MALM: Modular Multi-Information Adapters

Multi-Information Adapters (MALM) are a class of modular, parameter-efficient neural architecture components used to facilitate transfer, multitask learning, domain adaptation, and knowledge integration in both vision and language settings. These adapters extend pre-trained transformers or similar models by introducing lightweight, learnable units that specialize for given tasks, sources of information, or domains, and can be flexibly combined, stacked, or merged. Cross-disciplinary instantiations—ranging from multi-LoRA merges in vision [2411.14064] through multi-graph-attentional mitigation of LLM hallucinations [2506.12483] to compositional multilingual/domain adaptation in neural translation [2110.09574]—demonstrate MALM's adaptability and the unifying principle of multi-information injection.

## 1. Foundational Concepts and Architectures

MALM generalizes the adapter principle: selectively parameterized, low-dimensional or bottleneck modules are inserted into pre-trained backbone architectures, enabling efficient adaptation to tasks or domains with minimal full-model retraining. Common design patterns include:

- **Vision Transformer Adapters**: LoRA-style modules are injected into the key and value projection matrices of a frozen Vision Transformer (ViT) base. The low-rank update is parameterized as $W = W_0 + AB^\top$ with $A,B \in \mathbb{R}^{d\times r}$, $r \ll d$ [2411.14064].
- **LLM Hallucination Adapter**: For autoregressive language models, MALM is constructed as a plug-in between the final transformer block and output head. It processes (a) input queries, (b) context (partial outputs), (c) external knowledge, via multi-graph input to a graph attention network (GAT), outputting reweighted logits fused with the LLM [2506.12483].
- **NMT Multilingual/Domain Adapters**: Adapters are stacked per transformer sublayer, with separate parameters for language (LA) and domain (DA). Each consists of bottleneck projections and norm layers, and can be composed in encoder, decoder, or both, yielding fine-grained control over adaptation [2110.09574].

## 2. Adapter Merging, Composition, and Interaction

A core attribute of MALM is the ability to combine information from multiple adapters. This can occur via:

- **Linear (Summation) Merging**: For $N$ LoRA adapters, $W_{\rm merged} = W_0 + \sum_{i=1}^N \alpha_iA_iB_i^\top$; in practice, uniform weights $\alpha_i=1$ are used [2411.14064].
- **Concatenation Merging**: Adapters' low-rank matrices are concatenated, producing $A_{\rm cat} = [A_1\;\cdots\;A_N]$ and $B_{\rm cat} = [B_1\;\cdots\;B_N]$, so $W_{\rm merged} = W_0 + A_{\rm cat}B_{\rm cat}^\top$, which expands to a sum of all self and cross terms [2411.14064]. Empirically, cross-terms do not catastrophically degrade multitask performance.
- **Multi-Graph Fusion for LLMs**: Input, context, and knowledge are separately encoded, with their token representations interacting through a heterogeneous GAT as nodes in a multi-type edge graph, allowing the adapter to model complex dependencies between information sources [2506.12483].
- **Stacking for NMT**: Language and domain adapters can be sequentially stacked within each transformer layer, or restricted to encoder/decoder sides to control interference and transfer [2110.09574].

Adapter merging strategies determine the degree of parameter and inference sharing, the extent of information cross-contamination, and computational overhead.

## 3. Training, Inference, and Objective Functions

MALM relies on modular training with the following regime:

- **Single-Adapter Training**: Each LoRA or adapter module is trained independently on its designated task or domain, leaving the backbone frozen. The ViT-based vision adapters use cross-entropy (classification) or $L_1$ loss (regression) [2411.14064], while LLM-based adapters use standard next-token cross-entropy [2506.12483]. For NMT, adapters are trained with negative log-likelihood on (possibly synthetic) parallel text [2110.09574].
- **Merging and Inference**: After training, adapters can be merged without retraining. For task-inference, this greatly speeds up evaluation: a merged LoRA for $N$ tasks requires a single backbone pass ($O(d\,r\,N)$ per projection) instead of $N$ sequential passes [2411.14064].
- **Zero-shot and Domain Adaptation**: In NMT, training only DAs on a new domain in a subset of languages and leveraging back-translation for unobserved pairs enables robust cross-lingual transfer [2110.09574].
- **No Additional Losses**: LLM MALM does not require hallucination-specific or auxiliary losses; the adapter’s architectural graph constraints suffice to reduce hallucination [2506.12483].

## 4. Empirical Results, Performance, and Trade-offs

MALM’s performance—quantified over multi-task, faithfulness, and adaptation metrics—depends on task similarity, adapter architecture, and merging strategy. Direct findings include:

- **Vision Adapters**: Merged LoRA adapters for tasks with dissimilar data, such as FireRisk and Galaxy10, demonstrate F1 retention within $1$ point of single-task (e.g., LoRA-64, F1 $\approx$ 75.9 vs. 76.7), but similar-domain merges (e.g., UTKFace classification and regression) cause substantial accuracy degradation (RMSE increase from $0.868$ to $1.268$) [2411.14064].
- **LLM Hallucination Mitigation**: MALM-adapted LLMs achieve +2–130% relative improvement in ROUGE-2 and set SOTA on faithfulness (FEQA) across HaluEval, TruthfulQA, NQ, and TriviaQA. Expert preference rates are high: 79.4% (GPT-4) and 65.6% (human) outcome selection favoring MALM [2506.12483].
- **NMT Domain Adaptation**: In full-resource multilingual adaptation, language+domain adapters achieve BLEU within 1–2 points of domain-tagged full-finetuning baselines (e.g., LA+DA combo: BLEU 42.7 vs. 46.0, Table 3), while in partial-resource settings, encoder- or decoder-only DAs with back-translation give gains of $3$–$4$ BLEU (e.g., 36.9 achieved for unseen directions) [2110.09574].
- **Computational Cost**: Merging adapters leads to linear inference time reductions, requiring just a single forward pass for $N$ tasks compared to unmerged deployment [2411.14064].

Below is a table showcasing the adapter domains and key empirical outcomes:

| Setting                | Adapter Strategy       | Main Metrics               | Key Results         |
|------------------------|-----------------------|----------------------------|---------------------|
| Vision (ViT+LoRA)      | Concatenation Merge   | Accuracy, F1, RMSE, NME    | Near-single-task for dissimilar; degradation for similar tasks [2411.14064] |
| LLM Hallucination      | Graph-attention MALM  | ROUGE, FEQA, Human Eval    | SOTA on faithfulness, large non-hallucination rate gains [2506.12483]      |
| NMT Multilingual/Domain| LA + (encoder/decoder) DA; DADropout; BT | BLEU, off-target rate | +3–4 BLEU on unseen, off-target rate down from 23% → 6% [2110.09574]      |

## 5. Limitations and Diagnostic Findings

MALM demonstrates strong adaptability but is subject to several constraints:

- **Performance Drop Upon Merging**: Across vision and translation, merging adapters for similar domains induces larger degradation due to overlapping information interfering (cross-terms in $A_iB_j^\top$) [2411.14064], [2110.09574].
- **Catastrophic Forgetting**: In NMT, naïve stacking of language and domain adapters for unseen language/domain combinations causes BLEU to drop below $20$, with high off-target rate [2110.09574].
- **No Dynamic Weighting**: Current merging schemes use uniform weighting; adaptive or learned merging coefficients are left to future work [2411.14064].
- **Scope of Hallucination Mitigation**: LLM-based MALM reduces input-, context-, and fact-conflicting hallucinations, but fine-grained numerical reasoning and multi-hop inference remain unresolved [2506.12483].
- **Access Restrictions**: LLM MALM requires access to model hidden states, currently limiting it to open-source models [2506.12483].

## 6. Extension Directions and Ongoing Work

Proposed improvements and research avenues for MALM include:

- **Weighted and Orthogonal Merging**: Introducing learnable $\alpha_i$ per adapter or enforcing span orthogonality across $A_i$ matrices may further reduce interference [2411.14064].
- **Dynamic Gating and Routing**: At inference, activating only a subset of relevant adapters or information sources could increase efficiency and decrease destruction from irrelevant domains [2411.14064].
- **Auxiliary Losses and Reasoners**: For LLMs, integrating hallucination-aware losses or explicit chain-of-thought modules into the graph can provide additional mitigation signals [2506.12483].
- **Continual Update and Modular Expansion**: MALM facilitates continual learning by enabling addition (or removal) of adapters without retraining the backbone [2411.14064].
- **Generalization across Modalities**: Extending MALM principles to generative vision models (e.g., diffusion) or to other PEFTs (e.g., DoRA, O-LoRA, QLoRA) is a plausible frontier [2411.14064].

## 7. Significance and Synthesis

MALM consolidates multiple lines of adapter research under a unified paradigm of multi-information modularity. Empirical validations across disparate domains (vision, language modeling, translation) confirm that MALM-style adapters provide substantial practical benefit: parameter-efficient multitask deployment, robust domain transfer without catastrophic forgetting, and enhanced factual faithfulness in LLMs. The balance between modular composition, computational efficiency, and interference minimization remains an active area, with prospects of learned weighting and dynamic selection promising further gains.

Source: https://www.emergentmind.com/topics/multi-information-adapters-malm