LLaVA-NeuMT: Multimodal Multilingual Translation

• The paper presents a multimodal translation framework that uses selective layer-neuron modulation to achieve state-of-the-art BLEU scores while updating only a fraction of parameters.
• LLaVA-NeuMT leverages the LLaVA-1.5-7B backbone to fuse visual and textual cues, employing a layer selection mechanism to mitigate multilingual interference.
• Empirical evaluations show that optimized neuron partitioning boosts translation accuracy and reduces computational costs on established benchmarks.

LLaVA-NeuMT is a multimodal multilingual translation framework that introduces selective layer-neuron modulation within large-scale language-vision models to address challenges in cross-lingual adaptation and computational efficiency for multimodal machine translation (MMT). By explicitly distinguishing and controlling the adaptation of language-specific and agnostic components at both the layer and neuron level, LLaVA-NeuMT mitigates multilingual interference, reduces redundant parameter updates, and achieves state-of-the-art translation performance while fine-tuning only a fraction of the model's parameters (Wei et al., 25 Jul 2025).

1. Architectural Foundations

LLaVA-NeuMT is built upon the LLaVA-1.5-7B multimodal LLM backbone, which jointly processes textual and visual modalities. The architecture consists of:

• A text encoder ($\mathcal{E}_t$): a transformer encoder that processes the source sentence $X^s$ and outputs hidden states $\mathcal{T}$.
• A vision encoder ($\mathcal{E}_v$): a Vision Transformer (ViT) that projects input image $I$ to visual tokens $\mathcal{V}$ (extracted from the penultimate layer).
• A lightweight vision–language connector that fuses $\mathcal{V}$ with the LLM’s embedding space, constructing a multimodal input sequence: $$[CLS]_v, V_1, \ldots, V_m, [SEP]_v, T_1, \ldots, T_n$$.
• The LLM decoder, which generates the target sentence $Y^t$ by attending over the fused multimodal representation.

The process is formalized as:

$Y^t = \mathcal{F}(X^s, I, s, t; \theta)$

with $X^s$0 and $X^s$1. Cross-modal attention mechanisms within the connector enable effective fusion of textual and visual cues, supporting disambiguation during translation.

2. Layer Selection Mechanism

A central innovation in LLaVA-NeuMT is the layer selection mechanism designed to identify and fine-tune only those layers that are most informative for multilingual multimodal translation. The approach relies on:

• Redundancy-based Importance Scoring: For each layer $X^s$2 within the vision–language connector and LLM backbone, a redundancy score $X^s$3 is computed using activations from both the pretrained model ($X^s$4) and after supervised fine-tuning ($X^s$5):

$X^s$6

where lower $X^s$7 indicates greater representational adaptation, i.e., stronger relevance for the fine-tuning task.

• Selection Algorithm: Layers are ranked by increasing $X^s$8, and a threshold fraction $X^s$9 (experimentally optimal at $\mathcal{T}$0) determines the top $\mathcal{T}$1 layers ($\mathcal{T}$2) for adaptation. Layers outside $\mathcal{T}$3 are frozen, thereby minimizing redundancy and cross-lingual interference arising from over-parametrized adaptation.

This systematic layer selection enables scalable multilingual adaptation by concentrating parameter updates on layers with high adaptation utility.

3. Neuron-Level Adaptation Strategy

Within layers identified as important, LLaVA-NeuMT applies neuron-level adaptation to further refine language pair-specific and shared knowledge:

• Neuron Importance Scoring: For each neuron $\mathcal{T}$4 in selected layers, an importance score per data instance is defined as:

$\mathcal{T}$5

where $\mathcal{T}$6 is the activation, and $\mathcal{T}$7 is the back-propagated gradient for that neuron.

• Language-Specific and Agnostic Partitioning: For $\mathcal{T}$8 language pairs and input samples, the mean and variance of $\mathcal{T}$9 are computed:

$\mathcal{E}_v$0

Language-specific neurons ($\mathcal{E}_v$1) exhibit maximal $\mathcal{E}_v$2 for pair $\mathcal{E}_v$3; language-agnostic set ($\mathcal{E}_v$4) includes neurons with low variance.

• Selective Gradient Masking: During mini-batch fine-tuning for a given language direction, only neurons in $\mathcal{E}_v$5 receive nonzero gradients:

$\mathcal{E}_v$6

Allowing only these sets to update, LLaVA-NeuMT achieves significant parameter savings—empirically, when $\mathcal{E}_v$7 (40% of layers) and using a 1:9 specific:agnostic neuron ratio per layer, only 40% of the total parameters are actively adapted.

4. Training Regimen and Optimization

• Objective Function: Standard cross-entropy loss is used for autoregressive generation. No additional contrastive or alignment objectives are applied, but visual tokens inherently produce cross-modal attention facilitating multimodal learning.
• Implementation Details:
  • Backbone: LLaVA-1.5-7B
  • Hardware: DeepSpeed ZeRO-3, 4×A100 (80 GB) GPUs
  • Training configuration: 4 epochs; per-GPU batch size 16; BF16 mixed precision; AdamW optimizer (lr=2e-5, weight_decay=0); 3% linear warm-up followed by cosine decay
  • Maximum text length: 2048 tokens; images resized to a consistent aspect ratio.

5. Empirical Evaluation

LLaVA-NeuMT is evaluated on two benchmarks:

Dataset	Text-only SOTA	LLaVA-1.5-SFT	LLaVA-NeuMT (40%)	LLaVA-NeuMT (80%)
M3-Multi30K	50.4	53.3	53.4	54.3
M3-AmbigCaps	51.2	58.0	58.3	59.7

Key findings:

• On M3-Multi30K (En→Fr/Cs/De/Lv/Hi/Tr), LLaVA-NeuMT achieves a BLEU-4 score of 53.4 with only 40% parameter updates, surpassing both text-only SOTA (50.4) and full-parameter LLaVA-1.5-SFT (53.3).
• On M3-AmbigCaps, which features greater lexical ambiguity, LLaVA-NeuMT (40%) attains 58.3 (vs. 58.0 for LLaVA-1.5-SFT; 51.2 for text-only).
• Ablation studies show optimal performance at $\mathcal{E}_v$8 (80% layers); the neuron ratio of 1:9 (specific:agnostic) outperforms alternatives (e.g., only agnostic or only specific neurons).

6. Analysis and Interpretive Observations

The selective adaptation scheme reveals several architectural and linguistic trends:

• Layerwise Importance: The initial 80% of transformer layers (in both the vision–language connector and LLM) show consistently high importance across all language pairs ($\mathcal{E}_v$90.5 by heatmap). Importance profiles are stable across languages, highlighting an architectural rather than purely linguistic basis for adaptation.
• Neuron Variance: Within selected layers, the top 10% of neurons by cross-lingual variance encode language-specific phenomena; the remaining 90% function as an agnostic backbone. High-variance neurons spatially localize to attention projection and MLP down-projection modules, supporting cross-modal adaptation.
• Qualitative Disambiguation: Integration of visual tokens enables explicit disambiguation not seen in text-only translation. For example, with the source sentence “There is a small house beside the bank,” image context enables LLaVA-NeuMT to select the riverbank sense in German (“neben dem Ufer”) as opposed to the financial institution sense (“neben der Bank”) rendered by text-only models.

7. Significance and Implications

LLaVA-NeuMT demonstrates that targeted, selective adaptation at granularity levels below the model or layer—namely, at both the layer and neuron—offers a practical and principled solution to scalability challenges in multilingual multimodal translation. By fine-tuning approximately 40% of parameters, the model not only surpasses full fine-tuning baselines in BLEU score, but also substantially reduces computational costs and multilingual interference. The explicit modeling and partitioning of layer/neuron roles offers general insight into the dynamics of cross-lingual and cross-modal adaptation, with implications for scalable deployment of multimodal translation systems in multi-domain, multi-language contexts (Wei et al., 25 Jul 2025).