MLLM: Architecture, Adaptation, & Applications

• MLLM is a multimodal large language model that fuses visual, textual, and other modality data using a three-part architecture: modality encoder, connector, and language decoder.
• Research in MLLMs emphasizes advanced adaptation techniques like Low-Rank Adaptation and progressive curriculum learning to enhance cross-modal retrieval and reasoning.
• MLLM applications span domain-specialized tasks such as traffic analysis, facial recognition, and CAD generation while addressing challenges in evaluation, robustness, and deployment efficiency.

MLLM usually denotes a multimodal LLM: a LLM augmented with perception modules so that it can ingest and reason over multiple modalities, most commonly images and text, and in some settings video, audio, point clouds, or interleaved multimodal sequences. In the literature surveyed here, MLLMs are used for visual question answering, OCR and text-in-image understanding, chart and table comprehension, image captioning, multimodal mathematical or commonsense reasoning, retrieval, spatial reasoning, traffic video analysis, robotic manipulation, image editing, and CAD generation (Wen et al., 1 Jun 2025). The acronym is not fully stable across subfields: one report uses “MLLM” to mean multilingual LLM, illustrating an ambiguity in current terminology rather than a settled naming convention (Huang et al., 2024).

1. Conceptual scope and canonical architecture

A recurrent architectural pattern presents the MLLM as a three-part system: a modality encoder, a connector or projector that maps non-text features into the LLM’s token space, and an autoregressive or decoder-style language backbone. Face-MLLM describes this pattern explicitly as a vision encoder, an MLP projection head, and a 7B-class LLM decoder, with visual tokens concatenated with text tokens for standard autoregressive decoding (Sun et al., 2024). FedMLLM gives the same decomposition more generally as vision encoder, LLM, cross-modal modules or adapters, and PEFT layers such as LoRA (Xu et al., 2024).

Within this template, current systems vary in how tightly they fuse modalities. Some retain a relatively standard image-text pipeline with frozen encoders and lightweight adapters. Others reinterpret the decoder backbone itself as an embedding model. U-MARVEL, for example, adapts Qwen2-VL-7B-Instruct for universal multimodal retrieval by switching decoder-only causal attention to bidirectional attention and using mean pooling over the last hidden states, while masking instruction tokens during pooling to reduce embedding bias (Li et al., 20 Jul 2025). CAD-MLLM extends the same broad design to text, multi-view images, and point clouds, aligning all of them to the LLM space and then treating CAD command generation as autoregressive program synthesis (Xu et al., 2024).

This architectural family includes several important connector variants. BLIP-2 is cited as using a Q-Former bridge in the broader FedMLLM discussion, whereas Face-MLLM explicitly does not use a Q-Former and instead projects image patch features directly into the LLM embedding space (Xu et al., 2024, Sun et al., 2024). Spatial-MLLM introduces a dual-encoder design, pairing the Qwen2.5-VL visual encoder with a second branch initialized from the VGGT backbone, then summing projected semantic and structural features into unified visual tokens (Wu et al., 29 May 2025). Across these variants, the central design problem remains the same: how to preserve modality-specific structure while exposing the LLM to a unified token stream.

2. Representation learning, adaptation, and reasoning

A large fraction of current MLLM work concerns not the base architecture but the adaptation recipe. A standard mechanism is Low-Rank Adaptation (LoRA), where a weight update is factorized as

$\Delta W = BA$

with low rank $r \ll d$, drastically reducing trainable parameters. Traffic-MLLM applies LoRA to the query, key, value, and output projections of Qwen2.5-VL-3B, freezing all other layers and updating only the low-rank matrices during supervised fine-tuning (Xiu et al., 14 Sep 2025). FedMLLM uses the same idea in federated settings, writing model parameters as $w = w_0 + BA$ and exchanging only LoRA-adapted updates across clients (Xu et al., 2024).

Another major line of work recasts MLLMs as embedding learners. U-MARVEL trains a Qwen2-VL-7B-Instruct backbone with InfoNCE for universal multimodal retrieval and reports that bidirectional attention with mean pooling outperforms last-token extraction with compression prompts, while instruction masking during pooling gives a small additional improvement (Li et al., 20 Jul 2025). This is paired with a progressive curriculum—text retrieval adaptation, cross-modal alignment on image-text pairs, and instruction-tuned multimodal retrieval—followed by filtered hard-negative mining and reranker distillation. The resulting system reaches a Local score of 63.2 on M-BEIR as a single model and 64.8 with reranker fusion (Li et al., 20 Jul 2025).

A second adaptation theme is explicit reasoning optimization. Hint-GRPO argues that applying GRPO directly to MLLMs suffers from low data utilization—many prompt groups yield all-wrong outputs and thus zero effective advantage—and from text bias, where the model increasingly ignores image evidence and relies on text alone. It addresses this with adaptive hints injected into the answer stream and with test-time text-bias calibration of token logits. On geometry reasoning, Debiased Hint-GRPO raises Qwen2-VL-7B average accuracy from 33.92 for the authors’ GRPO baseline to 38.55, and raises Qwen2.5-VL-3B from 36.83 to 41.95 (Huang et al., 31 Mar 2025). This directly challenges the older assumption that process-reward methods necessarily dominate outcome-reward methods for multimodal reasoning.

Spatiotemporal and geometric specialization deepen this trend. Traffic-MLLM augments Qwen2.5-VL-3B with 3D patching, 2D-RoPE, MRoPE, RMSNorm, SwiGLU, and a knowledge-prompting module that combines CoT with RAG to inject traffic regulations and authoritative interpretations (Xiu et al., 14 Sep 2025). Spatial-MLLM targets purely 2D spatial reasoning by combining a semantic encoder with a geometry-informed spatial encoder and by performing space-aware frame sampling that maximizes voxel coverage under tight token budgets (Wu et al., 29 May 2025). Both systems suggest that general-purpose image-text pretraining remains insufficient when the task depends on persistent geometry, temporal coherence, or domain-specific rule grounding.

3. Evaluation, benchmark design, and measurement theory

The rapid growth of MLLMs has produced a parallel growth of benchmarks, but recent work treats benchmark design itself as a research problem. MITV argues that full-coverage evaluation on large multimodal benchmarks is resource-intensive and often redundant. It constructs difficulty labels from the responses of ten representative MLLMs and turns evaluation into an adaptive interview, using expected information gain, difficulty updates, and level-distribution control to approximate full-benchmark rankings with far fewer questions. At equal budgets, MITV consistently exceeds random sampling in SRCC, PLCC, and KRCC, especially in the low-question regime (Wen et al., 1 Jun 2025).

A stronger critique comes from the Information Density Principle, which treats benchmark quality as a product of four factors: low fallacy, high difficulty, low redundancy, and high diversity. It formalizes benchmark insight as

$$E(I) \propto (1 - D_{fal}) \cdot D_{dif} \cdot (1 - D_{red}) \cdot D_{div}$$

and applies this framework to 19 MLLM benchmarks and 17,912 instances (Li et al., 13 Mar 2025). The reported results show that newer benchmarks are often harder and less redundant, but diversity remains a recurring bottleneck.

A complementary line uses structural statistics rather than item-level heuristics. GoldBench reorganizes benchmark abilities into a Piaget-inspired hierarchy of Perception, Memory, and Reasoning, then uses PLS-SEM to prune weak or redundant indicators. The final Gold benchmark reduces an 85-task pool to 12 indicators with all loadings at or above 0.75, VIFs between 1.12 and 2.75, and maximum HTMT 0.86; its benchmark-level correlation with human preference reaches Pearson $r = 0.7359$ (Zou et al., 13 Jun 2025). This framing treats benchmark construction as a latent-variable modeling problem, rather than a simple collection of tasks.

Together, these studies undermine a common misconception that benchmark scale alone guarantees evaluative quality. The current literature instead emphasizes informativeness, human alignment, and adaptive efficiency as primary design criteria (Wen et al., 1 Jun 2025, Li et al., 13 Mar 2025, Zou et al., 13 Jun 2025).

4. Domain-specialized MLLMs

A major development in the literature is the shift from general-purpose MLLMs to domain-specialized systems trained on curated multimodal corpora and evaluated on task-specific benchmarks.

System	Domain	Representative result
Traffic-MLLM (Xiu et al., 14 Sep 2025)	Traffic video understanding and causal QA	44.10% overall accuracy on TrafficQA
Spatial-MLLM (Wu et al., 29 May 2025)	Spatial reasoning from 2D images/videos	48.4 average on VSI-Bench
Face-MLLM (Sun et al., 2024)	Face perception	91.2% on RAF-DB; 83.5% on EmotioNet
CAD-MLLM (Xu et al., 2024)	Parametric CAD generation	Point-conditioned SegE 1.66, DangEL 0.46, FluxEE 0.044
AIC MLLM (Xiong et al., 2024)	Robotic manipulation of articulated objects	0.72 success on train categories, 0.70 on test categories
MIND-Edit (Wang et al., 25 May 2025)	MLLM-guided image editing	HumanEdit PSNR 22.2714

Traffic-MLLM is emblematic of domain grounding. It targets two deficiencies of generic visual-LLMs: weak modeling of continuous spatiotemporal dynamics and poor adaptation to traffic-specific regulations, which can induce hallucinated judgments. On TrafficQA it reaches 44.10% overall accuracy, with 46.50 on Basic, 41.60 on Attribution, and 39.20 on Counterfactual, while also achieving 83.10% on real-world Mapillary after CARLA-data fine-tuning (Xiu et al., 14 Sep 2025).

Spatial-MLLM addresses a different gap: spatial intelligence from purely 2D observations. Its dual-encoder design and space-aware frame sampling let it approach or surpass many 3D-input systems without explicit depth or point clouds. On VSI-Bench it reaches an average of 48.4, with 65.3 on Object Counting and 63.1 on Object Size (Wu et al., 29 May 2025).

Face-MLLM shows the same pattern for specialized perception. The paper attributes the poor performance of general MLLMs on faces to the absence of face-centric, attribute-rich captions in broad web corpora. After re-annotating LAION-Face and reformulating traditional face datasets into QA format, Face-MLLM reaches 91.2% on RAF-DB, 71.8% on LFWA, 83.5% on EmotioNet, and age MAE 5.06 on AgeDB (Sun et al., 2024).

The same specialization logic appears in robotics and generation. AIC MLLM uses visual masks for position correction and textual joint prompts for rotation correction in SE(3) manipulation of articulated objects, while MIND-Edit uses an MLLM not only to optimize text instructions but also to generate a visual embedding that conditions IP-Adapter inside Stable Diffusion v1.5 (Xiong et al., 2024, Wang et al., 25 May 2025). CAD-MLLM treats CAD construction sequences as a multimodal generative language, aligning text, image, and point inputs to a Vicuna-7B backbone and outperforming prior conditional CAD baselines on topology and enclosure metrics (Xu et al., 2024).

A plausible implication is that “general MLLM” and “specialist MLLM” are no longer cleanly separable categories. The literature increasingly treats specialization as a first-class design axis rather than a downstream fine-tuning afterthought.

5. Reliability, security, and systems concerns

The same breadth that makes MLLMs attractive also produces a large reliability surface. One axis is adversarial robustness. The TiFA workshop attack report shows that a simple combination of suffix injection and vanilla PGD can sharply degrade LLaVA-1.5. The reported attack raises Helpful success from 48.09 to 80.92, Honest from 62.25 to 69.54, and Total from 48.63 to 60.47 under the challenge metric (Guo et al., 2024). The paper’s analysis emphasizes strong language bias: once the injected suffix creates a shortcut to the target answer, the model tends to ignore visual evidence.

The threat compounds in multi-agent settings. “The Wolf Within” shows that a single compromised MLLM can act as an operative inside an MLLM society, generating prompts that induce another agent to emit harmful content. On LLaVA, attack success rates exceed 80% in many prohibited categories and reach 100.00% in Children harm, Economic harm, Fraud/deception, and High-risk government decision making (Tan et al., 2024). This shifts the security problem from isolated jailbreaks to covert propagation through inter-agent communication.

A second axis is deployment efficiency. Inf-MLLM addresses the long-context and streaming-video bottleneck by exploiting “attention saddles” and maintaining a fixed-size KV cache of recent plus relevant tokens. It reports stable performance up to 4M-token texts and multi-round conversations with 1-hour-long videos on a single GPU, while providing a reported 2× speedup over H2O under long-context conditions (Ning et al., 2024). This reframes MLLM inference as a cache-management and long-term memory problem as much as a model-capacity problem.

A third axis is distributed training under heterogeneity. FedMLLM studies federated fine-tuning of MiniCPM-V-2_6-int4 with LoRA across aligned, missing, cross, and hybrid modality scenarios. Across Hateful-Memes and CrisisMMD, adaptive server-side methods such as FedAdam, FedYogi, and FedAdagrad outperform local training, while prompt-based modality-agnostic strategies and layer-masked regularization further improve robustness (Xu et al., 2024). Here the central issue is not hallucination but instability under multimodal non-IID partitions.

6. Open problems and emerging directions

Several open problems recur across the literature. One is grounding: Hint-GRPO identifies text bias as a systematic failure mode in multimodal reasoning, while Traffic-MLLM frames domain hallucination as a consequence of weak regulatory grounding (Huang et al., 31 Mar 2025, Xiu et al., 14 Sep 2025). Another is evaluation validity: Information Density and GoldBench both argue that many benchmarks remain redundant, weakly separated, or only loosely aligned with human preferences (Li et al., 13 Mar 2025, Zou et al., 13 Jun 2025).

A third is data-centric iteration. MLLM-DataEngine proposes a closed loop in which evaluation identifies bad cases, Adaptive Bad-case Sampling selects new target types, GPT-4 generates incremental multimodal supervision, and Interactive Prompt Optimization reduces generation failures. Over two rounds it builds the GPTVQA dataset and improves MiniGPT-4 variants on MMBenchmark, A-OKVQA, and MME (Zhao et al., 2023). This suggests that MLLM progress may depend as much on evaluation-driven data refinement as on larger backbones.

A fourth is cross-modal unification. ES-Merging, in a biological setting, argues that parameter-space heuristics are insufficient for merging specialized MLLMs. Instead it estimates layer-wise and element-wise coefficients from embedding-space signals and shows that this representation-aware merge can outperform standard merging baselines and even surpass task-specific fine-tuned models on several interaction prediction benchmarks (Lee et al., 15 Mar 2026). This suggests a broader research direction in which MLLMs are composed not by naïve averaging but by representation-aware fusion.

Finally, the acronym itself remains unsettled. In most of the cited literature, MLLM refers to multimodal LLMs, but MERaLiON-TextLLM uses the same abbreviation for multilingual LLMs (Huang et al., 2024). This ambiguity is more than terminological: it reflects a convergence of research agendas around language, perception, retrieval, grounding, and alignment. Current usage strongly favors the multimodal sense, but the broader pattern suggests that “MLLM” is becoming a container term for several distinct extensions of the LLM paradigm rather than a single, fully standardized object.