MLLM-Engaged Joint Optimization (MEJO)

• MLLM-Engaged Joint Optimization (MEJO) is a paradigm where multimodal large language models and auxiliary modules are jointly optimized to improve cross-modal alignment.
• It integrates language reasoning, visual embeddings, and task-specific controls through shared objectives and joint loss formulations across various tasks.
• Empirical results in image editing, video QA, surgical recognition, and autonomous driving demonstrate MEJO’s effectiveness in reducing modality gaps and enhancing performance.

Searching arXiv for the cited papers and related "MLLM-Engaged Joint Optimization" usage. MLLM-Engaged Joint Optimization (MEJO) denotes a family of multimodal learning and inference schemes in which a multimodal LLM (MLLM) is not treated as a frozen instruction parser alone, but as an actively engaged component whose language reasoning, visual understanding, retrieval behavior, sampling policy, task heads, or serving controls are optimized jointly with other modules under shared objectives. Across recent work, the term has been used in image editing, zero-shot composed image retrieval, long-form video question answering, surgical triplet recognition, autonomous driving, and MLLM serving analysis. The common thread is that MLLM-derived signals are made operational inside the optimization loop rather than appended as post hoc prompts, with joint objectives intended to reduce modality gaps, improve task alignment, and coordinate heterogeneous subproblems (Wang et al., 25 May 2025, Tu et al., 26 May 2025, Tan et al., 26 Feb 2026, Zhang et al., 16 Sep 2025, Liu et al., 14 Dec 2025, Moghadampanah et al., 27 Dec 2025).

1. Conceptual scope and defining properties

In current usage, MEJO is a paradigm rather than a single algorithm. Its defining structure is jointness: multiple signals or modules that would otherwise be trained separately are optimized together so that the MLLM’s internal representations become directly useful for downstream control, alignment, or prediction. In "MIND-Edit" the MLLM jointly produces an optimized instruction $T_{\text{opt}}$ and an MLLM-derived visual embedding $v_{\text{insight}}$ for diffusion-based editing (Wang et al., 25 May 2025). In "MVFT-JI" the MLLM synthesizes supervision for two complementary retrieval tasks, while inference fuses VLM and MLLM signals through late fusion (Tu et al., 26 May 2025). In "MSJoE" the MLLM and a lightweight sampler are jointly optimized through reinforcement learning so that query reasoning and key-frame selection co-adapt (Tan et al., 26 Feb 2026).

A second defining property is active engagement of the MLLM in task structure. The MLLM may clarify ambiguous instructions, generate target texts, reason out visually grounded sub-queries, provide expert-level semantic prompts, or share parameters with dense spatial heads. This differs from pipelines in which an MLLM is merely prompted once and left outside training. In surgical triplet recognition, MEJO explicitly targets both inter-task and intra-task optimization conflicts through shared-specific disentanglement and coordinated gradient learning (Zhang et al., 16 Sep 2025). In autonomous driving, DrivePI uses a trainable Qwen2.5-VL 0.5B backbone whose shared hidden states feed text, occupancy, flow, and planning heads in parallel under a single joint loss (Liu et al., 14 Dec 2025).

A third property is task-dependent coupling between the MLLM and auxiliary modules. The auxiliary component can be a diffusion model interface, a Q-Former, a sampler, prompt-integration modules, dense voxel decoders, or stage-wise serving controls. This suggests that MEJO is best understood as a design pattern for cross-module co-optimization, not as a fixed architectural template.

2. Recurring architectural patterns

One recurrent pattern is dual-path conditioning. In MIND-Edit, the MLLM produces two complementary outputs from the same source image $I_{\text{src}}$ and raw instruction $T_{\text{raw}}$: the language head $\mathbf{F}_t$ generates $T_{\text{opt}}$, while the vision head $\mathbf{F}_v$ maps intermediate hidden states to $v_{\text{insight}}$. These two signals then enter decoupled cross-attention branches of Stable Diffusion v1.5 through IP-Adapter, with fused output

$Z_{\text{new}} = Z' + Z''.$

The design keeps the diffusion model fixed and injects language and image guidance in parallel, rather than forcing a single textual condition to carry all editing intent (Wang et al., 25 May 2025).

A second pattern is shared latent query spaces. MVFT-JI removes pseudo-text adapters and directly fine-tunes the VLM’s Q-Former using shared learnable query tokens $\{q_j\}_{j=1..k}$. The same query-token machinery is used for composed queries, target texts, captions, and images, so that multiple tasks constrain a common representational substrate. Similarity is computed through token-level cosine similarity with max-over-target-token aggregation, and both training tasks operate on this shared token geometry (Tu et al., 26 May 2025).

A third pattern is reasoning-to-selection coupling. MSJoE first uses the MLLM to generate up to $v_{\text{insight}}$0 distinct queries from a sparse preview, then constructs a query-frame similarity matrix

$v_{\text{insight}}$1

and finally feeds $v_{\text{insight}}$2 to a 1D U-Net sampler of approximately $v_{\text{insight}}$3M parameters. The sampler predicts per-frame probabilities for without-replacement key-frame selection, after which the MLLM answers from the selected high-resolution frames. Query reasoning, frame selection, and answer generation therefore form a closed loop rather than a one-way preprocessing stack (Tan et al., 26 Feb 2026).

A fourth pattern is explicit decomposition of shared and task-specific subspaces. In surgical triplet recognition, the Shared-Specific-Disentangled (S$v_{\text{insight}}$4D) scheme assigns expert-level semantic augmentation to a shared branch through GMM Prompt Integration (GPI) and prefix tuning, while a Temporal-Spatial Prompt (TSP) adapter constructs task-specific features for instrument, verb, target, and triplet heads. The MLLM contribution appears in a probabilistic prompt pool built from GPT-4o-generated instrument-anchored attributes encoded by CLIP (Zhang et al., 16 Sep 2025).

A fifth pattern is unified multimodal tokenization for dense spatial outputs. DrivePI converts multi-view images and LiDAR into a latent BEV feature $v_{\text{insight}}$5, patchifies it, performs cross-attention between pooled patch summaries and patch features, and linearly projects the result into MLLM tokens. The same fused hidden states support text QA, 3D occupancy, occupancy flow, and action diffusion in parallel. Here MEJO is realized as end-to-end joint training over a single 4D spatial-aware MLLM backbone (Liu et al., 14 Dec 2025).

3. Optimization formulations

MEJO methods span supervised multi-objective training, contrastive joint learning, reinforcement learning, gradient coordination, and serving-time constrained optimization.

In MIND-Edit, the MLLM is jointly fine-tuned with a language-generation loss and a visual-embedding alignment loss:

$v_{\text{insight}}$6

$v_{\text{insight}}$7

$v_{\text{insight}}$8

A training design detail is that $v_{\text{insight}}$9 consumes the 21st hidden layer while $I_{\text{src}}$0 consumes the final layer, which is used to reduce gradient interference (Wang et al., 25 May 2025).

In MVFT-JI, MEJO is instantiated by two InfoNCE objectives: target-text retrieval and text-to-image retrieval. The joint objective is

$I_{\text{src}}$1

with $I_{\text{src}}$2 in experiments. The theoretical justification decomposes the retrieval target as

$I_{\text{src}}$3

and links both InfoNCE terms to mutual-information lower bounds (Tu et al., 26 May 2025).

In MSJoE, the optimization is purely reinforcement-based. The joint objective maximizes expected QA reward over the coupled MLLM-sampler policy:

$I_{\text{src}}$4

The MLLM is updated with GRPO, the sampler with REINFORCE, and no supervised loss is added. The reward is additive, combining accuracy, format correctness, and an informativeness term that encourages peaked query-frame similarities (Tan et al., 26 Feb 2026).

In surgical triplet recognition, the multi-task backbone is trained with BCE losses for instrument, verb, target, and IVT prediction, while the triplet head additionally uses Coordinated Gradient Learning (CGL) to rebalance positive and negative BCE gradients for head and tail classes. The total form is

$I_{\text{src}}$5

with architectural disentanglement serving as the primary mechanism for inter-task conflict mitigation (Zhang et al., 16 Sep 2025).

In DrivePI, joint optimization is explicitly multi-head:

$I_{\text{src}}$6

with default weights $I_{\text{src}}$7. The text head uses autoregressive next-token cross-entropy; occupancy follows FlashOcc-style multi-component losses; flow uses weighted $I_{\text{src}}$8; planning uses $I_{\text{src}}$9 trajectory loss (Liu et al., 14 Dec 2025).

A distinct systems-level formulation appears in "Modality Inflation: Energy Characterization and Optimization Opportunities for MLLM Inference," where MEJO is a joint optimization problem over encoder configuration, visual token budget, batching, and per-stage DVFS:

$T_{\text{raw}}$0

subject to latency, accuracy, memory, and throughput constraints. Here jointness applies to inference-time control variables rather than to learned task heads (Moghadampanah et al., 27 Dec 2025).

4. Representative instantiations across domains

Work	Domain	MEJO mechanism
MIND-Edit (Wang et al., 25 May 2025)	Text-guided image editing	Jointly fine-tunes one MLLM to generate $T_{\text{raw}}$1 and $T_{\text{raw}}$2, then injects both into Stable Diffusion v1.5 through decoupled cross-attention
MVFT-JI (Tu et al., 26 May 2025)	Zero-Shot Composed Image Retrieval	Uses an MLLM to synthesize $T_{\text{raw}}$3, $T_{\text{raw}}$4, and captions, then jointly optimizes target-text retrieval and text-to-image retrieval on a Q-Former VLM
MSJoE (Tan et al., 26 Feb 2026)	Long-form video QA	Jointly evolves MLLM query reasoning and a 1D U-Net key-frame sampler through GRPO and REINFORCE
MEJO for surgical triplet recognition (Zhang et al., 16 Sep 2025)	Surgical scene understanding	Combines Shared-Specific-Disentangled learning, MLLM-powered probabilistic prompt pools, TSP task prompts, and Coordinated Gradient Learning
DrivePI (Liu et al., 14 Dec 2025)	Autonomous driving	Jointly trains a 4D spatial-aware MLLM for text QA, 3D occupancy, occupancy flow, and planning with shared hidden states
Modality Inflation guide (Moghadampanah et al., 27 Dec 2025)	MLLM serving and systems optimization	Jointly optimizes stage-wise energy via encoder configuration, token budget, batching, and DVFS under QoS constraints

These instantiations show that the MLLM can engage at multiple levels: semantic clarification, latent representation shaping, query synthesis, retrieval supervision generation, dense spatial grounding, policy learning, or even serving-time control. Taken together, they suggest that MEJO is less a domain-specific method than a reusable optimization stance in which MLLM outputs are promoted from auxiliary hints to variables that materially determine the training or inference objective.

5. Empirical characteristics and reported gains

In image editing, MIND-Edit reports HumanEdit results of CLIP-I $T_{\text{raw}}$5, LPIPS $T_{\text{raw}}$6, PSNR $T_{\text{raw}}$7, and SSIM $T_{\text{raw}}$8. On ComplexMultistepImageEditing, it reports CLIP-I $T_{\text{raw}}$9, LPIPS $\mathbf{F}_t$0, PSNR $\mathbf{F}_t$1, and SSIM $\mathbf{F}_t$2. The ablation labeled “#6 (Ours, both joint)” exceeds the independently trained “#3 (both but independently trained)” on all listed metrics, supporting the claim that jointly training $\mathbf{F}_t$3 and $\mathbf{F}_t$4 improves alignment and fidelity (Wang et al., 25 May 2025).

In zero-shot composed image retrieval, MVFT-JI reports FashionIQ average $\mathbf{F}_t$5 and $\mathbf{F}_t$6, CIRR $\mathbf{F}_t$7, $\mathbf{F}_t$8, and $\mathbf{F}_t$9, and CIRCO $T_{\text{opt}}$0 and $T_{\text{opt}}$1. Ablations show that removing either inference component or either training task degrades performance on multiple benchmarks, with target-text retrieval identified as the most critical for compositional understanding and text-to-image retrieval described as stabilizing cross-modal grounding (Tu et al., 26 May 2025).

In long-form video QA, MSJoE reports an $T_{\text{opt}}$2 average accuracy gain upon the base MLLM with $T_{\text{opt}}$3 frames, a $T_{\text{opt}}$4 point gain with $T_{\text{opt}}$5 frames, and $T_{\text{opt}}$6 higher accuracy than the strongest baseline method. With $T_{\text{opt}}$7 frames, the reported scores include MLVU $T_{\text{opt}}$8, LongVideoBench $T_{\text{opt}}$9, VideoMME-Long $\mathbf{F}_v$0, VideoMME-Avg $\mathbf{F}_v$1, and LVBench $\mathbf{F}_v$2. The paper further reports that total pipeline time increases by approximately $\mathbf{F}_v$3 versus uniform sampling, although the final QA step is approximately $\mathbf{F}_v$4 faster than uniform sampling due to prefix cache hits in vLLM (Tan et al., 26 Feb 2026).

In surgical triplet recognition, MEJO-B reports $\mathbf{F}_v$5 on CholecT45, surpassing TERL-B $\mathbf{F}_v$6 by $\mathbf{F}_v$7, with component scores $\mathbf{F}_v$8, $\mathbf{F}_v$9, and $v_{\text{insight}}$0. On CholecT50, MEJO-B is reported to outperform the second-best method (CoT) by $v_{\text{insight}}$1 $v_{\text{insight}}$2. Progressive ablations on Fold 1 show MSTCN at $v_{\text{insight}}$3, TransFPN at $v_{\text{insight}}$4, $v_{\text{insight}}$5 at $v_{\text{insight}}$6, $v_{\text{insight}}$7 at $v_{\text{insight}}$8, and $v_{\text{insight}}$9 at $Z_{\text{new}} = Z' + Z''.$0, which isolates the contribution of each MEJO component (Zhang et al., 16 Sep 2025).

In autonomous driving, DrivePI reports OccScore $Z_{\text{new}} = Z' + Z''.$1, RayIoU $Z_{\text{new}} = Z' + Z''.$2, and mAVE $Z_{\text{new}} = Z' + Z''.$3 on OpenOcc; planning $Z_{\text{new}} = Z' + Z''.$4 average $Z_{\text{new}} = Z' + Z''.$5 without ego status and $Z_{\text{new}} = Z' + Z''.$6 with ego status on nuScenes; and a collision rate of $Z_{\text{new}} = Z' + Z''.$7 with ego status. Relative comparisons include outperforming OpenDriveVLA-7B by $Z_{\text{new}} = Z' + Z''.$8 mean accuracy on nuScenes-QA, reducing collision rate by $Z_{\text{new}} = Z' + Z''.$9 over ORION from $\{q_j\}_{j=1..k}$0 to $\{q_j\}_{j=1..k}$1, exceeding FB-OCC by $\{q_j\}_{j=1..k}$2 RayIoU on OpenOcc, improving occupancy-flow mAVE from $\{q_j\}_{j=1..k}$3 to $\{q_j\}_{j=1..k}$4, and reducing planning error from VAD’s $\{q_j\}_{j=1..k}$5m to $\{q_j\}_{j=1..k}$6m (Liu et al., 14 Dec 2025).

At the systems level, the modality-inflation study reports multimodal energy overheads ranging from $\{q_j\}_{j=1..k}$7 to $\{q_j\}_{j=1..k}$8 over text-only baselines on NVIDIA A100-80GB. Qwen2.5-VL is reported at $\{q_j\}_{j=1..k}$9 energy and $v_{\text{insight}}$00 latency, while LLaVA-OneVision is reported at $v_{\text{insight}}$01 energy despite $v_{\text{insight}}$02 total input tokens. Stage-level measurements identify encoder-dominated and prefill-dominated bottlenecks, and stage-wise DVFS is presented as an effective optimization that allows energy savings with only modest performance impact (Moghadampanah et al., 27 Dec 2025).

6. Limitations, misconceptions, and open problems

A common misconception is that MEJO implies full end-to-end retraining of all components. The literature does not support that interpretation. In MIND-Edit, Stable Diffusion v1.5 and IP-Adapter are pretrained and kept fixed while only the MLLM, including $v_{\text{insight}}$03 and $v_{\text{insight}}$04, is fine-tuned (Wang et al., 25 May 2025). In MVFT-JI, the BLIP-2 ViT-L/14 image encoder is frozen, while the Q-Former and shared learnable query tokens are fine-tuned (Tu et al., 26 May 2025). In DrivePI, the vision encoder is frozen during both stages, while the spatial projector, MLLM, and task heads are jointly optimized in the second stage (Liu et al., 14 Dec 2025).

A second misconception is that MEJO is synonymous with supervised multi-task learning. MSJoE uses reinforcement learning only, with GRPO for the MLLM and REINFORCE for the sampler, and explicitly states that no supervised loss is added. By contrast, surgical MEJO uses BCE-based multi-task supervision plus gradient gating, and MVFT-JI uses dual InfoNCE objectives. The term therefore spans heterogeneous optimization regimes rather than a unique loss family (Tan et al., 26 Feb 2026, Zhang et al., 16 Sep 2025).

Several limitations recur across domains. MIND-Edit reports that performance can be constrained by Stable Diffusion v1.5’s generative quality and IP-Adapter’s capacity, and that extremely fine-grained attribute control or highly compositional edits may remain challenging when instructions are still ambiguous or when the implicit visual embedding under-specifies local details (Wang et al., 25 May 2025). MVFT-JI notes that subtle attribute changes may be underrepresented in synthetic $v_{\text{insight}}$05 and captions, that MLLM hallucinations can introduce label noise, and that one-to-one text-image training pairs can create distribution gaps in multi-target settings such as CIRCO (Tu et al., 26 May 2025). MSJoE identifies missing temporal cues, overfitting to visually salient but irrelevant frames, and brittle query generation when preview frames are too sparse (Tan et al., 26 Feb 2026).

In knowledge-heavy domains, prompt quality and domain specificity become central failure modes. Surgical MEJO depends on GPT-4o-generated, surgeon-validated instrument attributes; noisy prompts may degrade performance, and transferring the prompt pool beyond cholecystectomy may require new knowledge extraction (Zhang et al., 16 Sep 2025). DrivePI uses fixed task weights and does not apply reinforcement learning or closed-loop finetuning, which the paper identifies as opportunities for improved planning in complex scenarios (Liu et al., 14 Dec 2025).

At the serving layer, modality inflation indicates that MLLM engagement can incur substantial energy and latency overhead because image inputs add encoder stages and expand prefill token sequences. The energy study argues that architecture-aware, input-aware, and stage-aware control is necessary, and motivates dynamic stage-aware controllers, multi-GPU or disaggregated pipelines, dynamic token routing, and online optimization of encoder configuration, visual token budget, batching, and per-stage DVFS (Moghadampanah et al., 27 Dec 2025).

Taken together, these works suggest that MEJO is best viewed as a general principle of coupling MLLM-derived reasoning with jointly optimized auxiliary structures under shared constraints. Its empirical successes are domain-specific, but its central proposition is stable across settings: cross-modal alignment improves when the MLLM is trained or controlled as part of the task mechanism itself rather than used only as an external prompt-conditioned interface.