OmniBridge: Unified Multimodal Framework
- OmniBridge is a unified multimodal framework that integrates vision-language understanding, generation, and retrieval via latent space alignment.
- It employs a bidirectional transformer to align pretrained LLM hidden states with diffusion and retrieval embeddings, improving performance without full system retraining.
- A two-stage decoupled training strategy separates supervised fine-tuning from semantic-guided diffusion, ensuring robust multimodal reasoning and task integration.
Searching arXiv for primary and related papers on OmniBridge and adjacent "bridge" formulations in multimodal modeling. to=arxiv_search.search 新天天彩票={"query":"OmniBridge multimodal latent space alignment OmniBridge arXiv 2025", "max_results": 10} code omitted to=arxiv_search.search ＿亚洲 code omitted മാറില്ല to=arxiv_search.search 彩神争霸快_code omitted 】【：】【“】【query":"OmniBridge multimodal latent space alignment", "max_results": 5} to=arxiv_search.search 大发彩票官网 微信里的天天中彩票={"query":"(Xiao et al., 23 Sep 2025) OmniBridge Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment", "max_results": 3} OmniBridge is a unified multimodal framework that supports vision-language understanding, generation, and retrieval within a single architecture by combining a language-centric design with latent space alignment. It reuses a pretrained multimodal LLM, introduces a lightweight bidirectional latent alignment module, and adopts a two-stage decoupled training strategy in which supervised fine-tuning and latent space alignment are separated from semantic-guided diffusion training. In its canonical formulation, the framework is built around Qwen2-VL-7B, a bidirectional latent alignment module called BiTransformer, and HunyuanDiT, with the stated objective of avoiding both task isolation and full retraining of large multimodal systems from scratch (Xiao et al., 23 Sep 2025).
1. Definition and model family
OmniBridge is defined as a unified and modular multimodal framework for three tasks: vision-language understanding, vision-language generation, and cross-modal retrieval. The system is explicitly language-centric: the pretrained multimodal LLM serves as the central reasoning engine, while latent space alignment is used to connect LLM hidden states, diffusion latents, and retrieval embeddings into a coherent shared space. The paper’s central claim is that latent space alignment, rather than a single autoregressive image-token stack or a loose tool-using pipeline, is the organizing principle that makes joint support for all three capabilities possible (Xiao et al., 23 Sep 2025).
This positioning is stated against two prevailing design families. First, single transformer or autoregressive token systems such as Chameleon, Emu3, Janus, and TransFusion train over mixed image and text tokens, but often require massive pretraining from scratch, use visual tokenizers that lose fine-grained details, and still underperform top diffusion models in image quality while failing to integrate retrieval well. Second, tool-like diffusion-plus-LLM systems such as DreamLLM, SEED-X, X-VILA, and Emu preserve strong generation quality, but keep the diffusion model largely separate, so the LLM has no direct generative capability and retrieval remains a separate dual-encoder or CLIP-like system. OmniBridge is presented as an alternative to both patterns (Xiao et al., 23 Sep 2025).
A common misconception is to treat OmniBridge as merely an LLM that writes prompts for a generator. That characterization is incomplete. The framework does use the LLM for prompt rewriting, but its distinctive mechanism is the explicit alignment of LLM hidden state space, diffusion latent space, and retrieval embedding space by means of a bidirectional transformer and learnable query embeddings, with separate training phases to reduce task interference (Xiao et al., 23 Sep 2025).
2. Architecture and latent-space alignment
The core backbone is Qwen2-VL-7B-Instruct, described as a pretrained multimodal LLM with 28 transformer layers and support for up to 32k tokens. In OmniBridge, Qwen2-VL functions as the central autoregressive model for multimodal reasoning and prompt rewriting, and its hidden states $\mathbf{H}$ provide the semantic substrate for alignment. Stage 1 adapts the model with LoRA; Stage 2 freezes the LLM and trains the alignment components instead (Xiao et al., 23 Sep 2025).
The central alignment module is the Bidirectional Transformer, or BiTransformer. It is implemented with a pretrained mT5 backbone of approximately 700M parameters, together with a linear down-projection from Qwen2-VL hidden states into the mT5 hidden dimension and cross-attention blocks inserted at layers 05OmniBridge multimodal latent space alignment OmniBridge arXiv 20255:58, 12, 16, and 20. The module takes final LLM hidden states for image-conditioned and text-only sequences, together with learnable query tokens $\mathbf{Q}_{\text{img}}$ and $\mathbf{Q}_{\text{text}}$, and produces cross-modal latent vectors by combining self-attention on the queries with cross-attention over LLM hidden states (Xiao et al., 23 Sep 2025).
The fused representation used for retrieval is defined as
$\mathbf{e}_m = \sigma(\alpha_m)\,\mathbf{e}_m^{\mathrm{BiT}} + \bigl(1-\sigma(\alpha_m)\bigr)\,\mathbf{e}_m^{\mathrm{LLM}}, \qquad m\in\{\mathrm{img},\mathrm{text}\},$
where $\alpha_{\text{img}}$ and $\alpha_{\text{text}}$ are learnable scalars and $\sigma$ is a sigmoid. These fused embeddings are L2-normalized and then used either as retrieval representations or as upstream conditioning for generation (Xiao et al., 23 Sep 2025).
The generation side is built on HunyuanDiT. OmniBridge does not modify the diffusion backbone heavily; instead, it learns to project BiTransformer outputs into the diffusion conditioning space and optimizes a latent regression objective. The visual encoder is the one inside Qwen2-VL-7B, reused directly for all tasks. This architecture is modular rather than monolithic: a pretrained MLLM remains the reasoning hub, BiTransformer serves as the latent alignment mechanism, and HunyuanDiT remains the image decoder (Xiao et al., 23 Sep 2025).
Bidirectionality is a specific architectural commitment rather than a stylistic detail. The paper reports that replacing BiTransformer with a unidirectional transformer produces significantly worse image quality, especially for long prompts such as 130-token prompts, because a unidirectional model cannot integrate late prompt content as effectively as a bidirectional one. This makes BiTransformer central to long-context compositional generation and retrieval (Xiao et al., 23 Sep 2025).
3. Two-stage decoupled training
OmniBridge is trained with a two-stage decoupled strategy. Stage 1 aligns LLM behavior for multimodal reasoning through supervised fine-tuning and reinforcement learning. Stage 2 freezes the LLM and trains BiTransformer, query embeddings, and diffusion-side modules to align cross-modal latent spaces via semantic-guided diffusion and contrastive retrieval losses. The separation is motivated explicitly as a response to task interference (Xiao et al., 23 Sep 2025).
Stage 1 uses several small-scale instruction datasets rather than massive from-scratch pretraining. The reported mixture includes 7K multimodal understanding samples from Mulberry-260K, 2K unimodal CoT examples from Magpie-Reasoning-V2-250K-CoT, 3K short and 4K long image-generation captions generated by Qwen3-30B-A3B, 2.5K image-editing samples from MagicBrush, and Flickr30K pairs for retrieval. The principal objective for understanding is autoregressive next-token modeling:
$\mathcal{L}_{\mathrm{AR}} = - \mathbb{E}_{x \sim \mathcal{D}_{\mathrm{und}}} \left[ \sum_{i = \ell_{\mathrm{con}}}^{\ell-1} \log P_{\theta}(x_{i+1} \mid x_1, \ldots, x_i) \right].$
In addition, the framework uses R1-style distillation on long Chain-of-Thought sequences and StepGRPO, a stepwise group relative policy optimization method that rewards correctness and conciseness while penalizing repetitive or irrelevant steps (Xiao et al., 23 Sep 2025).
Stage 2 is centered on semantic-guided diffusion training and latent alignment. Learnable query embeddings are gradually substituted for explicit text conditioning by means of the mixture
$\mathbf{z}_{\mathrm{cond}} = \beta \cdot \mathbf{z}_{\mathrm{text}} + (1-\beta)\cdot\mathbf{z}_{\mathrm{query}},$
with a replacement schedule that begins with 15% replacement, then linearly anneals through a 15%→75% progressive stage, and ends at 100% replacement. This procedure is described as enabling the queries to inherit semantic meaning from text while ultimately becoming sufficient conditioning carriers on their own (Xiao et al., 23 Sep 2025).
The reported generation loss is a latent regression objective,
$\mathcal{L}_{\mathrm{Gen}} = \mathbb{E}_{\mathbf{z}_0 \sim q,\, t \sim \mathcal{U}(0,1)} \left[ \left\| \mathbf{z}_t - \hat{\mathbf{z}}_t(\mathbf{c}) \right\|_2^2 \right],$
while retrieval uses an InfoNCE contrastive objective,
$\mathbf{Q}_{\text{img}}$0
The stated advantage of decoupling is that Stage 2 can improve generation and retrieval without perturbing the LLM’s understanding behavior, because the LLM is frozen during latent alignment (Xiao et al., 23 Sep 2025).
4. Unified task pathways
For understanding tasks, the data flow is comparatively direct. Images are encoded by the internal Qwen2-VL visual encoder and projected into the LLM embedding space; text tokens and visual tokens are then interleaved and processed autoregressively by Qwen2-VL. The LLM head produces answer text, often with Chain-of-Thought. BiTransformer is not required for basic Stage-1 understanding, although the same hidden states later serve as the input substrate for retrieval and generation alignment (Xiao et al., 23 Sep 2025).
For generation, OmniBridge uses the LLM in a more explicitly mediating role. Qwen2-VL performs prompt rewriting: it expands short prompts with more explicit attributes and relations, and compresses long prompts to fit diffusion conditioning limits. For editing, it takes an input image and an editing instruction, and rewrites them into a caption describing the desired output image. The resulting caption is wrapped with special tokens <img> and </img>, passed through the LLM, and then processed by BiTransformer queries to produce latent conditioning vectors for HunyuanDiT (Xiao et al., 23 Sep 2025).
The editing path is not implemented as a separate editor with a separate representation. Instead, editing is unified with generation by translating image-plus-instruction inputs into the same latent conditioning pathway used for text-to-image generation. This yields a single semantic interface for both tasks, with the distinction residing in the prompt form rather than in task-specific architectural branching (Xiao et al., 23 Sep 2025).
For retrieval, the input may be an image, a text query, or a multimodal sequence. Qwen2-VL first computes hidden states; BiTransformer queries then attend bidirectionally over those hidden states; attention pooling yields image or text embeddings; and cosine similarity is used for ranking. The same shared embedding space supports image→text, text→image, and text→text retrieval. This makes retrieval an endogenous component of the framework rather than a separately trained dual encoder (Xiao et al., 23 Sep 2025).
A plausible implication is that OmniBridge’s “unified” designation is best understood operationally rather than merely architecturally. The same hidden-state substrate supports autoregressive reasoning, diffusion conditioning, and retrieval ranking, but the system preserves modularity by keeping the LLM, alignment module, and diffusion model distinct. That is a different notion of unification from token-only multimodal transformers (Xiao et al., 23 Sep 2025).
5. Empirical profile
The reported experimental setup uses Qwen2-VL-7B-Instruct as the base LLM, mT5 as the BiTransformer, HunyuanDiT as the diffusion model, and NVIDIA A800 GPUs, with a training cost reported as less than 10 GPU-days per model without RL. The full task suite covers multidisciplinary reasoning, mathematics, OCR and document understanding, multi-image reasoning, real-world reasoning, hallucination benchmarks, text-to-image generation, image editing, and Flickr30K retrieval (Xiao et al., 23 Sep 2025).
The most salient quantitative results are summarized below.
| Task area | Metric | OmniBridge |
|---|---|---|
| ScienceQA-Img | Accuracy | 93.3 |
| M3CoT | Overall | 62.2 |
| MathVista (mini) | Accuracy | 63.5 |
| ChartQA | Accuracy | 85.1 |
| RealWorldQA | Accuracy | 64.5 |
| MME | Score | 2352.0 |
| HallusionBench | Score | 53.4 |
| POPE | Score | 90.9 |
| GenEval | Overall | 0.61 |
| DPG-Bench | Overall | 78.93 |
| ImgEdit-Bench | Overall | 2.28 |
| Flickr30K | Average recall | 92.5 |
On multimodal reasoning, OmniBridge reports 93.3% on ScienceQA-Img, compared with 85.5% for Qwen2-VL-7B, and 62.2 on M3CoT overall, compared with 56.3 for Qwen2-VL-7B, 55.6 for LLaVA-CoT-11B, 43.1 for Emu3, and 43.8 for JanusFlow. On MathVista (mini), the reported value is 63.5. On ChartQA, it reports 85.1%, compared with 83.0 for Qwen2-VL-7B. On real-world reasoning, it reports 64.5 on RealWorldQA, compared with 57.3 for Emu3 and 55.8 for JanusFlow (Xiao et al., 23 Sep 2025).
On comprehensive and hallucination-oriented evaluations, the paper reports an MME score of 2352.0 and describes this as a new SOTA among unified open-source models, noting Emu3 at 1590.7 and JanusFlow at 1333.1. It further reports 53.4 on HallusionBench and 90.9 on POPE, compared with 50.6 and 90.6, respectively, for strong baselines in the table (Xiao et al., 23 Sep 2025).
On generation, OmniBridge reports 0.61 overall on GenEval, which is described as the same as Janus and only slightly below JanusFlow and TransFusion at 0.63. On DPG-Bench, it reports 78.93 overall, close to HunyuanDiT at 78.87 and PixArt-$\mathbf{Q}_{\text{img}}$1 at 80.54. The learnable-queries-only variant, denoted OmniBridge$\mathbf{Q}_{\text{img}}$2, reports 0.56 on GenEval without text in conditioning, which the paper uses as evidence that the learned query channel is semantically meaningful rather than merely auxiliary (Xiao et al., 23 Sep 2025).
On editing, OmniBridge reports 2.28 overall on ImgEdit-Bench, compared with 1.90 for MagicBrush, 1.88 for Instruct-P2P, 2.45 for AnyEdit, 2.80 for UltraEdit, and 2.96 for OmniGen. The paper emphasizes the removal subscore of 4.34, which is higher than GPT-4o at 3.66 and higher than all other listed baselines, and an action subscore of 3.64 (Xiao et al., 23 Sep 2025).
On retrieval, OmniBridge reports 92.5 average recall on Flickr30K, compared with 91.8 for InternVL-C, 92.2 for BLIP-2-G, and 91.4 for OpenCLIP-G, with a text→image $\mathbf{Q}_{\text{img}}$3 of 84.8. The paper presents this as evidence that the same latent space can remain competitive for retrieval while also supporting reasoning and generation (Xiao et al., 23 Sep 2025).
The ablation results are also structurally important. For M3CoT, the sequence is 56.3 for Qwen2-VL, 60.2 for OmniBridge without RL, and 62.2 for full OmniBridge. For RealWorldQA, the sequence is 58.2, 62.9, and 64.5. These results are used to argue that Stage 1 improves understanding, while Stage 2 adds latent alignment for generation and retrieval without degrading the understanding backbone because the LLM remains frozen during the second stage (Xiao et al., 23 Sep 2025).
6. Limitations and broader lineage
The paper notes several limitations directly or by implication. OmniBridge slightly underperforms Qwen2-VL on some OCR- and document-heavy tasks such as DocVQA and InfoVQA, which the authors attribute to an emphasis on high-level reasoning in Stage 1. It also acknowledges difficulty in highly detailed, pixel-level editing because the framework operates in a high-level latent space rather than through direct pixel operations. Additional limitations include training on fewer than 100K curated examples, the current focus on image-plus-text rather than temporal modalities such as video and audio within the core alignment framework, and the deployment complexity introduced by combining three backbones: Qwen2-VL, mT5, and HunyuanDiT (Xiao et al., 23 Sep 2025).
In a broader research context, OmniBridge belongs to a family of recent systems that use an explicit “bridge” abstraction to connect heterogeneous modalities through a shared representational hub. OmniCaptioner is described as a general bridge from pixels to language, converting heterogeneous visual inputs into fine-grained, long-context captions that LLMs can operate on purely in text space (Lu et al., 9 Apr 2025). OmniBind addresses arbitrary modality combinations and unequal data scales by aligning student modalities such as touch, thermal, event, point cloud, and audio to a teacher image-text space through Cross-modal Alignment Distillation and Adaptive Fusion (Lyu et al., 2024). OmniDepth bridges monocular and stereo depth reasoning through iterative bidirectional alignment of latent representations, using cross-attentive alignment rather than post-hoc fusion (Guan et al., 6 Aug 2025). HyperCLOVA X 8B Omni unifies text, vision, and audio through a shared next-token prediction interface over an interleaved multimodal sequence, and is described as an any-to-any omnimodal model rather than a task-specific bridge (Team, 5 Jan 2026).
This suggests a broader technical trajectory in which “bridge” systems are increasingly defined by representational interfaces rather than by end-to-end monolithic training. Within that trajectory, OmniBridge is specifically the variant that places a pretrained multimodal LLM at the center and uses latent space alignment to unify understanding, generation, and retrieval without requiring full retraining from scratch (Xiao et al., 23 Sep 2025).