OmniAlignNet: Cross-Modal Alignment Module

• OmniAlignNet is a dedicated architectural module that aligns visual and audio embeddings into a shared semantic space using modality-specific query vectors and self-attention layers.
• It employs a symmetric, CLIP-style contrastive loss that brings corresponding features together while separating non-matching pairs to enforce semantic consistency.
• Its design reduces modality-specific hallucinations and boosts efficiency, achieving notable performance improvements on cross-modal benchmarks with fewer training tokens.

OmniAlignNet is a dedicated architectural module for aligning vision and audio feature embeddings into a joint latent space, designed as a core innovation for cross-modal understanding within the OmniVinci omni-modal LLM. Its principal function is to harmonize high-level semantic information arising from visual and audio modalities, optimally preparing multi-modal inputs for unified large-scale language modeling and downstream reasoning.

1. Architectural Principles and Design

OmniAlignNet processes two separate streams of input embeddings: $\mathbf{E}_v \in \mathbb{R}^{N_v \times C}$ for vision and $\mathbf{E}_a \in \mathbb{R}^{N_a \times C}$ for audio, where $N_v$ and $N_a$ denote the number of visual and audio tokens per sequence, and $C$ is the embedding channel dimension. To aggregate variable-length modality-specific sequences into fixed-size semantic representations, OmniAlignNet introduces modality-specific query vectors, $\mathbf{Q}_v \in \mathbb{R}^{1 \times C}$ (vision) and $\mathbf{Q}_a \in \mathbb{R}^{1 \times C}$ (audio), which serve as learnable pooling agents within an attention-based projection scheme.

For each modality, the respective query vector attends over the modality’s embedding sequence, producing a fixed-length, high-level feature vector. Subsequently, these representations are refined via three stacked layers of self-attention, which capture higher-order intra-modal dependencies and context. The outputs are L2-normalized, resulting in batch-collated sets of omni-modal features: $\mathbf{V} \in \mathbb{R}^{K \times C}$ for vision and $\mathbf{A} \in \mathbb{R}^{K \times C}$ for audio, with $K$ being the minibatch size.

2. Contrastive Alignment Objective

At the core of OmniAlignNet is a symmetric, CLIP-style contrastive objective that enforces cross-modal alignment at the semantic level. For a minibatch of size $N$, the model computes pairwise similarities $s_{ij} = \mathbf{V}_i^\top \mathbf{A}_j$ between every pair of vision and audio features. The module then optimizes:

$$\mathcal{L}_{v \rightarrow a} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(s_{ii})}{\sum_{j=1}^N \exp(s_{ij})}$$

$$\mathcal{L}_{a \rightarrow v} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(s_{ii})}{\sum_{j=1}^N \exp(s_{ji})}$$

The final contrastive loss is the mean: $$\mathcal{L}_{o\text{-}align} = \frac{1}{2}(\mathcal{L}_{v \rightarrow a} + \mathcal{L}_{a \rightarrow v})$$ This loss pulls together the representations from the same video sample and pushes apart those from different samples, driving the model to construct a modality-invariant semantic embedding space.

3. Functional Role in Omni-Modal LLMs

OmniAlignNet supplies the foundational joint vision-audio embeddings to downstream modules in OmniVinci, facilitating a unified interpretation of multimodal inputs. This has the following direct consequences:

• The LLM backbone receives a coherent “token” sequence, irrespective of input modality, allowing it to learn, reason, and perform inference over genuinely fused cross-modal signals.
• Modality-specific hallucinations (errors stemming from treating input streams in isolation) are reduced, as the combinatorial alignment enforces a semantic consensus between modalities.
• High-level semantic reinforcement across modalities emerges (e.g., visual cues clarified by audio or vice versa), which is critical for downstream understanding and decision tasks.

A plausible implication is that such fusion yields not only higher accuracy on cross-modal benchmarks—such as DailyOmni, MMAR, and Video-MME—but also greater efficiency, as seen in OmniVinci’s ability to outperform comparably scaled models while employing significantly fewer training tokens.

4. Innovations and Distinct Architectural Features

Several distinctive design choices differentiate OmniAlignNet from alternate multi-modal fusion mechanisms:

• Query-based projection: Rather than relying on concatenation or pooling, learned query vectors enable each modality to be distilled into a compact, context-sensitive summary.
• Dedicated self-attention blocks: By inserting three layers of self-attention after projection, the module deepens intramodal context propagation before cross-modal fusion.
• Symmetric cross-modal loss: The bidirectional nature of the contrastive loss ensures that both modalities contribute equally to the shared latent space, preventing dominance of one modality.
• Orthogonality to temporal alignment: While OmniAlignNet is tasked with semantic alignment, temporal alignment techniques (e.g., Temporal Embedding Grouping, Constrained Rotary Time Embedding) are separately implemented, allowing for modular handling of temporal logic downstream.

5. Benchmark Performance and Practical Impact

Ablations reveal that introducing OmniAlignNet within OmniVinci results in significant gains on cross-modal tasks, with quantified improvements of +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio-centric tasks), and +3.9 on Video-MME (visual understanding) over comparable omni-modal baselines, achieved with just 0.2T training tokens compared to 1.2T in Qwen2.5-Omni.

The architecture enables practical deployment across several critical real-world domains:

Application Area	Modality Pairing	Task Enabled
Robotics	Visual + Spoken	Speech-driven vision-language navigation
Medical AI	Imaging + Narrative	Radiology video–report generation, diagnostics
Smart Factory	Image + Audio	Defect analysis, SPC chart recognition, QA

This supports a broad range of use cases requiring robust fusion and reasoning over vision and audio signals.

6. Future Directions and Integration Scope

OmniAlignNet’s projection-alignment paradigm is extendable to other modality pairings, suggesting future applicability in any context requiring shared semantic space construction. Possible directions include:

• Integration of additional modalities (e.g., haptics, text, sensor streams) via analogous projection and contrastive alignment schemes.
• Hierarchical or sequential fusion (staged multi-modality, as required by more complex tasks).
• Applying similar mechanisms to fuse not just modalities, but networks or graphs arising in knowledge organization or scientific data integration.

A plausible implication is that emerging omni-modal and cross-network AI systems will increasingly rely on modules structurally analogous to OmniAlignNet to harmonize diverse input streams and ensure unified semantic representations across domains.