Online Multi-Modal Fusion Techniques

• Online multi-modal fusion is the real-time integration of heterogeneous data streams using methods such as attention mechanisms, transformers, and graph-based models.
• It employs adaptive weighting, personalized query banks, and redundancy reduction to manage asynchronous and incremental data inputs efficiently.
• Applications in recommender systems, robotics, and industrial monitoring demonstrate its effectiveness in achieving low latency and improved scalability compared to offline methods.

Online multi-modal fusion is the real-time or low-latency integration of heterogeneous data streams—such as video, audio, text, wireless signals, or sensor measurements—to produce decision or inference outputs that leverage complementary information across modalities. Unlike batch or offline approaches, online fusion is constrained to process streaming or incrementally arriving data, imposing strict computational, memory, and often latency requirements. Modern online multi-modal fusion exploits deep neural architectures (e.g., attention, transformers, graph-based state models) alongside domain-specific mechanisms for redundancy pruning, personalization, weighted aggregation, and asynchronous updates. This field underpins applications ranging from user-interactive recommender systems and industrial process monitoring to robotics and automotive scene perception. Key frameworks include attention-based fusion modules, recurrent state estimators, and instance-level or category-level hashing schemes, all designed for scalability and adaptability to continuous input.

1. Core Principles of Online Multi-Modal Fusion

Online multi-modal fusion operates under several technical imperatives:

1. Streaming/Incremetal Data Intake: Modalities are ingested as chunks, frames, or time series with potentially differing and asynchronous rates (Deng et al., 2024, Bultmann et al., 2021, Yu et al., 2022).
2. Low-Latency Inference Requirements: Architectures must perform (near) real-time processing—typical latency budgets range from 1–50 ms per request depending on usage scenario (e.g., live streaming recommendation (Deng et al., 2024), UAV semantic fusion (Bultmann et al., 2021)).
3. Complementarity and Redundancy Management: Fusion models must amplify informative, modality-specific cues while suppressing redundant or noisy information—often via orthogonal projections (Deng et al., 2024), instance-specific weights (Zhan et al., 2024), or attention blocks.
4. Personalization and Adaptivity: Some domains require adaptation at the user, author, or instance level (e.g., per-streamer query vector banks (Deng et al., 2024), instance-level weights (Zhan et al., 2024)), supporting fine-grained, context-aware outputs.
5. Incremental/Online Model Updates: With data non-stationarity, label/category drift, or new class arrival, models must support efficient incremental updates (e.g., category-incremental hashing (Zhan et al., 2024)).
6. Task-Specific Output Objectives: Outputs may be real-valued regression (e.g., process monitoring), class/score vectors (semantic mapping), hash codes (retrieval), or sequential recommendations. Training and inference heads are closely tailored to target tasks.

The field differentiates itself from offline fusion by these strict, system-level requirements on latency, scalability, consistency, and domain-specific adaptivity.

2. Fusion Architectures and Methodologies

Online multi-modal fusion is instantiated through several architectural paradigms, each emphasizing different levels of cross-modal interaction, state updating, and weighting:

a. Attention and Orthogonal Projection-Based Fusion

The Multi-modal Fusion Module with Learnable Query (MFQ) in MMBee (Deng et al., 2024) is representative:

• Input representations ($X_v$, $X_s$, $X_t$) for video, speech, and text are projected into a common space, producing token sequences per modality.
• Orthogonal projections are computed via token-wise softmax correlations. For example:

$$\mathrm{Corr}_{vs} = \mathrm{Softmax}\bigl(X_v X_s^T\bigr)$$

Modal tokens are summed with terms from other modalities weighted by their "orthogonal" complement with respect to the anchor:

$$Y_v = X_v + X_s(1-\mathrm{Corr}_{vs}) + X_t(1-\mathrm{Corr}_{vt})$$

• Hierarchical attention: Modalities cross-attend to orthogonalized complements, concatenated and then subjected to global self-attention.
• Per-author learnable query banks probe the fused representation to extract topic- or user-specific features, further refined via intra-query self-attention. These queries are updated by end-to-end backpropagation.

This architecture achieves both low-latency operation and per-entity content personalization.

b. Recurrent and Dynamic-Weight Fusion

In indoor localization (Yu et al., 2022), fusion is framed as combining recurrent feature streams with adaptive, data-driven weighting:

• Each modality is encoded via a dedicated LSTM that evolves memory and hidden state per time tick.
• For fusion, a quality score $u_i^t$ for each modality is computed from recent hidden states, normalized by softmax to form $\{\alpha_i^t\}$.
• Fused feature:

$$h^t_{\text{fusion}} = \sum_{i=1}^M \alpha_i^t h^t_i$$

• The fusion weights $\alpha_i^t$ dynamically down-weight unreliable sources; the process is fully differentiable and operates online.

c. Transformer-Based Encoder-Decoder Fusion

For industrial process forecasting (Li et al., 22 Apr 2025), modality-specific encoders (CNN for video, FC for parameters) map inputs to low-dimensional embeddings. Fusion is performed by concatenating these embeddings as token sequences and passing through a lightweight transformer stack. No explicit cross-modal attention is used beyond self-attention at the fusion stage, yielding sub-100 ms inference per sample.

d. Score-Level and Late Fusion

Real-time UAV semantic fusion (Bultmann et al., 2021) employs late fusion:

• Semantic predictions are made independently by per-modality CNNs (LiDAR, RGB, thermal).
• Softmax class scores from different modalities are linearly combined with tunable or context-sensitive weights.
• Additional refinement integrates detection boxes and temporal smoothing in the score space, followed by log-domain Bayesian update for cumulative voxel-level 3D semantic mapping.

This pipeline is lightweight and robust to asynchronous, heterogeneous sensor rates.

e. Graph-Based State Fusion

The SAGA-KF (Sani et al., 2024) introduces a sensor-agnostic, graph-aware extension of the Kalman filter:

• Scene state is parameterized as a graph $G_t=(V_t,E_t)$ with node and edge attributes.
• Per-modality measurements (e.g., camera detections, LiDAR segments) are registered into a fused measurement graph via matching.
• The state-transition and update equations are made graph-aware, with topology-encoded dependencies (block-structured transition matrices $X_s$0, block-diagonal measurement covariances $X_s$1).
• Real-time operation is achieved by exploiting sparsity; new sensor data immediately triggers online fusion updates.

f. Hashing Fusion with Fine-Grained Instance Weights

HCFW (Zhan et al., 2024) for online retrieval:

• Compact binary codes are derived from category-level semantic embeddings, updated incrementally as new categories are encountered.
• Instance-level codes are composed as sign-projections of category codes and labels.
• Each modality $X_s$2 is assigned a linear projection $X_s$3; fine-grained per-instance weights $X_s$4 are learned to capture modality-specific reliability.
• Query-time fusion uses these learned weights to produce multi-modal hash outputs, all in an online, category-incremental setting.

3. Application Domains and Performance Results

Online multi-modal fusion supports a diverse range of industrial and scientific applications, each placing specific demands on scalability, robustness, and interpretability:

Application	Modalities	Notable Frameworks	Reported Latency or Error
Live streaming recommendations	Vision, speech, text	MFQ (MMBee) (Deng et al., 2024)	1–2 ms/request
Indoor localization	Wi-Fi, UWB, IMU	Multi-stream LSTM + softmax fusion (Yu et al., 2022)	0.07 m median error
Cookie drying process	Video, process parameters	CNN/GRU encoder, Transformer fusion (Li et al., 22 Apr 2025)	15 s MAE, <100 ms latency
UAV semantic mapping	LiDAR, RGB, thermal	Late score/probability fusion (Bultmann et al., 2021)	~9 Hz online
Autonomous tracking	Camera, LiDAR	SAGA-KF (graph-based KF) (Sani et al., 2024)	AMOTA ↑/AMOTP ↓/IDS ↓
Retrieval	Image, text	HCFW hashing with fine weights (Zhan et al., 2024)	0.85 MAP, 1.2 s per training round

For each domain, fusion methods are compared to single-modality or early/late fusion baselines, with consistent improvements in accuracy, robustness to noise/failure, and online feasibility.

4. Challenges and Limitations

While online multi-modal fusion delivers strong empirical gains, several challenges persist:

• Memory and Personalization Overhead: Maintaining per-entity or instance-specific parameters (e.g., author-query banks (Deng et al., 2024), fine-grained weights (Zhan et al., 2024)) incurs $X_s$5 storage, and in cold-start regimes, latent parameters may underfit.
• Fusion Granularity Trade-off: Early fusion (joint feature or token-level) provides rich interaction but can be computationally demanding or prone to redundancy; late fusion (score or logit-level) is lightweight but may lose nuanced cross-modal information (Bultmann et al., 2021, Ji et al., 2023).
• Temporal and Category-Incremental Consistency: Ensuring robustness to temporally evolving or incrementally expanding class spaces is non-trivial, especially for retrieval or classification tasks where drifting code spaces can degrade long-term utility (Zhan et al., 2024).
• Asynchronous, Missing, or Heterogeneous Modalities: Real-world streams may be asynchronous, partially missing, or arrive with different temporal structures. Late fusion and graph-based approaches handle asynchrony robustly; recurrent and attention models require careful masking or temporal alignment (Sani et al., 2024, Yu et al., 2022).
• Offline Preprocessing vs. Online Efficiency: Several solutions offload heavy featurization to offline or upstream modules (e.g., K7-8B in MMBee (Deng et al., 2024)), limiting pure end-to-end online operation.
• Domain Adaptation and Extensibility: Adapting to distribution shifts across environments and domains still requires substantial engineering (e.g., retraining backbone networks or updating calibration parameters).

5. Advancements, Benchmarks, and Comparative Results

Recent contributions have advanced both the methodological rigor and empirical benchmarking of online multi-modal fusion:

• MFQ vs. Classical Fusion: Orthogonal projections and learnable queries (as in MFQ) outperform simple concatenation or vanilla multi-head attention by stripping redundant cross-modal correlations and enabling structure-aware selection (Deng et al., 2024).
• Recurrent Adaptive Weighting: In localization, the online softmax-weighted LSTM fusion achieves order-of-magnitude improvement in error tails over convolutional or one-shot approaches (Yu et al., 2022).
• Transformer-Based Online Fusion: Encoder–decoder designs with no explicit cross-modal attention can achieve competitive accuracy and speed when coupled with modular, token-level fusion pipelines (Li et al., 22 Apr 2025).
• Graph-Based State Fusion: SAGA-KF demonstrates fewer ID switches and improved multi-object tracking accuracy by formally embedding scene topology in the KF filtering dynamics (Sani et al., 2024).
• Fine-Grained Instance Fusion: Instance-level weighting in online hashing markedly improves MAP in both IID and category-incremental regimes, outperforming both batch and earlier online baselines (Zhan et al., 2024).

Ablation studies across models consistently show that omitting adaptive weighting, personalized or instance-specific mechanisms, or online co-distillation (when used) leads to 5–30% drops in accuracy depending on domain and metric (Deng et al., 2024, Zhan et al., 2024, Yu et al., 2022, Ji et al., 2023).

6. Extensions and Future Directions

Key directions for future research in online multi-modal fusion include:

• Learnable Interaction Topologies: Replacing fixed graph edge weights or transition functions with neural predictors (e.g., GNNs for $X_s$6 in SAGA-KF (Sani et al., 2024)).
• Explicit Cross-Modal Attention and Hierarchical Fusion: Integrating cross-modal attention layers within transformer or recurrent stacks to better capture higher-order interactions—currently only present at select stages in deployed systems (Li et al., 22 Apr 2025).
• Robustness to Modality Missingness and Domain Shifts: Development of universal, plug-and-play encoders and unsupervised domain adaptation techniques for unseen environments (Yu et al., 2022, Li et al., 22 Apr 2025).
• Scalable Personalization: Efficient caching, compression, or meta-learning for maintaining per-user/per-entity adaptation at massive scale (Deng et al., 2024).
• End-to-End and On-Device Deployment: Further reduction of model size, FLOPs, and reliance on offline feature extraction for embedded, edge, or resource-constrained systems (Bultmann et al., 2021).
• Physics-Guided and Self-Supervised Extensions: Embedding physical or prior constraints in online fusion loops, and leveraging self-supervision from aggregated predictions or maps (Li et al., 22 Apr 2025, Bultmann et al., 2021).

A plausible implication is that future online multi-modal fusion frameworks will increasingly combine learnable attention or graph modules, modular encoder–decoder stacks, and high bandwidth adaptivity (personalization, instance-level weighting) while balancing rigorously bounded latency and resource footprints.

---

References

• (Deng et al., 2024) MMBee: Live Streaming Gift-Sending Recommendations via Multi-Modal Fusion and Behaviour Expansion
• (Yu et al., 2022) Multi-Modal Recurrent Fusion for Indoor Localization
• (Li et al., 22 Apr 2025) Multi-Modal Fusion of In-Situ Video Data and Process Parameters for Online Forecasting of Cookie Drying Readiness
• (Bultmann et al., 2021) Real-Time Multi-Modal Semantic Fusion on Unmanned Aerial Vehicles
• (Sani et al., 2024) Graph-Based Multi-Modal Sensor Fusion for Autonomous Driving
• (Ji et al., 2023) Online Distillation-enhanced Multi-modal Transformer for Sequential Recommendation
• (Zhan et al., 2024) High-level Codes and Fine-grained Weights for Online Multi-modal Hashing Retrieval