Omni-Fuse: A Family of Fusion Mechanisms

• Omni-Fuse is a family of fusion strategies that dynamically integrates heterogeneous information across scales, modalities, or dimensions.
• It is applied in various domains such as person re-identification, hyperspectral segmentation, and multimodal medical image fusion.
• Empirical evaluations demonstrate that conditional coordination via dynamic gating, query selection, and shared prompts improves performance in diverse tasks.

Searching arXiv for the papers and terminology relevant to “Omni-Fuse”. “Omni-Fuse” denotes a class of fusion mechanisms in which heterogeneous information is integrated pervasively rather than only at a terminal aggregation stage. In the works considered here, the term refers most directly to two different constructs: the unified aggregation gate used for omni-scale feature learning in OSNet for person re-identification, where “Omni-Fuse” corresponds to the model’s dynamic multi-scale fusion mechanism (Zhou et al., 2019), and the explicitly named spatial-spectral segmentation network “Omni-Fuse” for medical hyperspectral imaging, where fusion is distributed across feature enhancement, query selection, and decoding (Zhang et al., 9 Jul 2025). Closely related formulations extend the same design logic to all-in-one medical image fusion under degradations and misalignments in UniFuse (Su et al., 28 Jun 2025) and to parameter-space composition of modality specialists in omni-modal retrieval with Conan-embedding-v3 (Li et al., 8 Jun 2026). Across these usages, the recurring theme is that fusion is treated as an organizing principle for representation learning, alignment, and downstream retrieval or segmentation.

1. Terminological scope and conceptual range

The term is not used uniformly across the literature. In "Omni-Scale Feature Learning for Person Re-Identification" (Zhou et al., 2019), “Omni-Fuse” is not a separate standalone module name in the paper’s text; it corresponds to the unified aggregation gate that dynamically fuses multiple convolutional streams of different receptive-field sizes. In "Omni-Fusion of Spatial and Spectral for Hyperspectral Image Segmentation" (Zhang et al., 9 Jul 2025), Omni-Fuse is the formal model name for a spatial-spectral omni-fusion network. In "UniFuse: A Unified All-in-One Framework for Multi-Modal Medical Image Fusion Under Diverse Degradations and Misalignments" (Su et al., 28 Jun 2025), the paper does not define a separate formal module called Omni-Fuse, but it explicitly presents a “unified all-in-one” and “general fusion framework” that integrates alignment, restoration, and fusion in a single-stage design.

A concise comparison is useful because the same term spans different technical objects.

Context	What is fused	Mechanism
OSNet	Multi-scale spatial streams	Unified aggregation gate
Omni-Fuse for MHSI	Spatial and spectral features	CFE, SQS, two-stage decoder
UniFuse	Alignment, restoration, and fusion	DAPL, OUFR, FA, UFRF
Conan-embedding-v3	Modality specialists in parameter space	Decouple--fuse--recover

A common misconception is that Omni-Fuse designates one standardized architecture. The cited works do not support that reading. A more accurate characterization is that Omni-Fuse names a family of fusion-centric strategies whose specifics depend on whether the target problem is person re-identification, hyperspectral segmentation, multimodal medical image fusion, or omni-modal retrieval.

2. Omni-scale fusion in OSNet

In OSNet, the fusion mechanism appears inside an omni-scale residual bottleneck block composed of multiple parallel streams, each designed to capture a different spatial scale (Zhou et al., 2019). The baseline residual mapping is

$$\bm{y} = \bm{x} + \tilde{\bm{x}}, \quad \text{s.t.} \quad \tilde{\bm{x}} = F(\bm{x}),$$

where $F$ is a Lite 3×3 layer. Multi-scale learning is introduced by stacking $t$ Lite 3×3 layers in $F^t$, yielding a receptive field of size $(2t+1)\times(2t+1)$. The residual branch becomes

$$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$

with $T=4$ in the main OSNet, so the largest receptive field is 9×9.

The critical fusion equation is

$$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$

where $G(\bm{x}^t)$ is produced by the unified aggregation gate. The gate is a small shared mini-network comprising global average pooling, an MLP with one ReLU hidden layer, and sigmoid activation. It outputs a channel-wise vector rather than a single scalar, so the weighting is fine-grained at the channel level. Because the gate is conditioned on the current input image, the fusion is dynamic rather than fixed after training.

The architectural rationale is explicitly framed in terms of homogeneous and heterogeneous scales. Different streams first produce homogeneous-scale features: small-scale streams capture local details such as shoes, logos, glasses, and accessories; medium-scale streams capture body parts or local context such as torso plus emblem; large-scale streams capture global body appearance and full-person context. Heterogeneous-scale features then arise through learned, input-dependent stream fusion, allowing the network either to emphasize one scale or to “pick and mix” several.

The gate is shared across streams within a block. The paper writes the gradient with respect to the shared gate as

$$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$

and argues that this pools supervision from all streams while keeping parameter count independent of the number of streams. Efficiency is further supported by the use of pointwise and depthwise convolutions in the order pointwise $F$0 depthwise, rather than depthwise $F$1 pointwise.

The ablations tie the Omni-Fuse interpretation directly to performance. With more streams, performance improves from single stream ($F$2) at R1 86.5 / [mAP](https://www.emergentmind.com/topics/mean-average-precision-map) 67.7 to four streams plus unified AG at 93.6 / 81.0. For $F$3, unified AG outperforms concatenation (91.4 / 77.4), addition (92.0 / 78.2), and separate AGs (92.9 / 80.2). Channel-wise weighting is better than stream-wise weighting (93.6 / 81.0 versus 92.6 / 80.0), and dynamic gates outperform fixed gates (93.6 / 81.0 versus 91.6 / 77.5). These results establish that the fusion mechanism is not merely multi-stream aggregation; it is specifically scale-diverse, channel-wise, and input-adaptive.

3. Spatial-spectral Omni-Fuse in medical hyperspectral segmentation

In medical hyperspectral imaging, the explicitly named Omni-Fuse model addresses binary segmentation from a hyperspectral cube

$F$4

where $F$5 is the spatial resolution and $F$6 is the number of spectral bands (Zhang et al., 9 Jul 2025). The target is

$F$7

for tumor versus non-tumor regions. The paper emphasizes three difficulties: high dimensionality, spectral redundancy, and nontrivial cross-dimensional alignment between spatial and spectral information.

The model is a dual-stream spatial-spectral network with three main fusion components. The cross-dimensional feature enhancement module refines spatial and spectral features through bidirectional attention. The spectral-guided spatial query selection module selects the spatial tokens most correlated with spectral information. The two-stage cross-dimensional decoder performs coarse-to-fine segmentation and uses a mask-refinement stage to better localize tumor boundaries. The paper’s central claim is that fusion should be omnipresent: in feature enhancement, in query selection, and again in decoding.

Primary feature extraction is split by modality. The spatial branch uses a CNN to produce $F$8, followed by a Swin Transformer encoder yielding $F$9. The spectral branch uses a bidirectional Mamba-based block to model long-range dependencies across spectral bands, producing $t$0, which is flattened into spectral tokens $t$1.

Within the cross-dimensional enhancement module, spatial features are updated with deformable self-attention,

$t$2

while spectral features are updated with self-attention,

$t$3

Bidirectional cross-attention then exchanges information:

$t$4

$t$5

The resulting interpretation is direct: spatial features become spectral-aware, spectral features become spatial-aware, and both are refined through mutual conditioning.

After enhancement, the model tokenizes both modalities and computes spectral-guided spatial query selection:

$t$6

This selects the spatial queries with the strongest spectral correspondence. The two-stage decoder then uses these queries to generate a coarse mask $t$7, followed by mask refinement with

$t$8

where $t$9 uses 3 layers of mask attention, and a linear layer predicts the final refined mask $F^t$0.

Training uses a combination of pixel-wise classification loss and Dice loss,

$F^t$1

with $F^t$2 and $F^t$3, and a two-stage total loss

$F^t$4

with $F^t$5.

Empirical evaluation is reported on the MDC dataset with 538 scenes and 60 spectral bands over 450 nm to 750 nm, and on the GPCC dataset with 600 scenes and 40 spectral bands, both resized to 256 × 256 and split 3:1:1 with patient-centric hard splits. Omni-Fuse achieves the best reported results on both datasets.

Dataset	DSC / IoU / HD
MDC	84.12 / 69.39 / 54.84
GPCC	79.78 / 66.84 / 72.96

The paper reports DSC improvements of +5.73% on MDC and +4.02% on GPCC over the second-best method, QSQL-FL, and all DSC p-values are below 0.05. Ablations show MDC DSC rising from 70.54 in a stripped baseline to 84.12 in the full model, with identifiable gains from CNN, Mamba, CFE, SQS, SSD, and mask refinement. Replacing the two-stage decoder with the SAM decoder reduces MDC performance from 84.12% to 79.39%, and using only pseudo-color images instead of hyperspectral information reduces DSC by 9.74% on MDC and 8.21% on GPCC. The representational analysis further reports reduced spectral redundancy from 0.5755 → 0.46328 on MDC and 0.6450 → 0.5154 on GPCC.

4. All-in-one medical image fusion as an Omni-Fuse formulation

UniFuse extends the fusion-centered design principle to multimodal medical image fusion under diverse degradations and misalignments (Su et al., 28 Jun 2025). The motivating claim is that conventional multimodal fusion typically assumes high-quality, pixel-wise aligned source images, whereas real data exhibit motion artifacts, metal artifacts, noise, low-dose PET noise, rigid and non-rigid misalignment, and strong modality differences among MRI, CT, and PET. Traditional pipelines are staged as restoration, then registration, then fusion. UniFuse instead integrates these tasks in a single framework.

The architecture has four core parts: Degradation-Aware Prompt Learning (DAPL), Omni Unified Feature Representation (OUFR), Feature Alignment (FA), and Universal Feature Restoration and Fusion (UFRF). The input pair is $F^t$6, where $F^t$7 is degraded and $F^t$8 is a high-quality reference image from the other modality. DAPL uses a shared encoder to produce $F^t$9 and $(2t+1)\times(2t+1)$0, forms a patch sequence $(2t+1)\times(2t+1)$1, and then reorders it into forward, backward, and bidirectional sequences. These are processed by Mamba blocks to produce $(2t+1)\times(2t+1)$2, which encode global degradation-related information from different directional views.

A degradation classifier predicts

$(2t+1)\times(2t+1)$3

with cross-entropy loss

$(2t+1)\times(2t+1)$4

and the hidden states are transformed into a prompt-selection matrix $(2t+1)\times(2t+1)$5 used to select prompts from $(2t+1)\times(2t+1)$6 via

$(2t+1)\times(2t+1)$7

The same prompt is shared by both alignment and restoration/fusion, creating an explicit bridge between these tasks.

OUFR uses Spatial Mamba to reduce modality differences during alignment. For the $(2t+1)\times(2t+1)$8-th Spatial Mamba block, the output is

$(2t+1)\times(2t+1)$9

where $$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$0. A contrastive modality-alignment loss is introduced:

$$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$1

with temperature $$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$2. FA then predicts a deformation field and is regularized by

$$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$3

The all-in-one character is most explicit in UFRF. After warping, the input becomes

$$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$4

UFRF is a multi-scale U-Net-like architecture with ALSN embedded in the skip connections. At scale $$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$5,

$$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$6

where $$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$7 is the base network and $$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$8 is a LoRA branch for degradation type $$\tilde{\bm{x}} = \sum_{t=1}^T F^t(\bm{x}), \quad T \geq 1,$$9. The LoRA analogy is written as

$T=4$0

and extended to multiple branches by

$T=4$1

Training jointly optimizes degradation classification, modality-unified representation, alignment, restoration, and fusion with

$T=4$2

The reported datasets are BraTs2020 with 494 aligned pairs, SynthRAD2023 with 360 aligned pairs, and FDG-PET/CT with 2939 aligned pairs, all trained end-to-end for 500 epochs with batch size 3, resized to $T=4$3, using AdamW with learning rate $T=4$4, cosine annealing to $T=4$5, and gradient clipping at $T=4$6-norm 1 on a single RTX 4090. UniFuse reports the best performance across datasets, with FLOPs of 395 G on BraTS2020 and SynthRAD2023. On BraTS2020 the reported scores are $T=4$7, $T=4$8, $T=4$9, $$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$0, and $$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$1; on SynthRAD2023 they are $$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$2, $$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$3, $$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$4, $$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$5, and $$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$6; on FDG-PET/CT they are $$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$7, $$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$8, $$\tilde{\bm{x}} = \sum_{t=1}^T G(\bm{x}^t)\odot \bm{x}^t, \quad \text{s.t.} \quad \bm{x}^t \triangleq F^t(\bm{x}),$$9, $G(\bm{x}^t)$0, and $G(\bm{x}^t)$1. The paper also states that ALSN reduces UFRF parameters by 68.22%, from 2.986M to 0.948M.

5. Parameter-space fusion and projector repair in omni-modal retrieval

Conan-embedding-v3 does not use the label Omni-Fuse, but it provides a closely related formulation in which fusion occurs directly in parameter space rather than in feature space (Li et al., 8 Jun 2026). The target is a single embedding space for text, image, video, document, and audio. The embedding system maps input $G(\bm{x}^t)$2 into an $G(\bm{x}^t)$3-normalized vector,

$G(\bm{x}^t)$4

and retrieves by inner-product similarity. For paired query–target examples $G(\bm{x}^t)$5, the retrieval loss is

$G(\bm{x}^t)$6

The model is built in three stages: independently training specialists for image, video, visual-document, and audio retrieval; fusing the shared backbone updates using task-vector arithmetic while copying audio-only modules directly; and repairing the audio mismatch through Projector Recovery followed by balanced rehearsal. If the base model is $G(\bm{x}^t)$7, each specialist is trained as

$G(\bm{x}^t)$8

and the task vector for shared parameters is

$G(\bm{x}^t)$9

The fused backbone is then

$$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$0

with coefficients $$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$1 and $$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$2. Audio-only modules satisfy

$$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$3

The central failure mode is Projector Drift. Audio is introduced through an external audio encoder and projector, so the audio projector $$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$4 is trained against an audio-specialist backbone $$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$5, with $$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$6. After fusion, however, inference uses

$$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$7

The audio encoder and projector can be copied unchanged and yet audio retrieval still collapses. The diagnosis is that structural completeness does not imply representational compatibility: the projector remains calibrated to the geometry of the audio-specialist backbone, while the merged backbone has moved in parameter space.

The diagnostics are specific. Audio task vectors are nearly orthogonal to visual updates with $$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$8. Audio has the largest global update norm with $$\frac{\partial \mathcal{L}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \frac{\partial \tilde{\bm{x}}}{\partial G} = \frac{\partial \mathcal{L}}{\partial \tilde{\bm{x}}} \left(\sum_{t=1}^T \bm{x}^t\right),$$9. Audio updates remain large across layers and peak in deeper layers. A t-SNE plot on audio-text pairs reports $F$00 for the audio specialist, $F$01 after direct fusion, and $F$02 after recovery.

Projector Recovery freezes the fused backbone and audio encoder and updates only the audio projector. In the ablation table, projector-only tuning restores MAEB from 32.68 to 55.82 while keeping image, video, and visual-document performance the same as direct fusion. Balanced multi-modal rehearsal then uses lightweight LoRA on the backbone and vision encoder with a lower learning rate and limited steps of about 2000, lifting visual scores to 77.2 image, 65.1 video, and 79.0 visual-document while keeping MAEB at 55.61. On MMEB-V2, which includes 78 tasks, the final model reports 74.96 overall MMEB, 77.20 image, 65.10 video, and 79.00 visual-document. The model is also evaluated on the 30-task MAEB audio suite and reports 55.61 MAEB with 59.32 task-type average. The merging-method ablation further shows that TIES and DARE do not solve the audio drift before recovery.

6. Shared design principles, empirical patterns, and limitations

Across these works, fusion is repeatedly implemented as a learned coordination mechanism rather than a simple post hoc concatenation. In OSNet, the defining property is dynamic per-input channel-wise fusion of multi-scale streams (Zhou et al., 2019). In the hyperspectral Omni-Fuse model, fusion is abundant and iterative across enhancement, query selection, and decoding rather than deferred to a late stage (Zhang et al., 9 Jul 2025). In UniFuse, a shared degradation-aware prompt explicitly couples alignment with restoration and fusion, so joint optimization replaces staged processing (Su et al., 28 Jun 2025). In Conan-embedding-v3, fusion composes independently trained specialists in parameter space, but projector-based modalities reveal that fusion can fail at the interface between modality-specific modules and a shared backbone (Li et al., 8 Jun 2026).

These patterns support several constrained interpretations. One is that Omni-Fuse methods tend to treat heterogeneity as structured rather than incidental: scales in OSNet, spatial and spectral dimensions in MHSI, degradation categories and modality gaps in UniFuse, and modality-specific backbones plus projectors in omni-modal retrieval. Another is that fusion is often paired with an explicit control signal: input-dependent gates, spectral-guided query selection, degradation prompts, or projector recovery. This suggests that effective fusion in heterogeneous systems often requires some mechanism for conditional coordination rather than uniform averaging.

The literature also identifies clear limitations. The hyperspectral Omni-Fuse paper evaluates only binary segmentation on two MHSI datasets, and one of them, GPCC, is private; it also does not provide detailed FLOPs or latency comparisons in the supplied description (Zhang et al., 9 Jul 2025). UniFuse is general within the paper’s multimodal medical setting, but it is not presented as a universal fusion model for every imaging regime (Su et al., 28 Jun 2025). Conan-embedding-v3 explicitly notes a slight regression relative to the strongest visual-only Qwen3-VL-Embedding-8B while emphasizing the value of adding audio to a unified model (Li et al., 8 Jun 2026). OSNet’s Omni-Fuse interpretation is likewise task-specific: its fusion mechanism is optimized for person re-identification rather than for arbitrary multimodal or cross-dimensional fusion (Zhou et al., 2019).

Taken together, the arXiv literature portrays Omni-Fuse not as a single canonical module, but as a recurrent architectural doctrine: build specialized representations, expose their complementarities, and use learned mechanisms to coordinate them without erasing the structure that makes each source informative.