Fused Multi-Modal Embeddings (PROFUSEme)

• Fused multi-modal embeddings (PROFUSEme) are unified latent representations that combine clinical, radiology, and pathology data into a shared inference state.
• They employ diverse fusion strategies, including intermediate, progressive, and token-level methods, to balance cross-modal interactions with computational efficiency.
• These methods have demonstrated improved predictive performance across applications from prostate cancer survival analysis to sentiment analysis and segmentation.

Searching arXiv for the cited papers to ground the article. Fused multi-modal embeddings are latent representations constructed from two or more data modalities so that downstream models can operate on a shared representation rather than on isolated unimodal streams. In the narrow, explicit sense, “PROFUSEme” names a prostate cancer survival model that fuses clinical, radiology, and pathology embeddings through an intermediate-fusion Transformer and a Cox proportional hazards head (You et al., 17 Sep 2025). In a broader sense, the label “PROFUSEme” (Editor’s term) is a convenient abstraction for a family of methods that produce a single shared embedding, token set, or fused decision functional from heterogeneous modalities, even when the original papers use other names such as Progressive Fusion, SFusion, FuseLIP, or AGFF-Embed (Shankar et al., 2022, Liu et al., 2022, Schlarmann et al., 3 Jun 2025, Hu et al., 5 Feb 2026).

1. Terminology, scope, and definitional variants

The literature does not use a single canonical term for this object. The prostate-cancer system “PROFUSEme: PROstate Cancer Biochemical Recurrence Prediction via FUSEd Multi-modal Embeddings” uses the name explicitly for a fused patient-level embedding connected to survival analysis (You et al., 17 Sep 2025). By contrast, “Progressive Fusion for Multimodal Integration” states that the paper refers to the method as Progressive Fusion, also abbreviated and variably spelled as Pro-Fusion or ProFusion, and that the name “PROFUSEme” does not appear there (Shankar et al., 2022).

Across papers, the fused object itself takes several forms. In Progressive Fusion it is an iterative shared embedding $h^{(t)}$ updated through repeated fusion and backprojection into unimodal encoders (Shankar et al., 2022). In SFusion it is a shared feature map $f_s$ obtained from available modalities only, without synthesizing or zero-padding missing ones (Liu et al., 2022). In the explicit PROFUSEme survival model it is a masked-mean-pooled Transformer output $z \in \mathbb{R}^{768}$ built from modality tokens for pathology, radiology, and clinical covariates (You et al., 17 Sep 2025). FuseLIP instead emits a single transformer embedding at the <eot> position from a unified sequence of text and discrete image tokens (Schlarmann et al., 3 Jun 2025). AGFF-Embed is a useful counterexample because its core fusion is at the similarity level rather than at the embedding level: it aggregates multiple similarity terms with a smooth $\log\sum\exp$ operator rather than defining one primary fused vector (Hu et al., 5 Feb 2026).

A common misconception is that a fused multi-modal embedding must always be a single concatenated vector. The surveyed systems show otherwise. The fused object may be a vector, a token stack, a voxel- or pixel-aligned map, or a similarity functional, depending on the downstream objective and the locus of fusion in the architecture (Liu et al., 2022, Schlarmann et al., 3 Jun 2025, Hu et al., 5 Feb 2026).

2. Fusion paradigms and representational regimes

The main design axis is where interaction occurs. Progressive Fusion formulates the classic early-versus-late tradeoff directly: early fusion can model complex cross-modal interactions but suffers from heterogeneous feature spaces, dimensional mismatch, and high sample complexity, whereas late fusion preserves modularity and pretraining compatibility but risks discarding information that becomes relevant only jointly across modalities (Shankar et al., 2022). Its central update,

$$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$

defines a progressive regime in which the late fused representation is repeatedly projected back into earlier unimodal processing.

The explicit PROFUSEme survival model adopts intermediate fusion. Pathology, radiology, and clinical representations are first reduced to patient-level vectors $V_P$, $V_R$, and $V_C$, each modality vector is linearly projected to a $1 \times 768$ token, learnable positional encodings are added, and a compact 4-layer Transformer encoder performs self-attention across the three modality tokens before masked mean pooling yields the fused embedding $z$ (You et al., 17 Sep 2025). This is neither pure early fusion nor classical late ensembling; it is a token-level cross-modal interaction layer inserted after modality-specific feature extraction but before prediction.

SFusion represents a different regime: N-to-One fusion under variable modality availability. Available modality feature maps are tokenized, passed through stacked self-attention layers to extract inter-modal correlations, and then aggregated with a modality-wise voxel-level softmax into a shared embedding

$f_s$0

where the softmax is defined only over the available set $f_s$1 (Liu et al., 2022). This construction makes the fused representation missingness-aware by design.

FuseLIP pushes fusion to the earliest possible token level. Images are discretized into 128 visual tokens, text tokens are appended in a single sequence, and one bidirectional transformer processes the joint vocabulary so that cross-modal interaction occurs at every layer of encoding rather than only after unimodal features have been formed (Schlarmann et al., 3 Jun 2025). By contrast, AGFF-Embed shows that hybrid perceptual matching can require simultaneous global and fine-grained comparisons, so it computes global-to-global, fine-to-global, global-to-fine, and fine-to-fine similarities and fuses them via

$f_s$2

a formulation chosen specifically to remain compatible with hard-negative amplification (Hu et al., 5 Feb 2026).

These formulations suggest that “fused multi-modal embedding” is best understood as a representational regime rather than a single architectural primitive. The decisive question is not merely whether modalities are combined, but whether the combination produces a shared object that downstream inference treats as primary.

3. Architectural mechanisms for constructing fused representations

Several recurrent mechanisms appear across the literature. Progressive Fusion uses backward projections from the fused state into each modality encoder, usually with residual addition or concatenation plus projection. The same modality encoders and fusion head are reused across iterations, parameter growth is limited, and the authors report that gains saturate at small refinement depth, typically $f_s$3 (Shankar et al., 2022). This yields a message-passing interpretation in which cross-modal context reaches early layers before irreversible pooling.

SFusion decomposes fusion into a correlation extraction module and a modal attention module. The correlation extraction stage is a Transformer-style self-attention stack over tokens derived from the available modalities; the modal attention stage then computes voxel-level weights over modalities and produces the final shared representation from the original upstream features, preserving scale because the modality weights sum to $f_s$4 at each voxel (Liu et al., 2022). An important edge case is explicitly handled: when only one modality is present, $f_s$5 and the fused output reduces to that modality’s features.

Other architectures emphasize structurally asymmetric interaction. “Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion” uses a shared single network with modality-specific batch normalization in the encoder and introduces parameter-free, bidirectional fusion operations—channel shuffle and pixel shift—at multiple layers (Wang et al., 2021). Channel shuffle exchanges channel segments between modalities; pixel shift adds spatially shifted features from the opposite modality. The paper argues that symmetric fusion blocks are “not very compatible with the bidirectional fusion scheme,” and the asymmetric operations are introduced specifically to avoid redundant representations (Wang et al., 2021).

Refiner-style systems act not by changing the fusion operator itself but by constraining the latent fusion space. Refiner Fusion Network attaches per-modality decoders to the fused latent and penalizes the cosine discrepancy between each reconstructed modality surrogate and the original modality feature, enforcing what the paper calls a modality-centric responsibility condition (Sankaran et al., 2021). A plausible implication is that such a constraint is especially relevant when the fused representation is expected to support both multimodal reasoning and unimodal fallback.

At larger scale, omni-modal retrieval systems use parameter-space rather than token-space fusion. Conan-embedding-v3 trains modality specialists independently from a shared initialization, fuses shared backbone task vectors with fixed coefficients $f_s$6 and $f_s$7, copies audio-only modules directly, and then repairs the projector interface through Projector Recovery before balanced rehearsal (Li et al., 8 Jun 2026). Here the fused embedding space emerges after weight-space composition rather than from explicit token concatenation.

4. Objectives, supervision, and representation shaping

The objective function determines what information the fused representation is forced to preserve. In the explicit PROFUSEme model, the fused patient embedding is optimized through the Cox proportional hazards negative partial log-likelihood,

$f_s$8

with the linear risk head $f_s$9 mapping the fused embedding to log-risk (You et al., 17 Sep 2025). This makes the embedding survival-aware rather than merely discriminative.

Progressive Fusion is task-agnostic and uses the final fused state $z \in \mathbb{R}^{768}$0 for either classification or regression. The paper gives standard cross-entropy for classification and $z \in \mathbb{R}^{768}$1 regression loss for time-series forecasting, with task loss applied after unrolling the refinement steps (Shankar et al., 2022). SFusion is likewise trained end-to-end with downstream task losses, specifically cross-entropy for human activity recognition and Dice-based segmentation losses, while its internal self-attention and modal attention machinery remain entirely differentiable (Liu et al., 2022).

Contrastive learning dominates retrieval-oriented formulations. FuseLIP combines a SigLIP-style sigmoid contrastive objective over unimodal or multimodal pairs with masked multimodal modeling on the same masked input, weighted by $z \in \mathbb{R}^{768}$2 (Schlarmann et al., 3 Jun 2025). Conan-embedding-v3 uses CLIP-style InfoNCE with $z \in \mathbb{R}^{768}$3-normalized embeddings and temperature $z \in \mathbb{R}^{768}$4 across specialist training, projector recovery, and rehearsal (Li et al., 8 Jun 2026). AGFF-Embed uses contrastive training over its similarity-level fused score and augments it with Explicit Gradient Amplification, where hard negatives are reweighted according to

$z \in \mathbb{R}^{768}$5

with $z \in \mathbb{R}^{768}$6 in the reported setting (Hu et al., 5 Feb 2026).

Self-supervision can make the fused space more data-efficient. ProFusion3D introduces self-supervised mask modeling across PV and BEV views with masked token reconstruction, denoising, and cross-modal attribute prediction, and shows improved label efficiency on nuScenes (Mohan et al., 2024). DeepEarth trains fused modality and Earth4D embeddings through masked reconstruction inside an autoencoding world model, although the paper does not enumerate a full loss decomposition beyond that high-level description (Legel et al., 7 Mar 2026). These examples suggest that fused embeddings increasingly function as generative state variables rather than only as discriminative summary vectors.

5. Empirical domains and reported behavior

The explicit PROFUSEme application is biochemical recurrence prediction after radical prostatectomy. On internal 5-fold nested cross-validation, the intermediate-fusion model over clinical, pathology, and radiology modalities achieved a mean C-index of $z \in \mathbb{R}^{768}$7 with $z \in \mathbb{R}^{768}$8, and it reported a C-index of $z \in \mathbb{R}^{768}$9 on the hold-out CHIMERA 2025 validation leaderboard (You et al., 17 Sep 2025). The same study reports that the best late-fusion three-modality baseline reached $\log\sum\exp$0, while radiology alone underperformed but still contributed positively when fused (You et al., 17 Sep 2025).

Progressive Fusion was evaluated on AV-MNIST, CMU-MOSI, CMU-MOSEI, and multimodal financial time series. Selected results include LFLSTM on HEALTH with MSE improving from $\log\sum\exp$1 to $\log\sum\exp$2 and robustness from $\log\sum\exp$3 to $\log\sum\exp$4, LF-Transformer on FB with MSE from $\log\sum\exp$5 to $\log\sum\exp$6 and robustness from $\log\sum\exp$7 to $\log\sum\exp$8, and overall reports of up to $\log\sum\exp$9 reduction in MSE and $$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$0 improvement in robustness over late-fusion baselines (Shankar et al., 2022). The paper also reports consistent sentiment gains, for example MIM on MOSI improving from $$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$1 to $$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$2 in Acc$$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$3 and from $$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$4 to $$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$5 in Acc$$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$6 (Shankar et al., 2022).

SFusion was tested on SHL2019 human activity recognition and BraTS brain-tumor segmentation. On validation HAR, overall accuracy increased from $$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$7 with EmbraceNet to $$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$8 with SFusion, while ablations showed major degradation when the correlation extraction module or the modal attention module was removed (Liu et al., 2022). In BraTS2020, average Dice improved over GFF in WT from $$h^{(t)} = F([e_1^{(t)}, \ldots, e_K^{(t)}]), \qquad e_k^{(t+1)} = E_k(x_k \oplus \phi_k(W_k h^{(t)})),$$9 to $V_P$0, in TC from $V_P$1 to $V_P$2, and in ET from $V_P$3 to $V_P$4 (Liu et al., 2022).

FuseLIP, which targets truly multimodal embeddings rather than post hoc fusion, reports strong gains on multimodal retrieval-style tasks. Under CC3M+MM training, FuseLIP-B reached $V_P$5 on text-guided image transformation retrieval and $V_P$6 on OI-Pos; under CC12M+MM training, FuseLIP-B reported $V_P$7 on TGIT and $V_P$8 on the MMEB grounding category (Schlarmann et al., 3 Jun 2025). Conan-embedding-v3 addresses an omni-modal retrieval setting spanning text, image, video, visual documents, and audio, reaching $V_P$9 overall on MMEB-V2 and $V_R$0 on MAEB after decouple–fuse–recover training with projector repair (Li et al., 8 Jun 2026).

The breadth of these results matters. Fused embeddings are not confined to one modality pair or one task family: they appear in prognosis, sentiment analysis, time-series forecasting, segmentation, retrieval, recommendation, world modeling, and autonomous driving. This breadth suggests that the decisive abstraction is the shared representational object rather than any fixed backbone.

6. Limitations, misconceptions, and open directions

A first limitation is terminological. “PROFUSEme” is not a universal field-wide name; outside the prostate-cancer paper, several cited systems do not use the term explicitly, and at least one source states this directly (Shankar et al., 2022, You et al., 17 Sep 2025). Treating it as a universal method name obscures the fact that the literature contains multiple non-equivalent fusion formalisms.

A second limitation is computational. Progressive Fusion scales roughly linearly with refinement depth $V_R$1, SFusion inherits the $V_R$2 burden of self-attention over modality tokens, and early-fusion token systems such as FuseLIP trade training efficiency for higher inference cost than score-fusion baselines because images must be tokenized and encoded in a single joint sequence (Shankar et al., 2022, Liu et al., 2022, Schlarmann et al., 3 Jun 2025). ProFusion3D similarly adds cross-view mappings, multi-stage fusion blocks, and multiple decoders (Mohan et al., 2024).

A third limitation is modality imbalance and interface fragility. SFusion can struggle when a noisy modality dominates attention weights; Progressive Fusion warns about over-conditioning by dominant modalities and recommends gating or smaller backprojections for such cases; Conan-embedding-v3 identifies Projector Drift as a specific failure mode in which a projector remains calibrated to a specialist backbone after backbone fusion, causing severe audio regression until projector-only recovery is performed (Liu et al., 2022, Shankar et al., 2022, Li et al., 8 Jun 2026). These are not peripheral issues: they show that fused spaces can fail even when the unimodal components remain strong.

Another common misconception is that missing modalities are automatically solved by fusion. Some systems explicitly address missingness—SFusion by constructing tokens only for available modalities, and the explicit PROFUSEme survival model by mask-aware mean pooling over present modality tokens—but many architectures require additional training strategies such as modality dropout, gated scaling, or sparse backprojection to remain stable under absent inputs (Liu et al., 2022, You et al., 17 Sep 2025, Shankar et al., 2022).

Open directions recur across the papers. Conan-embedding-v3 proposes adaptive fusion coefficients, modality-aware routing, and projector-specific calibration after fusion (Li et al., 8 Jun 2026). Progressive Fusion points to sparse or hierarchical backprojections for many-modality settings (Shankar et al., 2022). AGFF-Embed identifies extension of hard-negative amplification toward perceptual-pattern weights as a future step once global versus fine-grained decomposition stabilizes (Hu et al., 5 Feb 2026). DeepEarth suggests that fused multi-modal embeddings can be organized around precise spatio-temporal positional encoders rather than around modality identity alone, a direction that becomes increasingly relevant for world models and geo-temporal forecasting (Legel et al., 7 Mar 2026).

Taken together, the literature supports a broad but technically precise view: fused multi-modal embeddings are shared representational objects designed to preserve complementary modality information while enabling unified downstream inference. In the explicit PROFUSEme formulation this object is a Transformer-pooled patient embedding optimized with Cox partial likelihood (You et al., 17 Sep 2025). In the broader research landscape, it may be iteratively refined, missingness-aware, token-level early fused, projector-recovered after weight-space merging, or even defined at the similarity level. The unifying principle is not one architecture, but the systematic conversion of heterogeneous modality-specific signals into a shared computational state that downstream models can treat as primary.