Deep Holistic Orthogonal Fusion (DHOF)

• DHOF is a fusion mechanism that decomposes modality-specific and modality-common embeddings using a projection-residual scheme to retain complementary information.
• It reduces redundancy by orthogonally projecting modality-specific features against a shared subspace, leading to more discriminative multimodal representations.
• Empirical results show DHOF improves survival prediction C-index across multiple TCGA cohorts by effectively fusing histopathology and genomic data.

Searching arXiv for the cited DHOF-related papers to ground the article in current records. arXiv search query: "MurreNet Modeling Holistic Multimodal Interactions Between Histopathology and Genomic Profiles for Survival Prediction (Liu et al., 7 Jul 2025)" Deep Holistic Orthogonal Fusion (DHOF) is a fusion mechanism introduced within MurreNet for multimodal cancer survival prediction from histopathology Whole Slide Images (WSIs) and genomic profiles. In that architecture, DHOF receives a refined modality-common embedding $f_c$ and a refined modality-specific embedding $f_s$, projects each specific feature onto the common direction, subtracts the aligned component, aggregates the resulting orthogonal residuals, and maps them to the final multimodal representation $f_{mm}$ used by the survival head. Its stated purpose is to preserve complementary information, reduce redundancy between modalities, and avoid the limitations of straightforward fusion strategies such as simple concatenation or prediction-level late fusion (Liu et al., 7 Jul 2025).

1. Definition, scope, and conceptual motivation

DHOF was introduced because existing multimodal fusion strategies in survival prediction often over-focus on modality-common information, underuse modality-specific information, and mix features in ways that allow redundant overlap to dominate. In MurreNet, these concerns are addressed by first decomposing each modality into modality-specific and modality-common representations and then fusing them through an explicitly orthogonality-structured mechanism rather than by flat concatenation (Liu et al., 7 Jul 2025).

Within MurreNet, DHOF is presented as the fusion counterpart of the Multimodal Representation Decomposition (MRD) module. MRD disentangles the sources of variation by separating pathology-specific, genomics-specific, and shared representations. DHOF then uses the common representation as a shared explanatory subspace and extracts the complementary, non-redundant contribution from the specific representation by removing the component aligned with that shared subspace. The resulting fusion is therefore structured around the distinction between what is common across modalities and what remains uniquely informative after commonality has been accounted for (Liu et al., 7 Jul 2025).

The terminology requires some care. In the MurreNet paper, “Deep Holistic Orthogonal Fusion” is the explicit name of the fusion strategy. In related literature, closely allied ideas appear under different names. DOLG formulates “Deep Orthogonal Local and Global” fusion for image retrieval, where “holistic” is treated as approximately equivalent to “global,” and DOF formulates “Deep Orthogonal Fusion” for multimodal prognosis using attention-gated tensor fusion plus an orthogonalization loss rather than projection-residual fusion (Yang et al., 2021, Braman et al., 2021).

2. Position within the MurreNet architecture

DHOF operates after a sequence of unimodal encoding, multimodal decomposition, early fusion, cross-attention, and Transformer-based refinement. The pathology WSI embedding is

$\mathrm{h}_p^o \in \mathbb{R}^d,$

obtained by CLAM plus a fully-connected layer, and the genomic embedding is

$\mathrm{h}_g^o \in \mathbb{R}^d,$

obtained by grouping genomic signals and a fully-connected layer (Liu et al., 7 Jul 2025).

MRD produces modality-specific representations

$$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$

and modality-common representations through a co-attention encoder $\Gamma_c$: $$\mathrm{h}_p^{c}, \mathrm{h}_g^{c} = \Gamma_c(\mathrm{h}_p^o,\mathrm{h}_g^o).$$ The co-attention is defined through

$$\mathbb{A} = \text{Linear}(\mathrm{h}_p^o)^{\top} \cdot \text{Linear}(\mathrm{h}_g^o),$$

$$\mathrm{h}_{p}^{c} = \text{MLP}(\mathbb{A}^\top) \ast \mathrm{h}_p^o,\qquad \mathrm{h}_{g}^{c} = \text{MLP}(\mathbb{A}) \ast \mathrm{h}_g^o,$$

where $f_s$0 denotes element-wise multiplication (Liu et al., 7 Jul 2025).

The decomposed outputs are then concatenated into a modality-specific fusion

$f_s$1

and a modality-common fusion

$f_s$2

A cross-attention module integrates these streams: $f_s$3 with

$f_s$4

where $f_s$5 (Liu et al., 7 Jul 2025).

Both streams are subsequently refined by a Transformer decoder $f_s$6 with multi-head self-attention, Nystrom approximation, and PPEG positional encoding. For the specific stream: $f_s$7

$f_s$8

$f_s$9

The same decoder produces the enhanced common representation $f_{mm}$0 from $f_{mm}$1. DHOF therefore receives already refined, multimodal representations rather than raw unimodal features (Liu et al., 7 Jul 2025).

3. Orthogonal decomposition and fusion geometry

At the DHOF stage, the modality-common fused representation is

$f_{mm}$2

and the modality-specific fused representation has indexed feature vectors

$f_{mm}$3

DHOF treats $f_{mm}$4 as a reference direction in feature space and decomposes each $f_{mm}$5 into an aligned component and an orthogonal residual (Liu et al., 7 Jul 2025).

The projection of each specific vector onto the common direction is

$f_{mm}$6

where

$f_{mm}$7

The orthogonal component is then

$f_{mm}$8

By construction,

$f_{mm}$9

This is the central operation of DHOF: the part of the specific representation aligned with the common subspace is removed, and the complementary residual is preserved (Liu et al., 7 Jul 2025).

After orthogonalization, the set of residual vectors is aggregated with a pooling operation: $\mathrm{h}_p^o \in \mathbb{R}^d,$0 and a fully-connected layer produces the final multimodal representation

$\mathrm{h}_p^o \in \mathbb{R}^d,$1

This $\mathrm{h}_p^o \in \mathbb{R}^d,$2 is then passed to the survival prediction head (Liu et al., 7 Jul 2025).

The descriptors “deep,” “holistic,” and “orthogonal” correspond to distinct aspects of the design. “Deep” refers to the fact that fusion is preceded by MRD, cross-attention, and Transformer decoders rather than being a single shallow concatenate-plus-FC step. “Holistic” refers to the explicit modeling of both modality-common and modality-specific information and their interactions. “Orthogonal” refers specifically to the projection-residual scheme above. A common misconception is that DHOF introduces an orthogonality penalty term; in the MurreNet formulation, no separate explicit orthogonality loss is defined, and the orthogonality is enforced analytically by the forward computation itself (Liu et al., 7 Jul 2025).

4. Training objective and optimization role

DHOF is embedded in a broader training objective: $\mathrm{h}_p^o \in \mathbb{R}^d,$3 Here $\mathrm{h}_p^o \in \mathbb{R}^d,$4 is the negative log-likelihood survival loss applied to the prediction derived from $\mathrm{h}_p^o \in \mathbb{R}^d,$5, while the remaining terms regularize the representation decomposition learned upstream (Liu et al., 7 Jul 2025).

The similarity loss is given as

$\mathrm{h}_p^o \in \mathbb{R}^d,$6

Conceptually, it reduces discrepancy between modality-common representations and the original modality space. The difference loss encourages modality-specific and modality-common representations to remain distinct: $\mathrm{h}_p^o \in \mathbb{R}^d,$7 The reconstruction loss is

$\mathrm{h}_p^o \in \mathbb{R}^d,$8

Together, these terms constrain the decomposition to remain similar, different, and representative in the senses defined by the paper (Liu et al., 7 Jul 2025).

DHOF has no separate loss of its own, but it lies directly on the path from the decomposed representations to the survival objective: $\mathrm{h}_p^o \in \mathbb{R}^d,$9 As a result, gradients from the survival head propagate through the orthogonal decomposition into the specific and common encoders, the cross-attention parameters, and the Transformer decoder. The paper characterizes this effect as an implicit orthogonality-structured regularizer: the survival task pressures the model to organize $\mathrm{h}_g^o \in \mathbb{R}^d,$0 and $\mathrm{h}_g^o \in \mathbb{R}^d,$1 so that the orthogonal residual remains discriminative, while the aligned component is effectively treated as redundant for the survival head (Liu et al., 7 Jul 2025).

5. Empirical evidence in survival prediction

MurreNet reports extensive experiments on six TCGA cancer cohorts and states that it achieves state-of-the-art performance in survival prediction. The ablation study isolates the contribution of DHOF by comparing a simple multimodal baseline, MRD without DHOF, MRD with DHOF, and progressively regularized variants (Liu et al., 7 Jul 2025).

Model A, which uses simple multimodal concatenation without MRD or DHOF, reports C-index values of $\mathrm{h}_g^o \in \mathbb{R}^d,$2 on BLCA, $\mathrm{h}_g^o \in \mathbb{R}^d,$3 on LUAD, $\mathrm{h}_g^o \in \mathbb{R}^d,$4 on UCEC, and $\mathrm{h}_g^o \in \mathbb{R}^d,$5 on STAD. Model B adds MRD but still uses simple fusion, yielding $\mathrm{h}_g^o \in \mathbb{R}^d,$6, $\mathrm{h}_g^o \in \mathbb{R}^d,$7, $\mathrm{h}_g^o \in \mathbb{R}^d,$8, and $\mathrm{h}_g^o \in \mathbb{R}^d,$9, respectively. Model C adds DHOF on top of MRD and reaches $$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$0 on BLCA, $$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$1 on LUAD, $$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$2 on UCEC, and $$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$3 on STAD (Liu et al., 7 Jul 2025).

These ablations quantify the specific effect of orthogonal fusion relative to decomposition alone. The transition from A to B shows modest gains from representation decomposition. The transition from B to C shows further consistent gains attributable to DHOF: BLCA improves from $$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$4 to $$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$5, LUAD from $$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$6 to $$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$7, UCEC from $$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$8 to $$\mathrm{h}_p^s = \Gamma_p(\mathrm{h}_p^o), \qquad \mathrm{h}_g^s = \Gamma_g(\mathrm{h}_g^o),$$9, and STAD from $\Gamma_c$0 to $\Gamma_c$1. Additional improvements arise when similarity, difference, and reconstruction losses are added, culminating in Model F with $\Gamma_c$2 on BLCA, $\Gamma_c$3 on LUAD, $\Gamma_c$4 on UCEC, and $\Gamma_c$5 on STAD (Liu et al., 7 Jul 2025).

The main comparison reported for the full model states that MurreNet achieves the best C-index among the compared multimodal baselines MCAT, CMTA, MoME, MOTCat, SurvPath, and PORPOISE. The paper does not provide explicit qualitative visualizations of projected versus orthogonal components, so the empirical case for DHOF is primarily quantitative rather than visualization-based (Liu et al., 7 Jul 2025).

6. Relation to other orthogonal fusion paradigms and extensions

The closest conceptual analogue in computer vision is DOLG, which performs orthogonal fusion between local descriptors and a global image descriptor for single-stage image retrieval. There, each local feature vector is projected onto the global direction, the orthogonal component is retained, that residual is concatenated with the global descriptor, and the result is aggregated into a compact final embedding. DOLG reports that orthogonal fusion outperforms simple concatenation and Hadamard product on Revisited Oxford and Paris benchmarks, and it frames the resulting descriptor as complementary rather than redundant (Yang et al., 2021).

A second related line is DOF for multimodal glioma prognosis. DOF learns unimodal prognostic embeddings from radiology, pathology, genomics, and clinical data, combines them via attention-gated tensor fusion, and adds a Multimodal Orthogonalization (MMO) loss

$\Gamma_c$6

with total loss

$\Gamma_c$7

Its best reported result is a median C-index of $\Gamma_c$8 for Rad+Path+Gen with $\Gamma_c$9, significantly outperforming the best unimodal model at $$\mathrm{h}_p^{c}, \mathrm{h}_g^{c} = \Gamma_c(\mathrm{h}_p^o,\mathrm{h}_g^o).$$0 with $$\mathrm{h}_p^{c}, \mathrm{h}_g^{c} = \Gamma_c(\mathrm{h}_p^o,\mathrm{h}_g^o).$$1. Unlike DHOF in MurreNet, orthogonality in DOF is penalty-based rather than enforced by a projection-residual geometry in the fusion layer (Braman et al., 2021).

Taken together, these works place DHOF within a broader family of orthogonality-oriented fusion designs. MurreNet’s version is distinguished by explicit decomposition into modality-common and modality-specific subspaces followed by analytical orthogonalization of the specific stream against the common stream. DOLG uses an analogous projection against a holistic global image descriptor, and DOF uses nuclear-norm-based decorrelation across modality embeddings. This suggests that “deep holistic orthogonal fusion” is best understood not as a single invariant formula across the literature, but as a design principle in which deep fusion preserves complementary information by explicitly separating shared structure from non-redundant residual structure (Liu et al., 7 Jul 2025, Yang et al., 2021, Braman et al., 2021).

The MurreNet discussion also identifies several possible extensions. DHOF currently uses a single common direction $$\mathrm{h}_p^{c}, \mathrm{h}_g^{c} = \Gamma_c(\mathrm{h}_p^o,\mathrm{h}_g^o).$$2 per sample as the shared subspace; more complex shared subspaces, multi-vector bases, or subspace learning could capture richer shared structure. The current orthogonal fusion is scalar-projection based and does not explicitly model more complex nonlinear orthogonal relationships. The method is implemented for two modalities, so extension to more than two modalities would require a careful generalization of the common-versus-specific decomposition and projection scheme. These are presented as potential future directions rather than established properties of the current model (Liu et al., 7 Jul 2025).