DeReF: Multimodal Decoupling, Reorganization & Fusion

• DeReF is a multimodal design pattern that separates heterogeneous inputs into distinct subspaces, enabling explicit reorganization before dynamic fusion.
• It employs specialized modules—such as shared/private encoders, regional cross-attention, and dynamic MoE fusion—to address fixed fusion schemes and information closure.
• Empirical studies across cancer survival, scene parsing, and text-to-image generation demonstrate improved representation quality and predictive performance compared to traditional methods.

The Decoupling-Reorganization-Fusion Framework (DeReF) is a multimodal design pattern in which heterogeneous inputs are first separated into structured subspaces, then rearranged or recalibrated into more informative organizations, and only afterward fused for downstream prediction or generation. In the cancer survival formulation, DeReF is presented explicitly as a framework that addresses two limitations of prior multimodal methods—“fixed fusion schemes” and “information closure in Mixture-of-Experts (MoE)”—by combining feature decoupling, random feature reorganization, and dynamic MoE fusion (Wang et al., 26 Aug 2025). Closely related systems in IVF-ET pregnancy prediction, RGB-X scene parsing, and identity-preserving text-to-image generation instantiate the same three-stage logic, although with task-specific modules and objectives (Ouyang et al., 8 Jan 2025, Huang et al., 2024, Chen et al., 28 May 2025).

1. Definition and conceptual structure

DeReF is organized around three sequential operations: decoupling, reorganization, and fusion. In the explicit cancer survival formulation, the sequence is described as: Feature Decoupling, Random Feature Reorganization, and Dynamic MoE Fusion (Wang et al., 26 Aug 2025). In DeFusion for IVF-ET pregnancy prediction, the same pattern is stated as follows: decoupling separates each modality into related (common/shared) and unrelated (unique/private) components; reorganization structures the feature space into explicit shared and private segments; fusion recombines these components for classification (Ouyang et al., 8 Jan 2025).

Across the cited works, the meaning of the three stages is consistent but not identical in implementation. In RoadFormer+, decoupling separates RGB-X features into global and local components at multiple scales; reorganization occurs through recalibration, alignment, and enhancement in the multi-scale heterogeneous feature fusion block; fusion integrates the processed features before semantic decoding (Huang et al., 2024). In subject-driven text-to-image generation, decoupling separates identity-relevant and identity-irrelevant information, reorganization refines these independent features “at multiple levels,” and fusion dynamically combines them through an MoE-based module before conditioning the denoising U-Net (Chen et al., 28 May 2025).

A plausible implication is that DeReF is better understood as a family of architectural constraints on multimodal interaction than as a single fixed network. The invariant principle is that fusion is deferred until the model has made explicit structural commitments about what kinds of information each latent segment should represent.

2. Decoupling as latent factor separation

The decoupling stage is the point at which DeReF systems attempt to disentangle heterogeneous information rather than immediately aggregate it. In DeFusion, the extracted image feature $\mathbf{f}_i$ and table feature $\mathbf{f}_t$ are mapped into common and unique subspaces by shared and private encoders: $$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$

$$\mathbf{f}_i^u = E_u(\mathbf{f}_i; \theta_i^u), \quad \mathbf{f}_t^u = E_u(\mathbf{f}_t; \theta_t^u).$$

The purpose is “to explicitly partition each modality’s feature vector into related (common/shared) and unrelated (unique/private)” (Ouyang et al., 8 Jan 2025).

In the cancer survival formulation, decoupling produces modality-specific, modality-shared, and modality-explored features. The Regional Cross-Attention Network is introduced because “standard cross-attention focuses only on inter-modal relationships; regional cross-attention also preserves intra-modal interactions, enhancing representation quality” (Wang et al., 26 Aug 2025). The decoupling loss is defined as

$\begin{split} \mathcal{L}_{\text{dis}} &= \text{Dis}(V_{\text{sp1}}, V_{\text{share}}) + \text{Dis}(V_{\text{sp2}}, V_{\text{share}}) \ &\quad + \text{Dis}(V_{\text{sp1}}, V_{\text{explore}}) + \text{Dis}(V_{\text{sp2}}, V_{\text{explore}}) \ &\quad + \text{Dis}(V_{\text{share}}, V_{\text{explore}}) - \text{Dis}(V_{\text{sp1}}, V_{\text{sp2}}). \end{split}$

Here, $\text{Dis}$ is “typically mean squared error (MSE), maximizing $V_{\text{sp1}}, V_{\text{sp2}}$ distance (distinctiveness), while minimizing distances among shared/explored and specific features (for interaction)” (Wang et al., 26 Aug 2025).

RoadFormer+ instantiates decoupling in a scale-aware manner. Its Hybrid Feature Decoupling Encoder (HFDE) uses a single weight-sharing ConvNeXt backbone to produce multi-scale features, then separates each scale into global features via MHSA and local features via lightweight CNN blocks: $$\mathbf{G}_i = \mathrm{Norm}\left(\mathrm{MHSA}(\mathbf{F}_i) + \mathbf{F}_i\right), \qquad \mathbf{L}_i = \text{InvertedResBlock}(\mathbf{F}_i) + \mathbf{F}_i.$$ This formulation explicitly distinguishes “long-range, context/global semantic information” from “edges, textures, and fine-grained details” (Huang et al., 2024).

In identity-preserving text-to-image generation, the decoupling objective is foreground-background separation. The Implicit-Explicit foreground-background Decoupling Module (IEDM) performs both feature-level and image-level decomposition. The implicit branch computes

$f_i = \text{Adapter}(E_I(x_i)),$

while the explicit branch uses segmentation and inpainting: $$x'_i = \text{Inpaint}(x_i, M_i), \qquad f'_i = E_I(x'_i).$$ The associated losses

$\mathbf{f}_t$0

are introduced to ensure that identity-related and identity-irrelevant features are separated while the two background representations remain aligned (Chen et al., 28 May 2025).

Taken together, these examples show that “decoupling” in DeReF is not restricted to one semantic split. Depending on the task, the partition may be shared/private, global/local, or identity-relevant/identity-irrelevant.

3. Reorganization as structuring, shuffling, and recalibration

The reorganization stage is the aspect that most clearly differentiates DeReF from a generic disentangle-then-concatenate pipeline. In DeFusion, reorganization is described as the architectural structuring of $\mathbf{f}_t$1, $\mathbf{f}_t$2, $\mathbf{f}_t$3, and $\mathbf{f}_t$4 into explicit “shared” and “private” spaces, with t-SNE and PCC analyses presented as evidence that the resulting latent organization reflects the intended semantics (Ouyang et al., 8 Jan 2025).

In the cancer survival framework, reorganization is operationalized more directly. Each decoupled feature $\mathbf{f}_t$5, with $\mathbf{f}_t$6, is divided into $\mathbf{f}_t$7 segments of size $\mathbf{f}_t$8, where $\mathbf{f}_t$9 is randomly chosen from $$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$0 in each forward pass. Segments with the same index are grouped across features: $$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$1 and the final reorganized feature is

$$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$2

The stated motivation is to “increase the diversity of feature combinations and granularity” and to “overcome the problem of information closure” before expert fusion (Wang et al., 26 Aug 2025).

RoadFormer+ uses “reorganization” in the sense of recalibration and multi-branch feature restructuring. Its Global Feature Recalibration Module (GFRM) cross-attends between modalities and then applies channel attention; its Local Feature Fusion Module (LFFM) combines local detail streams by convolutional interaction; and its Feature Enhancement and Integration Module (FEIM) adds spatial attention through directional pooling and coordinate-attention-style reweighting (Huang et al., 2024). The sequence

$$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$3

illustrates that reorganization can take the form of adaptive feature redistribution rather than random shuffling.

In the text-to-image setting, the data state that reorganization “refines and enhances the extracted independent features at multiple levels” before dynamic fusion (Chen et al., 28 May 2025). Although no standalone reorganization block is defined there in the same explicit way as in the cancer survival paper, the formulation suggests that DeReF can treat refinement of decoupled features as a reorganization operation when those refinements alter how information is presented to later fusion modules.

A common misconception is that reorganization is merely a synonym for concatenation order. The cited formulations do not support that reduction. In DeReF systems, reorganization changes either the semantic grouping, the granularity, or the attention-weighted configuration of decoupled features before fusion.

4. Fusion after decoupling and reorganization

Fusion in DeReF is deferred until the model has formed explicit latent partitions and, in some cases, transformed their organization. In DeFusion, the four components are concatenated and sent to a classifier: $$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$4 Training combines cross-entropy with reconstruction: $$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$5 The paper contrasts this with “simple” fusions such as addition and concatenation, where modalities are blended in a “disordered” way (Ouyang et al., 8 Jan 2025).

The reconstruction term is central to how DeFusion stabilizes the fusion stage. Its cross-reconstruction loss is

$$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$6

Reconstructing one modality from the other modality’s common feature and its own unique feature is used to encourage the common encoder to capture “truly shared aspects” and the unique encoder to preserve modality-specific information (Ouyang et al., 8 Jan 2025).

In the cancer survival framework, fusion is performed by a dense MoE with dynamic gating. All experts are activated, and the gating network computes sample-dependent weights: $$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$7

$$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$8

followed by

$$\mathbf{f}_i^c = E_c(\mathbf{f}_i; \theta^c), \quad \mathbf{f}_t^c = E_c(\mathbf{f}_t; \theta^c),$$9

The stated advantage is that dynamic weighting lets the system emphasize “the most prognostically relevant interactions for each patient/data sample” (Wang et al., 26 Aug 2025).

The MoE principle also appears in the text-to-image Feature Fusion Module (FFM). There,

$$\mathbf{f}_i^u = E_u(\mathbf{f}_i; \theta_i^u), \quad \mathbf{f}_t^u = E_u(\mathbf{f}_t; \theta_t^u).$$0

with $$\mathbf{f}_i^u = E_u(\mathbf{f}_i; \theta_i^u), \quad \mathbf{f}_t^u = E_u(\mathbf{f}_t; \theta_t^u).$$1. The fused representation $$\mathbf{f}_i^u = E_u(\mathbf{f}_i; \theta_i^u), \quad \mathbf{f}_t^u = E_u(\mathbf{f}_t; \theta_t^u).$$2 then conditions the denoising U-Net, and the diffusion loss is

$$\mathbf{f}_i^u = E_u(\mathbf{f}_i; \theta_i^u), \quad \mathbf{f}_t^u = E_u(\mathbf{f}_t; \theta_t^u).$$3

The paper states that this dynamic integration helps “even in cases of incomplete decoupling” (Chen et al., 28 May 2025).

RoadFormer+ uses a non-MoE fusion design, but the same staged logic is preserved. The dual-branch multi-scale heterogeneous feature fusion block runs Transformer-based modules for global structure and CNN-based modules for local detail “in parallel,” explicitly to mitigate “the classic trade-off between context size and local spatial precision” (Huang et al., 2024).

5. Representative implementations across domains

The DeReF pattern has been instantiated in substantially different application domains. The following table organizes the implementations described in the cited papers.

System	Decoupling	Reorganization/Fusion
DeFusion	Common/shared and unique/private encoders for image and table features	Reorganized shared/private latent spaces; concatenation for classification (Ouyang et al., 8 Jan 2025)
RoadFormer+	Global/local decoupling at multiple scales via HFDE	GFRM, LFFM, and FEIM for recalibration and multi-scale heterogeneous fusion (Huang et al., 2024)
Cancer DeReF	Modality-specific, modality-shared, and modality-explored features via regional cross-attention	Random feature reorganization plus dynamic dense MoE fusion (Wang et al., 26 Aug 2025)
Identity-preserving T2I	Implicit-explicit foreground-background decoupling	MoE-based FFM for dynamic integration before U-Net conditioning (Chen et al., 28 May 2025)

In IVF-ET pregnancy prediction, the multimodal inputs are “temporal embryo images (three consecutive days)” and “parental fertility table indicators,” with image features extracted by ResNet plus spatial-temporal position encoding and table features extracted by TabTransformer (Ouyang et al., 8 Jan 2025). In RGB-X scene parsing, the inputs are RGB together with depth, thermal, surface normal, or polarization, processed by a single weight-sharing backbone and fused across scales for semantic parsing (Huang et al., 2024). In cancer survival prediction, DeReF is applied to heterogeneous biomedical data such as “MRI, WSI, or Genomics data” on an in-house Liver Cancer dataset and TCGA datasets including BLCA, UCEC, and LUAD (Wang et al., 26 Aug 2025). In subject-driven generation, DeReF is used to preserve identity under changing prompts by separating subject and background information and fusing them dynamically (Chen et al., 28 May 2025).

This distribution across classification, segmentation, survival analysis, and generative modeling suggests that DeReF is not tied to a single output type. The available evidence instead points to a common architectural commitment: modalities or semantic factors are not fused monolithically.

6. Empirical evidence, interpretation, and scope

The cited works report empirical support for the DeReF pattern, although with different metrics and experimental settings. DeFusion states that experiments on “a new dataset including 4046 cases collected from Southern Medical University” show that the model “outperforms state-of-the-art methods,” and that performance on an eye disease prediction dataset reflects “good generalization” (Ouyang et al., 8 Jan 2025). The same source reports that t-SNE and correlation analysis show the decoupled related features of the two modalities to be “highly overlapping (high PCC), while unrelated ones are distinct” (Ouyang et al., 8 Jan 2025).

RoadFormer+ reports first place on the KITTI Road benchmark, state-of-the-art mean intersection over union on Cityscapes, MFNet, FMB, and ZJU, and a 65% reduction in learnable parameters compared to RoadFormer (Huang et al., 2024). The technical explanation given for these outcomes is the combination of weight sharing, scale-aware global/local decoupling, and heterogeneous fusion with channel and spatial recalibration.

The explicit DeReF cancer survival paper reports the following C-Index results:

Model	LC	BLCA	UCEC	LUAD	Overall (TCGA)
MCAT	0.614	0.672	0.649	0.659	0.660
CFDL (MoE fusion)	0.612	0.675	0.679	0.664	0.673
CCL (Cross-att)	0.648	0.640	0.686	0.657	0.661
DeReF (ours)	0.671	0.681	0.688	0.671	0.680

The same paper reports that removing Regional Cross-Attention causes a C-Index drop by 0.4–1.1%, removing Random Feature Reorganization drops C-Index by 0.7–3.4%, and removing the Modality-Explored Feature also harms performance (Wang et al., 26 Aug 2025). It further reports that Kaplan-Meier high-vs-low risk group separability is statistically significant with $$\mathbf{f}_i^u = E_u(\mathbf{f}_i; \theta_i^u), \quad \mathbf{f}_t^u = E_u(\mathbf{f}_t; \theta_t^u).$$4 (Wang et al., 26 Aug 2025).

For identity-preserving text-to-image generation, the reported quantitative results are CLIP-T $$\mathbf{f}_i^u = E_u(\mathbf{f}_i; \theta_i^u), \quad \mathbf{f}_t^u = E_u(\mathbf{f}_t; \theta_t^u).$$5, CLIP-I $$\mathbf{f}_i^u = E_u(\mathbf{f}_i; \theta_i^u), \quad \mathbf{f}_t^u = E_u(\mathbf{f}_t; \theta_t^u).$$6, DINO $$\mathbf{f}_i^u = E_u(\mathbf{f}_i; \theta_i^u), \quad \mathbf{f}_t^u = E_u(\mathbf{f}_t; \theta_t^u).$$7, and storage $$\mathbf{f}_i^u = E_u(\mathbf{f}_i; \theta_i^u), \quad \mathbf{f}_t^u = E_u(\mathbf{f}_t; \theta_t^u).$$8 MB for the full method, compared with lower values for ablated variants without IEDM or FFM (Chen et al., 28 May 2025). The data state that removing either module degrades performance and that all three decoupling losses contribute.

One recurring point of interpretation is that DeReF should not be conflated with any one fusion backend. The cited systems use concatenation, attention, dense MoE, parallel Transformer-CNN fusion, and diffusion conditioning. Another common misconception is that the framework necessarily guarantees interpretability. The available evidence is narrower: the papers report t-SNE, PCC, expert-weight visualization, or qualitative generation comparisons as indicators that the learned partitions are meaningful, but they do not reduce the framework to a fully solved interpretability problem (Ouyang et al., 8 Jan 2025, Wang et al., 26 Aug 2025).

Overall, the literature represented here characterizes DeReF as a structured response to entanglement, redundancy, and inflexible multimodal interaction. Its central claim is not merely that multiple modalities should be combined, but that the order of operations—decouple first, reorganize second, fuse last—is itself a design principle with measurable consequences across predictive and generative tasks.