---
title: Decoupled Representations with Knowledge Fusion
url: https://www.emergentmind.com/topics/decoupled-representations-with-knowledge-fusion-drkf
type: topic
---

# Decoupled Representations with Knowledge Fusion

Searching arXiv for the cited papers and related DRKF work.
to=arxiv_search.search 招商总代json code:
{"query":"id:2508.01644 OR id:2504.10529 OR id:2407.04955", "max_results": 10}
to=arxiv_search.search ＿天天json code:
{"query":"\"Decoupled Representations with Knowledge Fusion\" OR \"HeteRAG\" OR \"modality-Exclusive and modality-Agnostic\"", "max_results": 10}
Decoupled Representations with Knowledge Fusion (DRKF) denotes a modeling pattern in which a system first separates heterogeneous information into distinct representational views and then recombines those views through an explicit fusion mechanism. In the supplied literature, this pattern appears as decoupled retrieval- and generation-oriented chunk views in retrieval-augmented generation, as modality-exclusive and modality-agnostic subspaces in asynchronous multimodal video understanding, and as task-relevant shared and modality-specific factors in multimodal emotion recognition; the name “DRKF” is used explicitly for a multimodal emotion recognition framework built from an Optimized Representation Learning module and a Knowledge Fusion module [2504.10529][2407.04955][2508.01644].

## 1. Conceptual scope and defining structure

Across the cited works, DRKF is not a single operator but a recurring architectural schema. The decoupling step separates information that should not be forced into one undifferentiated embedding: retrieval versus generation views in HeteRAG, modality-exclusive versus modality-agnostic factors in MEA, and shared versus modality-specific task-relevant emotion information in DRKF. The fusion step then recombines these separated views in a task-dependent manner rather than by naive concatenation.

| Work | Decoupling axis | Fusion mechanism |
|---|---|---|
| HeteRAG | short chunks for generation vs context-enriched chunks for retrieval | chunk, multi-granular context, and metadata are fused for retrieval |
| MEA | modality-exclusive vs modality-agnostic representations | hierarchical cross-modal attention and Decoupled Graph Fusion |
| DRKF | task-relevant shared vs modality-specific features from speech and text | Fusion Encoder, Emotion Discrimination, Emotion Classification |

The common rationale is that different stages or modalities have different statistical and semantic requirements. HeteRAG states that retrieval prefers complete, context-rich representations, whereas generation prefers short, focused, context-light chunks. MEA states that modality heterogeneity makes undifferentiated multimodal fusion unreliable and indistinguishable. DRKF states that multimodal emotion recognition requires both task-relevant shared emotional information and task-relevant modality-specific information while suppressing task-irrelevant noise [2504.10529][2407.04955][2508.01644].

A recurring misconception is that decoupling means permanently isolating branches. The cited systems do not do this. HeteRAG maps retrieved chunk IDs back to short chunk text for prompting the LLM, MEA performs graph-based message passing after decoupling, and DRKF fuses optimized speech and text sequences with a self-attention-based Fusion Encoder. Decoupling therefore functions as a precondition for more structured fusion rather than as an alternative to fusion.

## 2. Representation decoupling mechanisms

In HeteRAG, decoupling is stage-specific. A document corpus $\{D_j\}_{j=1}^M$ is segmented into short chunks $\{ C^{(j)}_1, \dots, C^{(j)}_{N_j} \} = \mathcal{C}(D_j)$, and each chunk has two representations: a retrieval-oriented fused view and a generation-oriented short chunk view. The retrieval embedding is defined as
$$
\mathbf{e}_i^{(j)} = \mathcal{R}\left[\psi(C_i^{(j)}) \oplus \psi_{\text{ctx}}(\{C_{i_{\text{ctx}}}^{(j)}\}) \oplus \psi_{\text{meta}}(M_i^{(j)})\right],
$$
whereas generation uses only the focal chunk,
$$
\text{Ans} = \mathcal{G}\left(T(Q, C_i^{(j)})\right).
$$
This is a decoupling between a compound retrieval object and a minimal text snippet for generation [2504.10529].

In MEA, decoupling is factorized into modality-exclusive and modality-agnostic spaces. Exclusive representations are produced by separate encoders,
$$
\mathbf{h}^{e}_{m} = \mathcal{S}_{m}(\mathbf{Z}^{e}_{m}; \theta_{m}) \in \mathbb{R}^{d_h}, \quad m \in \{L, V, A\},
$$
while agnostic representations are produced by a shared encoder,
$$
\mathbf{h}^{a}_{m} = \mathcal{A}(\mathbf{Z}^{a}_{m}; \theta_{\mathcal{A}}) \in \mathbb{R}^{d_h}.
$$
The paper explicitly frames this as feature decoupling and disentangled representations. Independence between the exclusive and agnostic spaces is promoted with an HSIC disparity constraint, and modality specificity versus modality invariance is enforced with a double-discriminator strategy [2407.04955].

In DRKF for multimodal emotion recognition, decoupling is motivated by an information-theoretic decomposition of task-relevant multimodal information:
$$
I(X_1, X_2; Y) = S + P_1 + P_2,
$$
with
$$
S = I(X_1; X_2) - I(X_1; X_2 \mid Y), \qquad
P_1 = I(X_1; Y \mid X_2), \quad P_2 = I(X_2; Y \mid X_1).
$$
The architecture operationalizes this through speech and text encoders, progressive modality augmentation via residual autoencoders, and contrastive mutual information estimation between original and augmented views within each modality and between the two modalities. This setup is intended to emphasize task-relevant shared and private information while suppressing noise [2508.01644].

## 3. Knowledge fusion as the second stage

The fusion stage in HeteRAG occurs on the retrieval side. For each short chunk, the retriever receives the chunk itself, multi-granular contextual signals such as local neighbors and section or document-level context, and metadata such as titles, headings, subjects, abstracts, and keywords. HeteRAG also introduces adaptive prompt tuning with hierarchy-specific soft prompts,
$$
\tilde{C}_h = [\text{INST}_h] \oplus C,
$$
so that generic retrievers such as E5, BGE, and Jina can learn how to interpret the roles of focal text, context, and metadata in the fused sequence. The generation model never sees the retrieval-side fused representation; it sees only short retrieved chunks [2504.10529].

In MEA, fusion is split across two levels. Hierarchical Cross-Modal Attention constructs agnostic representations through multi-granularity cross-modal interaction, using Modality Reinforcement Units to transfer information between asynchronous sequences. After decoupling, Decoupled Graph Fusion performs message passing in two graphs: a heterogeneous graph over modality-exclusive nodes and a homogeneous graph over modality-agnostic nodes. For the exclusive graph, semantic strength and knowledge transfer are defined by
$$
\delta_{i,j} = \mathcal{Q}\big([\mathbf{W}_e \mathbf{h}^{e}_i, \mathbf{W}_e \mathbf{h}^{e}_j]; \theta_{\mathcal{Q}}\big),
\qquad
\xi_{i,j} = \frac{\exp(\mathrm{GeLU}(\delta_{i,j}))}
{\sum_{k \in \mathcal{N}_i} \exp(\mathrm{GeLU}(\delta_{i,k}))},
$$
followed by aggregation into $\mathbf{h}^{e}_{fin}$; the agnostic graph is constructed analogously for $\mathbf{h}^{a}_{fin}$. Final prediction uses the concatenation $[\mathbf{h}^{e}_{fin}, \mathbf{h}^{a}_{fin}]$ [2407.04955].

In DRKF, the Knowledge Fusion module contains a lightweight self-attention-based Fusion Encoder, an Emotion Discrimination submodule, and an Emotion Classification submodule. The Fusion Encoder operates on
$$
X_{fusion} = f_{FE}\big( \text{Concat}(C_{cls}, S_{seq}, C_{sep}, T_{seq}, C_{sep}) \big),
$$
with a learnable classification token and separator tokens. Through self-attention, the fused representation at $C_{cls}$ implicitly identifies a dominant modality while integrating complementary information from the other modality. The Emotion Discrimination submodule is trained on shuffled speech-text pairs to predict whether the pair is emotionally consistent or inconsistent, thereby forcing the fused representation to retain inconsistency cues even when dominant-modality selection is imperfect [2508.01644].

## 4. DRKF in multimodal emotion recognition

The explicitly named DRKF framework targets conversational multimodal emotion recognition from speech and text. Speech is encoded with a wav2vec2-based encoder into
$$
S_{seq} = \{s_1, \dots, s_m\} \in \mathbb{R}^{m \times d_z},
$$
and text is encoded with RoBERTa-large into
$$
T_{seq} = \{t_1, \dots, t_n\} \in \mathbb{R}^{n \times d_z}.
$$
Global embeddings are obtained by average pooling, and residual autoencoders produce augmented sequences $\tilde{S}_{seq}$ and $\tilde{T}_{seq}$. Progressive augmentation is supervised by a subspace-similarity term,
$$
\mathcal{L}_{MSE} = \frac{1}{2M} \sum_{i=1}^{M} \Big( \| S_{seq}^i - \tilde{S}_{seq}^{i} \|^2 + \| T_{seq}^i - \tilde{T}_{seq}^{i} \|^2 \Big),
$$
and a label-alignment term,
$$
\mathcal{L}_{KLD} = \frac{1}{M} \sum_{i=1}^{M} \sum_{c=1}^{C} y_i^c \log \frac{y_i^c}{\hat{y}_i^c},
$$
combined as $\mathcal{L}_a = \alpha \cdot \mathcal{L}_{MSE} + \mathcal{L}_{KLD}$.

Contrastive mutual information estimation then uses projection-space vectors
$$
z_S^i = g(S_{cls}^i),\quad \tilde{z}_S^i = g(\tilde{S}_{cls}^i),\quad
z_T^i = g(T_{cls}^i),\quad \tilde{z}_T^i = g(\tilde{T}_{cls}^i),
$$
and InfoNCE-style losses for intra-modal speech, intra-modal text, and inter-modal speech-text alignment. The batchwise CMIE objective is
$$
\mathcal{L}_c = \frac{1}{M} \sum_{i=1}^{M} \Big( \mathcal{L}_{MI_{S_i}} + \mathcal{L}_{MI_{T_i}} + \mathcal{L}_{MI_{S_i,T_i}} \Big).
$$
The total optimization target is
$$
\mathcal{L} = \mathcal{L}_a + \beta \cdot \mathcal{L}_c + \gamma \cdot \mathcal{L}_f + \delta \cdot \mathcal{L}_b.
$$

The empirical evaluation uses IEMOCAP, MELD, and M3ED. On IEMOCAP, DRKF is reported as “Ours(ORKF)” and achieves $ACC=80.7$, $WACC=79.9$, and $Avg=80.3$, compared with $ACC=78.9$ for DBT, $WACC=78.1$ for LLMSER, and $Avg=78.3$ for DBT. On MELD, it reports $WACC=66.7$, $WF1=65.4$, and $Avg=66.0$, compared with $66.2$, $65.1$, and $65.6$ for HiMul-LGG. On M3ED, it reports Precision $52.6$, Recall $51.6$, $ACC=50.6$, $F1=52.0$, and $Avg=51.7$, compared with Precision $47.1$, Recall $51.5$, $ACC=51.1$, $F1=49.2$, and $Avg=49.7$ for SAMS. The ablations further show that Progressive CMIE outperforms direct CMIE, self-attention fusion outperforms bidirectional cross-attention, and adding Emotion Discrimination improves IEMOCAP, MELD, and M3ED metrics; for example, in one comparison IEMOCAP ACC increases from $78.1$ to $79.0$, and M3ED $F1$ increases from $49.3$ to $51.1$ [2508.01644].

## 5. Related instantiations in multimodal video and retrieval-augmented generation

MEA can be read as a concrete instantiation of the same pattern for asynchronous multimodal video understanding. Its input sequences are language, visual, and acoustic streams,
$$
\mathbf{X}_L \in \mathbb{R}^{T_L \times d_L},\quad
\mathbf{X}_V \in \mathbb{R}^{T_V \times d_V},\quad
\mathbf{X}_A \in \mathbb{R}^{T_A \times d_A},
$$
processed by temporal 1D convolution, positional embedding, and BiLSTM. Predictive Self-Attention and a Weighted Attention Layer refine modality-exclusive features, Hierarchical Cross-Modal Attention constructs modality-agnostic features, HSIC and adversarial losses enforce decoupling, and Decoupled Graph Fusion combines modality-level nodes. The model is reported to achieve new SOTA on MOSI, MOSEI, and IEMOCAP, with MOSI results $Acc_7=42.5$, $Acc_2=84.4$, $F1=84.6$, $MAE=0.844$, and $Corr=0.735$, and MOSEI results $Acc_7=54.8$, $Acc_2=85.2$, $F1=85.1$, $MAE=0.563$, and $Corr=0.731$. Its ablations show that removing HSIC lowers MOSI $F1$ from $84.6$ to $83.5$ and MOSEI $F1$ from $85.1$ to $84.5$, removing the adversarial losses reduces MOSI $F1$ to $82.9$, removing the exclusive or agnostic branch lowers performance further, and replacing DGF with simple concatenation yields about $83.4$ on MOSI and $83.6$ on MOSEI instead of $84.6$ and $85.1$ [2407.04955].

HeteRAG extends the same logic to retrieval-augmented generation. It uses short chunks as atomic units for generation while constructing context-enriched retrieval inputs from chunk text, local neighbors, structural context, and metadata. The retrieval model is a dense bi-encoder trained with InfoNCE contrastive loss and scaled cosine similarity, while adaptive prompt tuning introduces hierarchy-specific soft prompts for heterogeneous retrieval inputs. On BEIR tasks—SciFact, NF-Corpus, and Trec-COVID—HeteRAG reports average gains of $+9.43\%$ nDCG@1 and $+7.76\%$ nDCG@10 versus naive RAG. On Trec-COVID with BGE and chunk size 64, Naive RAG obtains $65.00\% / 65.06\%$ for nDCG@1/@10, whereas HeteRAG obtains $87.00\% / 77.30\%$. In end-to-end QA on PopQA, NQ, SQuAD, TriviaQA, and HotpotQA using Wiki2018 and generators such as Llama3-8B-Instruct, Mistral-8B-Instruct, and Gemma-9B-Instruct, HeteRAG consistently improves EM, F1, and Recall over no RAG and naive RAG; for Llama3-8B on NQ, it reports EM $36.10\%$, F1 $48.24\%$, and Recall $57.46\%$, compared with $29.80\%$, $40.25\%$, and $47.01\%$ for naive RAG and $22.40\%$, $32.61\%$, and $37.45\%$ without RAG [2504.10529].

These two works show that the DRKF pattern is not confined to emotion recognition. In MEA, the central distinction is between shared and modality-specific latent factors under temporal asynchrony. In HeteRAG, the distinction is between what should support retrieval and what should be exposed to the generator. This suggests that DRKF is better understood as an architectural principle for managing heterogeneity than as a domain-specific recipe.

## 6. Empirical lessons, misconceptions, and open problems

The empirical record across the three papers supports three recurring lessons. First, decoupling alone is insufficient. MEA shows that removing DGF or HCA degrades performance, and DRKF shows that adding Emotion Discrimination improves results over a Fusion Encoder without that regularizer. HeteRAG likewise improves not merely by shortening chunks but by compensating retrieval with multi-granular context and metadata. A plausible implication is that the decisive step is not separation by itself but separation followed by structured recombination [2407.04955][2508.01644][2504.10529].

Second, the “shared” component should not be treated as the only useful component. In MEA, removing either modality-exclusive or modality-agnostic representations reduces performance; in DRKF, modality-specific emotional cues remain critical because real-world samples can be emotionally inconsistent across modalities. HeteRAG makes an analogous point at the pipeline level: retrieval profits from redundancy and broader context, but generation deteriorates when that same redundancy is passed verbatim into the LLM. This directly contradicts the common assumption that one unified representation is always optimal.

Third, the current literature leaves open several scaling and robustness issues. DRKF is evaluated only in the bimodal audio-text setting and does not yet report trimodal extensions or missing-modality handling. MEA requires all modalities present at training and inference, and its graph fusion operates at modality level with three nodes, so scaling to many modalities or finer-grained node sets would require careful design. HeteRAG highlights limited domain coverage, a focus on retrieval-side optimization, and the need for further validation on code, logs, and multimodal documents; it also suggests prompt token compression and more sophisticated multi-vector DRKF as future directions [2508.01644][2407.04955][2504.10529].

Within this literature, DRKF therefore names both a specific multimodal emotion recognition framework and a broader design principle: decouple representational roles that have conflicting objectives, then apply a fusion mechanism that preserves complementarity, suppresses noise, and remains sensitive to inconsistency. The cited works differ in modality, task, and optimization strategy, but they converge on the same structural claim: heterogeneous evidence is most useful when it is neither prematurely collapsed into a single space nor left unfused after disentanglement.

Source: https://www.emergentmind.com/topics/decoupled-representations-with-knowledge-fusion-drkf