Dual-Modality Node Encoder

• Dual-Modality Node Encoder is a technique that maps two distinct modalities into a unified embedding space for enhanced graph-based learning.
• It leverages architectural patterns such as shared-core, dual-pathway fusion, and graph-native attention to integrate diverse signals efficiently.
• It employs contrastive and reconstruction losses to align modality signals, boosting performance in low-data regimes and heterogeneous networks.

Searching arXiv for recent and foundational papers on dual-modality node encoders, shared encoders, and multimodal graph/node representation learning. A Dual-Modality Node Encoder is a representation-learning module that associates each node with two modalities and maps them into a unified embedding for downstream consumption, such as graph propagation, retrieval, classification, or anomaly scoring. In the cited literature, the two modalities may be text and image, structured data and text, graph structure and node attributes, or two visual modalities; the encoder may be implemented as a shared transformer, as paired modality-specific pathways coupled by reconstruction or attention, or as a heterogeneous graph attention mechanism that fuses modalities during message passing rather than only before or after it (Roy et al., 3 Mar 2025).

1. Problem setting and scope

In one common formulation, each node carries two modalities $m \in \{a,b\}$, and the objective is to produce a unified node embedding that can be consumed by downstream graph layers while generalizing well in low-data regimes and across domains. The motivation is particularly explicit in medical multimodal learning, where separate modality-specific encoders “double parameters and require more data to avoid overfitting,” while paired multimodal data are scarce; a shared encoder conditioned by learnable modality features is proposed precisely to reduce learnable parameters and improve efficiency and cross-modal generalization (Roy et al., 3 Mar 2025).

A broader graph-theoretic formulation appears in attributed networks, where the two modalities are graph structure and node attributes. AnomalyDAE defines an attributed network $G=\{V,E,A\}$ with adjacency matrix $A \in \mathbb{R}^{M \times M}$ and attribute matrix $X \in \mathbb{R}^{M \times N}$, then learns node embeddings $Z^{\mathcal V}$ and attribute embeddings $Z^{\mathcal A}$ jointly in latent space, explicitly modeling “the complex interactions between network structure and node attribute” (Fan et al., 2020).

In heterogeneous networks, the setting becomes typed and partially observed. HGNN-IMA defines a multi-modal heterogeneous network as $G=(V,E,M)$, with node types $\psi(v)\in O$, edge types $\phi(e)\in R$, and modality availability governed by $f(\psi(v)) \subseteq M$. Not all node types carry all modalities, and only one target node type is typically classified, so the encoder must simultaneously respect heterogeneity, missingness, and cross-modal influence (Li et al., 12 May 2025).

A related variant arises in multimodal knowledge graphs. There, literals and non-literals are treated as separate cases, modality-specific encoders map raw values to fixed-size vectors, and initial node features take the form $G=\{V,E,A\}$0, where identity features are concatenated with multimodal literal embeddings before relational message passing (Wilcke et al., 2020). This suggests that “Dual-Modality Node Encoder” names a family of designs rather than a single canonical layer.

2. Architectural patterns

The literature suggests three recurrent architectural patterns.

The first is the shared-core architecture. In the shared-encoder framework, a single transformer stack $G=\{V,E,A\}$1 of $G=\{V,E,A\}$2 layers processes inputs from all modalities, with recurrence

$G=\{V,E,A\}$3

and with the more general form

$G=\{V,E,A\}$4

where $G=\{V,E,A\}$5 and $G=\{V,E,A\}$6 are optional modality-specific sequence-to-sequence blocks before or after the shared stack. Early modality-specific layers consistently improved performance, whereas late layers showed mixed results (Roy et al., 3 Mar 2025). Closely related multimodal backbones extend this logic beyond graph nodes. MoMo uses a single transformer with all encoder layers processing both text and image modalities in one unified sequence, and VLMo uses a shared self-attention backbone with modality-specific experts, switching to vision-language experts in upper layers for fusion (Chada et al., 2023, Bao et al., 2021).

The second pattern is the dual-pathway architecture with explicit cross-modality coupling. In AnomalyDAE, a structure encoder first transforms $G=\{V,E,A\}$7 into node features and then applies GAT-like attention over graph neighborhoods, while a separate attribute encoder maps $G=\{V,E,A\}$8 through two non-linear layers to obtain attribute embeddings. The coupling occurs in the decoders:

$G=\{V,E,A\}$9

The first reconstructs structure from node embeddings; the second reconstructs node attributes from both node and attribute embeddings, making the attribute decoder the key site of cross-modality interaction (Fan et al., 2020).

The third pattern is graph-native multimodal attention. HGNN-IMA nests inter-modal attention inside heterogeneous inter-node attention. For neighbor $A \in \mathbb{R}^{M \times M}$0 of node $A \in \mathbb{R}^{M \times M}$1, modality-specific inter-node attentions $A \in \mathbb{R}^{M \times M}$2 are fused through inter-modal weights $A \in \mathbb{R}^{M \times M}$3 into a single neighbor weight

$A \in \mathbb{R}^{M \times M}$4

which is then used for propagation across modalities (Li et al., 12 May 2025). A graph-based text-image encoder for neural machine translation instantiates a related idea at the level of words and visual objects: source words and grounded visual objects become nodes in a unified multimodal graph, and cross-modal edges are gated by element-wise sigmoid vectors rather than scalar weights (Yin et al., 2020).

3. Input modeling, projections, and modality conditioning

A central design requirement is a common embedding space. In the shared-encoder framework, each modality-specific projection $A \in \mathbb{R}^{M \times M}$5 maps raw inputs $A \in \mathbb{R}^{M \times M}$6 into a common embedding dimension $A \in \mathbb{R}^{M \times M}$7, yielding

$A \in \mathbb{R}^{M \times M}$8

where $A \in \mathbb{R}^{M \times M}$9 is a learnable modality embedding or a modality feature vector, and $X \in \mathbb{R}^{M \times N}$0 is positional encoding (Roy et al., 3 Mar 2025). Text is tokenized with BPE and embedded through $X \in \mathbb{R}^{M \times N}$1; images are patchified and projected with $X \in \mathbb{R}^{M \times N}$2; structured or tabular inputs use an MLP stem $X \in \mathbb{R}^{M \times N}$3 and are then treated as token sequences in the same $X \in \mathbb{R}^{M \times N}$4-dimensional space. The implementation guidance specifies $X \in \mathbb{R}^{M \times N}$5, $X \in \mathbb{R}^{M \times N}$6, image size $X \in \mathbb{R}^{M \times N}$7, and a best-performing modality feature budget of 20 out of 768 input features (Roy et al., 3 Mar 2025).

Conditioning the shared core to distinguish modalities without duplicating parameters is a major theme. Two mechanisms were studied: a modality feature vector $X \in \mathbb{R}^{M \times N}$8 concatenated to every token embedding, and a modality token $X \in \mathbb{R}^{M \times N}$9 prepended to the sequence. Modality feature vectors were more effective in 6 of 8 tasks, and the best configuration allocated 20 of 768 input features to the modality vector (Roy et al., 3 Mar 2025). Because conditioning modifies $Z^{\mathcal V}$0, it propagates through all attention layers by changing the induced $Z^{\mathcal V}$1, $Z^{\mathcal V}$2, and $Z^{\mathcal V}$3 states in

$Z^{\mathcal V}$4

Multimodal knowledge-graph models use dedicated encoders rather than a shared transformer stem. For textual literals, a character-level one-hot sequence is passed through a 1D CNN with output size $Z^{\mathcal V}$5; for images, a MobileNet encoder yields $Z^{\mathcal V}$6; these are then concatenated with identity features and passed to an R-GCN (Wilcke et al., 2020). MemeFier provides another conditioning strategy: each image patch is multiplied element-wise by the global text embedding, and each text token is multiplied by the global image embedding before fusion, producing alignment-aware nodes prior to transformer self-attention (Koutlis et al., 2023). This suggests that modality conditioning can be implemented either as explicit embeddings, as token-level gating, or as relation-specific attention.

4. Learning objectives and fusion mechanisms

The training objective depends on the downstream task and on how the two modalities are intended to interact.

For paired modalities with retrieval or alignment goals, contrastive training is common. The shared-encoder framework uses a symmetric CLIP-style InfoNCE loss on paired examples,

$Z^{\mathcal V}$7

with cosine similarity and a temperature $Z^{\mathcal V}$8; the [CLS] token of the shared encoder output is used as the sequence summary, and small modality-specific heads $Z^{\mathcal V}$9 project the shared summary to task- or modality-specific output spaces (Roy et al., 3 Mar 2025).

For reconstruction-driven objectives, the node encoder is supervised through the decoders rather than directly through paired similarity. AnomalyDAE minimizes a weighted Frobenius reconstruction loss

$Z^{\mathcal A}$0

where $Z^{\mathcal A}$1 and $Z^{\mathcal A}$2 amplify penalties on non-zero entries. Per-node anomaly scores combine the same structure and attribute reconstruction errors, and nodes are ranked or thresholded accordingly (Fan et al., 2020).

For typed node classification, multimodal fusion is often embedded directly inside message passing. HGNN-IMA uses a classification loss over both fused and uni-modal heads together with an attention loss

$Z^{\mathcal A}$3

where $Z^{\mathcal A}$4 penalizes assigning large inter-modal weights to modalities that a neighbor does not possess (Li et al., 12 May 2025). In multimodal knowledge graphs, the objective is row-wise softmax cross-entropy over labeled entities after two R-GCN layers, with literal embeddings and structural information integrated in $Z^{\mathcal A}$5 (Wilcke et al., 2020).

Fusion itself occurs at multiple levels. The shared-encoder node pipeline extracts modality-specific [CLS] summaries, then fuses them into a unified node representation, for example

$Z^{\mathcal A}$6

or by averaging. An optional learned gating $Z^{\mathcal A}$7 can weight fusion according to modality embeddings (Roy et al., 3 Mar 2025). A plausible implication is that “fusion” in Dual-Modality Node Encoders is not a single operation but a layered process: projection into a common space, conditioning or alignment during encoding, and explicit or implicit aggregation into a final node embedding.

5. Use inside graph learning systems

When the node encoder is part of a larger graph model, its output becomes the interface between multimodal representation learning and topological propagation. The shared-encoder blueprint is explicit: preprocess each node’s modalities, project them to a common $Z^{\mathcal A}$8, apply modality conditioning, encode them with $Z^{\mathcal A}$9, extract [CLS] summaries, fuse them into $G=(V,E,M)$0, and pass the resulting embedding to downstream GNN layers such as GCN or GAT (Roy et al., 3 Mar 2025). If node-level supervision is available, supervised heads can be attached after graph propagation and trained jointly with contrastive losses when paired subsequences are present.

In multimodal knowledge graphs, entity nodes often lack raw literal values of their own. Their feature rows may therefore be zeros in $G=(V,E,M)$1, but they receive multimodal information from connected literal nodes during relation-specific message passing. The sparse first-layer implementation splits identity and feature components,

$G=(V,E,M)$2

after which the model proceeds with standard R-GCN propagation (Wilcke et al., 2020).

A distinct but related interpretation appears in UniG-Encoder, where the two modalities are original node attributes and topology-derived edge or hyperedge features. The projection matrix

$G=(V,E,M)$3

transforms node features into a joint node/edge space, and the transpose $G=(V,E,M)$4 maps encoded representations back to nodes; in the canonical construction,

$G=(V,E,M)$5

This makes topology itself a second modality fused with attributes through an encoder rather than through iterative message passing (Zou et al., 2023).

The handling of missing modalities is a recurring systems issue. The shared-encoder framework proposes a learned “missing-modality” token $G=(V,E,M)$6 interpreted via modality-drop training, or else a fallback to the present modality or to a learned imputer (Roy et al., 3 Mar 2025). HGNN-IMA instead augments training with an attention loss to mitigate the impact of missing modalities, and its experiments also consider imputing missing images with the text encoder output (Li et al., 12 May 2025). These designs reflect different assumptions: one treats absence as an input symbol to be encoded, while the other regularizes propagation to avoid placing mass on nonexistent modality channels.

6. Empirical behavior, limitations, and research directions

Empirical results indicate that no single design dominates across all settings, but several patterns recur.

Under a matched 125M-parameter budget, the shared transformer with modality vectors outperformed separate 6-layer CLIP encoders in 7 of 8 retrieval tasks and outperformed a larger 12-layer modality-specific model with 210M parameters in 5 of 8 tasks. On in-distribution PMC-OA retrieval, Recall@200 improved from 0.3965/0.3952 to 0.4152/0.4152 for image-to-text and text-to-image, respectively. The low-data effect was stronger: at PMC-OA training sizes of 0.66M and 0.33M image-text pairs, Recall@200 gains for shared versus separate encoders were +81.0% and +16.9% for image-to-text and +94.2% and +29.6% for text-to-image (Roy et al., 3 Mar 2025). This directly supports the claim that parameter sharing can improve generalization when paired data are scarce.

For structure-attribute graphs, AnomalyDAE reported ROC AUC scores of 97.81 on BlogCatalog, 97.22 on Flickr, and 90.05 on ACM, with improvements over Dominant of +19.68%, +22.32%, and +15.11%, respectively. Its sensitivity analysis further showed that relying on only one modality, by setting $G=(V,E,M)$7 or $G=(V,E,M)$8, degrades performance, emphasizing the benefit of cross-modality reconstruction (Fan et al., 2020).

For heterogeneous node classification, HGNN-IMA improved Macro-F1 by approximately 1.2% on DOUBAN, 1.6% on IMDB, and 0.3% on AMAZON over the best baseline, with gains of approximately 2.5% on larger AMAZON-1 and AMAZON-2. Ablations showed declines when removing the Cross-modal Influence Unit, replacing adaptive fusion with mean weights, removing alignment modulation, or removing the attention loss for missing modalities (Li et al., 12 May 2025).

At the same time, multimodal fusion is not uniformly beneficial. In multimodal knowledge graphs, per-modality ablations showed that on DMG (merged), structure alone achieved 0.5917 accuracy, adding textual features yielded 0.7317, adding visual features yielded 0.4042, and using all modalities gave 0.5317; on SYNTH (merged), structure alone yielded 0.6942, visual features alone drove performance to 0.9250, and all modalities yielded 0.8173 (Wilcke et al., 2020). A plausible implication is that a Dual-Modality Node Encoder must manage modality imbalance and modality-specific noise rather than merely concatenate signals.

The principal limitations reported across the literature are data scarcity, privacy restrictions, distribution shift, missing modalities, and graph-quality dependence. The shared-encoder work notes that modality conditioning helps but out-of-distribution robustness still depends on pretraining diversity, and that graph topology may introduce biases when modalities are missing or noisy (Roy et al., 3 Mar 2025). HGNN-IMA reports quadratic scaling in the number of modalities for its alignment modulation term, although the overhead is modest for two modalities (Li et al., 12 May 2025). Related multimodal alignment work also reports that stronger cross-modal alignment can come with “a small cost in cross-scene discernibility,” indicating a general tension between invariance and specificity (Kabra et al., 27 Feb 2026).

Current directions in the cited papers include extending dual-modality designs to more than two modalities, adding contrastive or mutual-information alignment terms, using factorized attention or vector quantization to separate modality-invariant and modality-dependent components, and integrating modality-aware conditioning with graph propagation or downstream generative modules (Li et al., 6 Apr 2026, Li et al., 12 May 2025). Across these variants, the defining property remains stable: a Dual-Modality Node Encoder is an encoder whose primary function is to transform two heterogeneous signals associated with a node into a representation that preserves modality-specific information, models cross-modal interaction, and remains usable by graph or task-specific layers.