Cross-Modality Fusion for Source Code

• Cross-modality fusion is the integration of diverse code modalities, including tokens, ASTs, graphs, and comments, to yield robust, semantic-rich representations.
• It employs deep neural architectures utilizing techniques like cross-attention, hierarchical fusion, and contrastive pre-training to combine signals from each modality.
• Empirical studies report significant gains in tasks such as summarization, retrieval, and vulnerability detection, demonstrating enhanced syntactic fidelity and robustness.

Cross-modality fusion for source code refers to the integration of heterogeneous sources of information—most commonly code tokens, abstract syntax trees (ASTs), control/data-flow graphs, and accompanying natural language comments—into unified vector representations suitable for downstream tasks such as code summarization, retrieval, translation, completion, and vulnerability detection. The fusion is typically realized by deep neural architectures that encode each modality separately or jointly, and then employ customized attention or alignment mechanisms to integrate relevant signals across modalities. Recent research demonstrates that cross-modal fusion improves the semantic richness, syntactic fidelity, robustness, and generalization of code representations over unimodal or naively concatenated approaches.

1. Modalities in Source Code Representation

Modern source code exhibits multiple inherently distinct modalities:

• Token Sequences: Lexical token streams capturing surface syntax and identifier usage.
• Abstract Syntax Trees (ASTs): Hierarchical grammar trees encoding language rules and syntactic structure.
• Control/Data-Flow Graphs (CFGs/DFGs): Graphs representing control or data dependencies and operational semantics.
• Natural Language Comments: Inline documentation that conveys intent or semantic rationale.

Modalities capture complementary properties: tokens record lexical content, ASTs encapsulate grammatical correctness and nesting, CFGs reveal execution paths, and comments provide human-centric semantics. Combining these views is essential for complete program understanding and resilient automated reasoning (Guo et al., 2022, Wang et al., 2021, Wan et al., 2019, Tran-Duong et al., 3 Apr 2026).

2. Architectural Paradigms for Fusion

Approaches to cross-modality fusion can be organized into several architectural classes:

1. Parallel Encoding + Late Fusion: Each modality is encoded using dedicated neural blocks (e.g., LSTM for tokens, GCN/TreeLSTM for AST/graph) and then the resulting vectors are concatenated and projected into a single embedding. This approach is exemplified by MMAN, where intra-modal attention is first computed, then the attended vectors are concatenated and mapped via a learned projection (Wan et al., 2019).
2. Cross-Attention and Alignment Fusion: Beyond concatenation, more sophisticated mechanisms allow the network to learn pairwise or element-level alignments. MMF3 introduces fine-grained fusion at the token/node level by explicitly aligning AST leaves with code tokens and aggregating their embeddings, complemented with standard cross-attention across full representations (Ma et al., 2022). M2TS couples multi-scale GCN outputs for AST with code token Transformers using inter-modal attention that highlights salient token features for each AST node (Gao et al., 2022).
3. Early Fusion in Graphs: Techniques such as GN-Transformer first merge sequence and structure by constructing a heterogeneous graph (e.g., Syntax-Code-Graph), followed by a unified graph-based self-attention mechanism over nodes of all types. This unification enables message-passing and joint encoding from the earliest layers (Cheng et al., 2021).
4. Adaptive and Hierarchical Fusion: In ContractShield, modalities are fused hierarchically: intra-modal self-attention refines each modality, pairwise cross-attention integrates signals, and a final adaptive weighting layer discounts unreliable modalities (e.g., under obfuscation) (Tran-Duong et al., 3 Apr 2026).
5. Masking and Prefix Token Control: UniXcoder employs a shared Transformer whose self-attention mask and behavioral “mode” are controlled by prefix-tokens, thus realizing encoder-only, decoder-only, or encoder-decoder semantics in a single architecture (Guo et al., 2022).
6. Contrastive Mutual Information Maximization: Models such as SynCoBERT and UniXcoder apply multi-modal contrastive pre-training objectives, forcing embeddings of different modalities (tokens, ASTs, comments) of the same code fragment to align in latent space (Wang et al., 2021, Guo et al., 2022).

3. Modality Integration: Mechanisms and Losses

Fusion mechanisms range from simple to highly interaction-aware.

Fusion Technique	Modalities Handled	Integration Principle
Concatenation + MLP	Tokens, AST, CFG	Late vector concatenation + linear mapping
Element-level alignment	Tokens, AST	Pairwise addition of matching embeddings
Graph co-encoding	Tokens + AST	Joint graph (heterogeneous) node self-attention
Cross-attention	Tokens, AST	AST as queries, code as keys/values
Multi-modal contrastive	Tokens, AST, comment	InfoNCE loss on paired/unpaired modalities
Hierarchical attention	Code, opcode, graph	Self-attn, cross-attn, adaptive weighted sum

Contrastive objectives such as InfoNCE (Guo et al., 2022, Wang et al., 2021) and specialized pre-training losses (identifier prediction, AST edge prediction) directly enforce semantic harmony across modalities. Generation objectives (e.g., code-to-comment in UniXcoder) tie program representations across language boundaries and foster modality-agnostic embeddings.

4. Empirical Impact of Cross-Modality Fusion

Empirical ablation across benchmarks reveals that each modality and fusion objective produces distinct, complementary gains:

• Code Search: Adding AST and comment to token-level models increases MRR from 71.5% to 74.4% (UniXcoder) and from 71.3% to 74.0% (SynCoBERT) (Guo et al., 2022, Wang et al., 2021).
• Clone Detection: MAP@R in POJ-104 rises from 88.7% (CodeT5-baseline) to 90.5% (UniXcoder) when AST and contrastive fusion are included (Guo et al., 2022).
• Summarization: MMF3 achieves BLEU-4 gains of ∼1.5–2 points over coarse or purely sequential baselines by applying fine-grained AST-token fusion (Ma et al., 2022); M2TS’s multi-scale AST fusion and ACF boost BLEU-4 by +3.19 points compared to single-scale (Gao et al., 2022).
• Vulnerability Detection under Obfuscation: ContractShield's hierarchical attention and adaptive weighting limit the Hamming score drop to just 1–3% under adversarial transformations, outperforming late-fusion baselines by 6–15% F1 (Tran-Duong et al., 3 Apr 2026).

Ablations indicate that replacing element-level fusion with holistic (coarse) alignment erodes performance, and omitting attention from intra- or inter-modal fusion similarly degrades accuracy and interpretability (Wan et al., 2019, Ma et al., 2022).

5. Technical and Methodological Innovations

Recent research has introduced several technical advancements:

• Injective AST Linearizations: Mapping ASTs into sequences via bijective operators (e.g., $\mathcal{F}$ in UniXcoder) preserves tree structure and enables seamless inclusion in sequential models (Guo et al., 2022).
• Fine-Grained AST-Token Correspondence: MMF3’s explicit leaf-node/token string matching and summation at the embedding level embeds identifier semantics directly (Ma et al., 2022).
• Multi-Scale Graph Convolutions: M2TS’s successive powers of adjacency matrices capture both local and global connectivity patterns in ASTs, which are then aggregated and fused with code token features (Gao et al., 2022).
• Hierarchical Attention with Adaptive Weights: ContractShield’s three-level architecture dynamically calibrates the importance of each modality in response to obfuscation or signal corruption (Tran-Duong et al., 3 Apr 2026).
• Prefix-Token Behavioral Control: In UniXcoder, prefix tokens steer the same Transformer weights into encoder, decoder, or seq2seq function without parameter duplication (Guo et al., 2022).
• Contrastive Pre-Training: Both UniXcoder and SynCoBERT align code, AST, and NL comment representations via InfoNCE losses, maximizing mutual information across modalities and enforcing semantic consistency (Wang et al., 2021, Guo et al., 2022).

6. Task Coverage and Cross-Modal Generalization

Cross-modality fusion has been applied to a range of major code intelligence tasks:

• Code Summarization: Integration of token, AST, and comment modalities produces summaries with higher BLEU, METEOR, ROUGE-L, and CIDER metrics on large Java and Python corpora; human expert studies also confirm improved clarity and correctness (Gao et al., 2022, Ma et al., 2022, Cheng et al., 2021).
• Code Search and Retrieval: Multi-modal attention boosts retrieval rates, notably when queries or code are obfuscated or contain mismatched identifiers (Wan et al., 2019, Guo et al., 2022, Wang et al., 2021).
• Clone Detection and Defect Detection: Fused embeddings detect semantic equivalence and functional defects more robustly; modal ablations consistently reduce accuracy (Guo et al., 2022, Wang et al., 2021).
• Cross-Language and Zero-Shot Transfer: Aligning code from multiple programming languages via a shared NL comment “pivot” or AST-linearizations yields notable gains in zero-shot code-to-code search (Guo et al., 2022).
• Vulnerability Identification in Adversarial Settings: ContractShield’s dynamic fusion strategy maintains high F1 and Hamming scores despite source-level or bytecode-level obfuscations (Tran-Duong et al., 3 Apr 2026).

7. Challenges, Limitations, and Future Directions

Despite clear empirical advantages, cross-modality fusion faces several technical challenges:

• Alignment Robustness: Fine-grained alignment in MMF3 is currently string-based; a learned or soft attention-based alignment could generalize to obfuscated or renamed identifiers (Ma et al., 2022).
• Extension to Additional Modalities: Several approaches fuse only token and AST modalities; integrating data-flow, API usage, or compilation artifacts is an open area (Cheng et al., 2021, Wan et al., 2019).
• Generalization Across Languages and Corpora: Most experiments are limited to Java and Python; comprehensive cross-language validation and application to more diverse industrial codebases remains a key direction (Ma et al., 2022).
• Human Evaluation Metrics: Automated measures (BLEU, METEOR, etc.) have weak correlation with human judgment of code summaries and explanations; integrating human-in-the-loop evaluation is needed (Ma et al., 2022, Cheng et al., 2021).
• Scalability vs. Richness: Early fusion graph approaches such as GN-Transformer and multi-modal GATs require large memory and computation per example; further research must balance expressiveness with scalability (Cheng et al., 2021, Tran-Duong et al., 3 Apr 2026).
• Adaptive Fusion Under Adversarial Conditions: ContractShield demonstrates the promise of dynamic modality weighting to circumvent obfuscation, but generalized approaches for automatically assessing signal reliability remain an open problem (Tran-Duong et al., 3 Apr 2026).

A plausible implication is that future progress will involve generalizing element-level alignment to learned, soft mechanisms; integrating additional semantic views; formalizing uncertainty-aware and adversarially robust fusion; and developing metrics that tie model outputs more strongly to developer-centric measures of utility and correctness.

---

References

• (Guo et al., 2022) UniXcoder: Unified Cross-Modal Pre-training for Code Representation
• (Ma et al., 2022) MMF3: Neural Code Summarization Based on Multi-Modal Fine-Grained Feature Fusion
• (Wan et al., 2019) Multi-Modal Attention Network Learning for Semantic Source Code Retrieval
• (Cheng et al., 2021) GN-Transformer: Fusing Sequence and Graph Representation for Improved Code Summarization
• (Wang et al., 2021) SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation
• (Tran-Duong et al., 3 Apr 2026) ContractShield: Bridging Semantic-Structural Gaps via Hierarchical Cross-Modal Fusion for Multi-Label Vulnerability Detection in Obfuscated Smart Contracts
• (Gao et al., 2022) M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarization