Deep Graph-Language Fusion for Structure-Aware Code Generation

Abstract: Pre-trained LLMs (PLMs) have the potential to transform software development tasks. However, despite significant advances, current PLMs struggle to capture the structured and relational attributes of code, such as control flow and data dependencies. This limitation is rooted in an architectural mismatch: whereas code structure is best represented by graphs, transformer-based LLMs process input as sequential token patterns and therefore lack explicit structural awareness. While recent research has explored integrating graph-based code representations using techniques like graph feature extraction, retrieval-augmented generation, and prompt engineering, existing approaches suffer from information loss during dense feature extraction or prompt encoding; notably, the potential of deep, token-level fusion of graph features within model internals has not been systematically explored. In this paper, we initiate such an exploration by introducing CGFuse, a novel framework that enables token-level integration of graph-derived representations by infusing learned graph features directly into the intermediate layers of pre-trained LLMs. CGFuse combines a graph neural network (GNN) with a LLM to explicitly preserve and exploit fine-grained structural information from code graphs, including abstract syntax trees and data-flow graphs. We systematically evaluate CGFuse across multiple LLMs, demonstrating up to 10-16% BLEU and 6-11% CodeBLEU improvements in code generation performance. These results highlight the potential of deep graph-PLM integration to advance the field toward more robust, capable AI-driven software development.

• The paper introduces CGFuse, which injects AST and data-flow graph representations into intermediate language-model layers through a residual fusion mechanism for structure-aware code generation.
• CGFuse improves BLEU, CodeBLEU, and exact-match scores across nine language models, with PLBART gaining 17.2 BLEU points and CodeT5 reaching 30.3 exact match on CONCODE.
• Shallow one-layer GNN experts consistently outperform deeper variants, while graph-enhanced natural-language models can surpass code-pretrained models despite substantially less pre-training, though single-seed testing and string-based metrics limit the evidence.

Motivation and problem statement

Transformer-based code models process source code as sequential token patterns, which limits their ability to represent structural relationships such as control flow and data dependencies. Prior work has addressed this mismatch through graph feature extraction, retrieval-augmented generation (RAG), soft prompting with GNN-generated features, or prefix-tuning with GNN experts (e.g., Pass-Tuning), but all of these operate on dense graph representations and incur information loss during feature extraction or prompt encoding. The paper argues that token-level fusion of graph features inside the model's intermediate layers — rather than at the input boundary — remains unexplored, and proposes CGFuse to fill this gap (2605.03689).

The CGFuse framework

CGFuse proceeds in three stages. First, each code snippet is represented as a code graph $$\mathcal{G}_c = (\mathcal{V}, \mathcal{E}, \mathcal{R})$$ that augments an abstract syntax tree (AST) with data-flow edges of types comingFrom ($co$) and calculatedBy ($ca$), alongside AST parent edges ($p$). Nodes are split into syntax nodes $\mathcal{S}$ (non-terminal constructs) and terminal nodes $\mathcal{T}$ (code tokens), following the GraphCodeBERT convention.

Second, a GNN "graph expert" is pre-trained on a node classification task over these graphs. Terminal nodes are initialized with PLM token embeddings; syntax nodes receive learned embeddings. Aggregation optionally uses relational message passing over the three edge types.

Third, the pre-trained GNN is attached to a target layer $l$ of the PLM (in the reported configuration, the decoder's last layer), and its node representations are added residually:

$h_f^l = h_\theta^l + \lambda h_g^l$

where $\lambda$ controls fusion strength. Only nodes present in the current context are fed to the GNN, and out-of-context edges are masked. Training first warms up the GNN alone for several epochs, then jointly fine-tunes the fused model on the downstream task.

Experimental setup

The evaluation covers nine PLMs spanning encoder-only (BERT, RoBERTa, CodeBERT, GraphCodeBERT, UniXCoder) and encoder-decoder architectures (BART, T5, PLBART, CodeT5), with parameters ranging from 139M to 285M. Three GNN backbones are compared: R-GCN, GraphSAGE (GS), and GIN. Experiments use the CONCODE dataset (~104k Java classes) for natural-language-to-code generation, evaluated with Exact Match (EM), BLEU, and CodeBLEU. All results come from single runs with one seed and a single-layer GNN expert — a design choice whose statistical robustness is not established in the paper.

Results

Gains across all nine models. Fusing a 1-layer GNN improves every tested PLM on all three metrics. Representative results include:

Model	BLEU (base → fused)	CodeBLEU	EM
BERT$_{R\text{-}GCN}$	33.4 → 48.5	31.2 → 43.1	8.0 → 10.1
RoBERTa$co$0	36.8 → 52.6	33.1 → 45.0	11.6 → 15.6
UniXCoder$co$1	40.5 → 57.5	35.9 → 49.5	14.0 → 21.2
PLBART$co$2	42.1 → 59.3	36.6 → 51.5	18.2 → 27.6
CodeT5$co$3	42.7 → 60.0	36.8 → 51.5	18.2 → 30.3

PLBART shows the largest gains among code-pretrained models (+17.2 BLEU, +14.9 CodeBLEU, +9.4 EM). These improvements fall within the paper's headline claim of 10–16% BLEU and 6–11% CodeBLEU gains.

NL-pretrained models benefit most. Models never exposed to code during pre-training gain substantially from graph infusion (e.g., BERT's +15.1 BLEU), indicating that structural embeddings can inject code knowledge without code-specific pre-training.

Sample efficiency versus code-pretrained counterparts. Using initialization chains such as RoBERTa → CodeBERT → GraphCodeBERT, the authors show that CGFuse-enhanced NL models outperform their code-pretrained descendants despite orders-of-magnitude less training: e.g., RoBERTa$co$4 beats CodeBERT by 15.7 BLEU while CodeBERT required roughly 137× more training steps and ~8M additional pretraining samples; against UniXcoder the gap is 12.1 BLEU despite 547× more steps and ~36M samples. This is arguably the strongest claim in the paper: token-level graph fusion may substitute for expensive code pre-training. The comparison excludes the T5 family because T5's limited vocabulary makes CodeBLEU unreliable — an acknowledged caveat.

Shallow GNNs win decisively. Averaged across the nine PLMs, single-layer GNNs dominate: R-GCN achieves 53.3 BLEU / 46.0 CodeBLEU / 17.5 EM at 1 layer, but performance collapses at 2–3 layers (e.g., GIN drops from 52.0 BLEU at 1-L to 14.1 at 2-L, and EM falls below 2). The authors attribute this to over-smoothing or noise accumulation in deeper aggregation. The practical implication is that only very shallow graph experts are useful for this fusion scheme, which constrains how much multi-hop structural context the mechanism can convey.

Limitations and open questions

The paper concedes several constraints. All experiments use a single run and seed, so variance is unquantified. Fusion was applied only at the decoder's last layer with a fixed $co$5 per model; the sensitivity of results to fusion depth and strength beyond the reported configurations is unexplored. Training a new GNN expert for every architectural or vocabulary change is computationally costly, and integrating GNN experts into decoders requires constructing graphs from incomplete or ambiguous partial code during generation — a problem the paper identifies but does not solve. Evaluation is limited to CONCODE with string-matching metrics (EM, BLEU, CodeBLEU); functional-correctness benchmarks such as HumanEval and MBPP are explicitly left as open validation targets, as is scalability to larger PLMs.

Conclusion

CGFuse demonstrates that infusing shallow-GNN representations of AST-plus-data-flow graphs directly into PLM intermediate layers yields consistent BLEU/CodeBLEU/EM improvements across nine models, and that NL-pretrained models augmented this way can surpass heavily code-pretrained counterparts at a fraction of the training cost. The approach's dependence on single-layer GNNs, single-seed evaluation, string-matching metrics, and per-architecture retraining leaves open whether the reported gains generalize to larger models, deeper fusion, and execution-based assessment of correctness.