SG-Trans: Structure-Guided Code Summarization

• The paper introduces SG-Trans, a Transformer model that incorporates token-level, statement-level, and data flow structural biases to enhance code summarization.
• It modifies self-attention by applying hard-masking for local symbolic structures and an emphatic bias for global syntactic relations, aligning attention with program semantics.
• Empirical results on Java and Python datasets show significant METEOR score improvements, validating the model’s hierarchical structure-variant attention design.

Code summaries are natural-language descriptions of source code that help developers comprehend programs and reduce the time required to infer program functionalities during software maintenance. "Code Structure Guided Transformer for Source Code Summarization" introduces SG-Trans, a Transformer-based approach for code summarization that incorporates code structural properties into self-attention as inductive bias, with explicit handling of both local symbolic information and global syntactic structure (Gao et al., 2021). The model is designed around the observation that programming languages are highly structured and that effective integration of such structure into Transformer-based summarization had been under-explored in this task domain (Gao et al., 2021).

1. Scope and problem setting

SG-Trans addresses source code summarization, a generation task in which a model produces a natural-language summary from source code. Prior work in this area had already shown that Transformer-based approaches achieve promising performance, but the integration of code structure into Transformer architectures remained limited (Gao et al., 2021).

The paper frames code structure as a source of inductive bias rather than as an external post-processing signal. This design choice places SG-Trans within a line of structure-aware code models that modify attention to reflect properties of program representations rather than treating source code as a flat token sequence. Related approaches include the Structure-induced Transformer (SiT), which encodes multi-view structural clues such as AST, control flow, and data dependency in self-attention (Wu et al., 2020), and the Code Transformer, which jointly learns from code context and AST-derived structure using multiple pairwise relations (Zügner et al., 2021). SG-Trans differs by emphasizing a particular division between local symbolic structure and global syntactic structure, and by distributing these biases across layers and heads in a hierarchical manner (Gao et al., 2021).

2. Structural information modeled by SG-Trans

SG-Trans explicitly incorporates three types of code structural properties (Gao et al., 2021).

First, it uses local symbolic structure at the token level, capturing relations among sub-tokens originating from the same code token. This is motivated by the observation that sub-tokens from the same code token, such as isPrime split into is and Prime, are more semantically related (Gao et al., 2021).

Second, it uses local symbolic structure at the statement level, capturing relations among tokens belonging to the same code statement. This encodes the fact that tokens co-occurring within a statement are often more contextually related than tokens elsewhere in the sequence (Gao et al., 2021).

Third, it uses global syntactic structure through the data flow graph (DFG), which captures data dependencies among variables and tokens at the program level (Gao et al., 2021). In the paper’s terminology, these structures jointly cover syntax-level and semantic-level information in code.

This partition of structure into local and global components is central to the model. A plausible implication is that SG-Trans treats fine-grained lexical and statement-level regularities differently from longer-range program semantics, rather than compressing them into a single structural representation. That organization is consistent with the paper’s later hierarchical allocation of attention heads across layers (Gao et al., 2021).

3. Structure-guided self-attention

The core architectural modification in SG-Trans is structure-guided self-attention, which injects local and global code structural information into the self-attention mechanism as inductive bias (Gao et al., 2021). The model defines distinct attention variants for token-level, statement-level, and data-flow structure.

For token-guided self-attention, the paper constructs an adjacency matrix $\mathbf{T} \in \mathbb{R}^{n \times n}$:

$t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$

and modifies attention as

$$\text{head}_t = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal}{\sqrt{d}} + \mathbf{T}\right)\mathbf{V}.$$

Its effect is to allow attention only between sub-tokens of the same token (Gao et al., 2021).

For statement-guided self-attention, the model defines an adjacency matrix $\mathbf{S} \in \mathbb{R}^{n \times n}$:

$s_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th tokens are from the same statement} \ -\infty & \text{otherwise} \end{cases}$

and uses

$$\text{head}_s = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal}{\sqrt{d}} + \mathbf{S}\right)\mathbf{V}.$$

This constrains attention to pairs of tokens within the same statement (Gao et al., 2021).

For data flow-guided self-attention, the model introduces an adjacency matrix $\mathbf{D}$:

$d_{ij}= \begin{cases} 1 & \text{if there is a data flow from token $j$to$i$} \ 0 & \text{otherwise} \end{cases}$

and incorporates a tunable factor $\mu$:

$$\text{head}_f = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal + \mu \cdot (\mathbf{QK}^\intercal \mathbf{D})}{\sqrt{d}}\right)\mathbf{V}.$$

This mechanism enhances attention between tokens with direct data-flow relations (Gao et al., 2021).

The contrast with vanilla self-attention is explicit:

$t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$0

SG-Trans adds structural inductive biases at the attention-logit level via addition for token and statement guidance and via multiplication with the adjacency matrix for data-flow guidance (Gao et al., 2021). The paper states that token- and statement-guided attention use hard masking with $t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$1, whereas data-flow guidance uses an emphatic bias controlled by $t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$2.

4. Hierarchical structure-variant attention

Beyond adding structure to attention, SG-Trans proposes hierarchical structure-variant attention, which distributes different kinds of structural bias across different layers and attention heads (Gao et al., 2021). The stated motivation is the principle of compositionality: higher-level meaning is built from low-level parts.

In this design, local structure—tokens and statements—is emphasized in lower layers and heads, while global structure—data flow—becomes more dominant in higher layers and heads (Gao et al., 2021). Let $t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$3 be the number of layers, $t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$4 the number of heads per layer, and $t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$5 a hyperparameter controlling the distribution. In layer $t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$6, the numbers of heads are defined as

$t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$7

$t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$8

$t_{ij}= \begin{cases} 0 & \text{if $i$-th and$j$-th subtokens are from the same code token} \ -\infty & \text{otherwise} \end{cases}$9

where $$\text{head}_t = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal}{\sqrt{d}} + \mathbf{T}\right)\mathbf{V}.$$0 is the number of token-guided heads, $$\text{head}_t = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal}{\sqrt{d}} + \mathbf{T}\right)\mathbf{V}.$$1 the number of statement-guided heads, $$\text{head}_t = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal}{\sqrt{d}} + \mathbf{T}\right)\mathbf{V}.$$2 the number of data flow-guided heads, and $$\text{head}_t = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal}{\sqrt{d}} + \mathbf{T}\right)\mathbf{V}.$$3 the number of standard self-attention heads without structural bias (Gao et al., 2021).

The layer-level multi-head attention is then

$$\text{head}_t = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal}{\sqrt{d}} + \mathbf{T}\right)\mathbf{V}.$$4

where $$\text{head}_t = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal}{\sqrt{d}} + \mathbf{T}\right)\mathbf{V}.$$5 denotes concatenation and $$\text{head}_t = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal}{\sqrt{d}} + \mathbf{T}\right)\mathbf{V}.$$6 is an output projection (Gao et al., 2021).

This architecture operationalizes a hierarchy of structure rather than merely aggregating multiple relations. A plausible implication is that SG-Trans treats structure not only as an attention constraint but also as a depth-dependent representation prior, aligning lower-layer processing with local syntax and upper-layer processing with broader program semantics.

5. Empirical evaluation

The empirical study uses two benchmark datasets: Java: 87,136 samples and Python: 92,545 samples (Gao et al., 2021). Evaluation is reported with BLEU-4, METEOR, and ROUGE-L (Gao et al., 2021).

The paper reports the following METEOR results for representative systems:

Model	Java (METEOR)	Python (METEOR)
NeuralCodeSum	27.46	19.96
GREAT	27.15	19.75
CodeTransformer	24.22	14.29*
Transformer+GNN	27.17	20.12
SG-Trans	27.85*	20.52*

On both datasets, SG-Trans outperforms all listed baselines (Gao et al., 2021). The abstract summarizes the main gain as follows: compared with the best-performing baseline, SG-Trans improves 1.4% and 2.0% in terms of METEOR score on the two benchmark datasets (Gao et al., 2021). The paper also states that “*” indicates $$\text{head}_t = \mathrm{softmax}\left(\frac{\mathbf{QK}^\intercal}{\sqrt{d}} + \mathbf{T}\right)\mathbf{V}.$$7 in t-tests.

The experimental analysis further reports that each structural type, as well as hierarchical attention, is crucial to peak performance (Gao et al., 2021). Human evaluators consistently rated SG-Trans outputs as more adequate, concise, and fluent than baselines, and heatmap analysis showed that SG-Trans attention heads align well with code structure, unlike vanilla Transformer (Gao et al., 2021). These observations suggest that the model’s improvements are not limited to automatic metrics but are also reflected in the organization of attention and in human judgment of summary quality.

6. Position within structure-aware code modeling

SG-Trans belongs to a broader research trajectory that seeks to align Transformer computations with structural properties of code rather than relying solely on sequential token order. The Structure-induced Transformer represents code structure as a multi-view adjacency matrix over AST, control flow, and data dependency, then masks or re-weights attention accordingly (Wu et al., 2020). The Code Transformer instead generalizes relative-position attention to multiple structural relations, including contextual distance, AST shortest-path length, ancestor distance, sibling distance, and Personalized PageRank, and uses only language-agnostic features computed from source code and ASTs (Zügner et al., 2021).

Later work extends this line in other directions. StructCoder makes both encoder and decoder structure-aware, using AST and DFG information in the encoder and auxiliary tasks for AST paths prediction and data flow prediction in the decoder (Tipirneni et al., 2022). AST-T5 injects structure at pretraining time through AST-aware segmentation and AST-aware span corruption without changing the underlying Transformer architecture (Gong et al., 2024). CSA-Trans focuses on AST-based node positional encoding and SBM attention, reporting that it outperforms SG-Trans and other baselines in code summarization while being faster and more memory efficient in the reported comparison (Oh et al., 2024).

Within this landscape, SG-Trans is distinguished by two specific commitments (Gao et al., 2021). The first is the separation of structure into local symbolic information and global syntactic structure. The second is the claim that structural inductive bias is most effectively exploited when it is distributed hierarchically and compositionally through the network. This suggests a model of code representation in which token-level, statement-level, and data-flow information are not interchangeable views, but structurally distinct signals that benefit from different placements in the Transformer stack.

7. Significance and interpretation

The principal significance of SG-Trans lies in showing that source code summarization benefits from modifying self-attention directly with explicit program structure rather than merely appending structure-derived features or linearizing structured inputs (Gao et al., 2021). Its design makes the attention mechanism itself sensitive to token grouping, statement boundaries, and data dependencies, and then organizes these sensitivities across depth through hierarchical structure-variant attention.

The paper’s own summary identifies four contributions: systematically injecting both local symbolic and global syntactic code structure into Transformers, using hard masking for local structure and an emphatic data-flow bias for global structure, hierarchically allocating these biases across layers and heads, and producing new state-of-the-art results on standard datasets (Gao et al., 2021). A plausible implication is that SG-Trans helped establish a concrete template for later structure-aware code Transformers: encode structure inside attention, preserve distinctions among relation types, and align those relations with representational depth.

At the same time, the paper positions its advances within code summarization rather than as a universal solution for all code intelligence tasks. Its results concern two summarization benchmarks and three evaluation metrics, with supporting qualitative and human evaluation evidence (Gao et al., 2021). In that sense, SG-Trans is best understood as a task-specific yet architecturally influential demonstration that Transformer summarizers for source code can benefit materially from explicit, hierarchically organized structural inductive bias.