CodeT5+ Backbone

• CodeT5+ Backbone is a set of architectural augmentations over the original CodeT5, designed for advanced code retrieval, generation, and data-centric learning in software engineering.
• It integrates heterogeneous inputs—such as token segments, AST traversals, expert CWE data, and exemplar fixes—via a Fusion-in-Decoder mechanism to handle long, structured code sequences.
• Empirical results show significant improvements in exact-match, BLEU, and CodeBLEU scores for vulnerability repair and comment generation compared to the baseline.

CodeT5+ Backbone refers to the set of architectural augmentations, input handling strategies, and algorithmic extensions over the vanilla CodeT5 Transformer framework that enable joint retrieval, generation, and data-centric learning for complex software engineering tasks. These enhancements are motivated by limitations in standard sequence-to-sequence modeling for source code, such as inadequate handling of long code sequences, lack of explicit structural understanding, insufficient knowledge injection, and misalignment between retrieval and generation modules.

1. Core Architecture and Baseline Model

CodeT5+ builds on the original CodeT5 architecture, which employs a Transformer encoder-decoder stack for source code-related tasks. For the base configuration, CodeT5-base uses 6 encoder layers and 6 decoder layers, each with hidden dimensionality $d=768$ and $h=12$ self-attention heads. All Transformer modules—multi-head self-attention, feed-forward networks, layer normalization—are preserved from the vanilla CodeT5 specification (Zhou et al., 27 Jan 2024).

Multi-head attention operates as follows. For input embedding $X\in\mathbb{R}^{L\times d}$, head $i$ computes:

$$\text{head}_i = \text{softmax}\left( \frac{(X W_i^Q)(X W_i^K)^T}{\sqrt{d_k}} \right) (X W_i^V)$$

with the multi-head output defined as:

$$\text{MultiHead}(X) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) W^O$$

Positional information is encoded using sinusoidal positional encodings, where for position $pos$ and index $i$,

$$PE_{pos,2i} = \sin\left(\frac{pos}{10000^{2i/d}}\right), \quad PE_{pos,2i+1} = \cos\left(\frac{pos}{10000^{2i/d}}\right)$$

2. Input Representation, Segmentation, and Fusion

CodeT5+ extends input modeling via multi-modal segment representation, tailored for rich program analysis:

• Token Segments ($I_j$): Sub-sequences of the vulnerable function's BPE-tokenized code, each at most 512 tokens.
• AST Segments ($A_j$): Depth-first traversals of the code's abstract syntax tree as "type:value" tokens.
• Expert Knowledge ($D$): CWE type names (short natural-language phrases).
• Exemplar Pairs ($E_k$): Sequences created by concatenating vulnerable example code and ChatGPT-generated fixes.

Each segment $S$ is embedded as:

$E_S = \text{TokenEmb}(S) + \text{PosEmb}(S)$

After independent encoding, segment outputs ($I_j \in \mathbb{R}^{L_j \times d}$, $A_j \in \mathbb{R}^{M_j \times d}$, $D \in \mathbb{R}^{L_D \times d}$, $E_k \in \mathbb{R}^{L_k \times d}$) are flattened and concatenated along the sequence axis to yield a single encoder representation:

$$C_{\text{encoder}} = [I_1; \ldots; I_n; A_1; \ldots; A_m; D; E_1; \ldots; E_K]$$

This "Fusion-in-Decoder" (FiD) procedure enables processing of long and structurally diverse input by deferring full cross-segment attention to the decoder (Zhou et al., 27 Jan 2024).

3. Pre-Training and Code-Structure Adaptation

CodeT5+ incorporates a targeted pre-adaptation phase to improve code structure awareness. A large bug-fixing dataset ($\sim$500K triplets) is used, with 50% of training examples providing raw token sequences and 50% AST node sequences, each truncated to 512 tokens. Training employs standard sequence-to-sequence cross-entropy loss:

$$L_{\text{repair}} = -\log P( Y_i | X_i, AST_i, Name_i, ExampleFixPairs_i )$$

This yields significant gains in downstream sequence generation, specifically enhancing the model's handling of structural code features and augmenting its zero-shot repair capacity (Zhou et al., 27 Jan 2024).

4. Contextual Augmentation and Multi-LLM Collaboration

To exploit expert knowledge and exemplars, CodeT5+ allows seamless integration of external LLM outputs as context. In VulMaster, ChatGPT (GPT-3.5-turbo) is prompted with a vulnerable code snippet and CWE analysis to produce candidate repairs; these become additional input segments ($E_k$). Importantly, the architecture does not invoke ChatGPT during inference—fixes are incorporated statically. No learnable weighting distinguishes ChatGPT segments from others; all are encoded identically.

A lightweight relevance classifier is trained to prioritize exemplars most relevant to the target CWE category. For each $E_k$:

$p_k = \sigma( W_r \cdot E_k^{\text{CLS}} + b_r )$

with cross-entropy loss:

$$L_{\text{relevance}} = -\sum_{k=1}^K \left[ g_k \log p_k + (1-g_k)\log(1-p_k) \right]$$

where $g_k\in\{0,1\}$ indicates exact CWE match (Zhou et al., 27 Jan 2024).

5. Backbone Extensibility for Joint Retrieval-Generation

The CodeT5+ design generalizes to multi-modal and retrieval-augmented pipelines such as RAGSum (Le et al., 16 Jul 2025). Here, the encoder acts both as retriever (mapping queries and comments to $h_q, h_c \in \mathbb{R}^d$ for nearest-neighbor search) and as input model for the decoder. Retrieval and generation are tightly coupled via contrastive pre-training and a composite loss:

• Contrastive Loss: For a minibatch $\mathcal{B}$,

$$L_{\text{contrast}} = \sum_{i\in\mathcal{B}} (L_{q2q}^i + L_{q2c}^i)$$

where $L_{q2q}$ promotes code embedding alignment and $L_{q2c}$ aligns code with comments using cosine similarity and temperature $\tau$.

• Composite End-to-End Loss: For retrieved neighbors $R_i = \{ (q^r_j, c^r_j) \}$,

$L_i = \frac{1}{k} \sum_{j=1}^k \nu_j L_j$

with $\nu_j = \text{sim}(h_{q_i}, h_{q^r_j})$ weighting the cross-entropy generation loss $L_j$ for each retrieved example.

Self-refinement ("SR") further polishes the backbone by generating multiple candidates for each input and fine-tuning on the best-scoring outputs, scored via ROUGE-L (Le et al., 16 Jul 2025).

6. Performance Impact and Ablation Evidence

Empirical results on code vulnerability repair show the impact of backbone extensions. Using CodeT5+ with FiD, AST, CWE, and relevance yields significant uplifts:

Approach	EM (%)	BLEU	CodeBLEU
CodeT5 (vanilla)	10.2	21.3	32.5
+ bug-fix corpus & CWE data (fine-tuned)	16.8	24.2	35.3
VulMaster (CodeT5+FiD+AST+relevance…)	20.0	29.3	40.9

Ablations show the largest drop (20.0 → 13.6 EM) results from removing AST/code pre-adaptation. The Fusion-in-Decoder module confers ≈3.2 EM gain, while inputting AST structure and CWE context each adds approximately 1–1.2 points (Zhou et al., 27 Jan 2024).

In retrieval-augmented comment generation, RAGSum's CodeT5+ backbone achieves BLEU, METEOR, and ROUGE-L increments of 3–5 points from combined contrastive pre-training and self-refinement, with overall uplifts of 7–14% over state-of-the-art baselines (Le et al., 16 Jul 2025).

7. Significance, Applications, and Limitations

CodeT5+ backbones offer a unified architecture for structurally-aware, context-augmented program repair and code understanding. Key innovations include AST-aware adaptation, fusion of heterogeneous context via FiD, jointly-learned retrieval-generation pipelines, and explicit integration of expert and LLM-generated knowledge. These enable CodeT5+ to outperform existing approaches in both vulnerability repair and code summarization, nearly doubling exact-match and sequence-level metrics compared to vanilla CodeT5 in repair settings (Zhou et al., 27 Jan 2024), and yielding state-of-the-art comment generation (Le et al., 16 Jul 2025).

A plausible implication is that further scalability of this paradigm—especially for even longer code sequences and more expressive context—will depend on system-level advances in efficient Transformer segment handling, fine-grained retrieval conditioning, and continual adaptation to new expert knowledge resources.