CodeBERTa: Compact RoBERTa Code Encoder

• CodeBERTa is a RoBERTa-style code model that uses a six-layer Transformer to encode source code and documentation via masked-language modeling.
• It demonstrates strong syntactic aptitude with a 92.55% accuracy on AST node tagging, emphasizing grammatical regularities over structural complexity.
• Tree-enhanced variants inject AST-based positional embeddings, yielding measurable improvements in masked language modeling and clone detection tasks.

CodeBERTa is a RoBERTa-style adaptation of CodeBERT: a bidirectional Transformer encoder trained on source code and associated documentation via masked-language modeling rather than next-sentence prediction (Karmakar et al., 2021). Within the literature represented here, it appears both as a compact pretrained code model for diagnostic probing and as the base architecture for a later AST-aware variant, "Tree-Enhanced CodeBERTa," which augments its token embeddings with tree-based positional information derived from abstract syntax trees (ASTs) (Bartkowiak et al., 5 Jul 2025). Across these accounts, CodeBERTa is associated most strongly with syntactic competence, while later modifications target its weaker handling of explicit hierarchical program structure.

1. Model identity and lineage

Karmakar and Robbes describe CodeBERTa as a “RoBERTa-style” adaptation of CodeBERT that retains a bidirectional Transformer encoder and is trained on both source code and natural-language documentation, but omits next-sentence prediction and relies exclusively on masked-language modeling (Karmakar et al., 2021). Their reconstruction places it on the CodeSearchNet dataset, with roughly 6.4 million method-level snippets across Java, Python, JavaScript, PHP, Ruby, and Go, together with docstrings. The same source characterizes CodeBERTa as having 6 Transformer layers, half the depth of BERT and CodeBERT.

The two sources describe related but not identical architectural instantiations. The probing paper states that, by design, CodeBERTa uses the same hidden size and head-count as its CodeBERT predecessor—768 dimensions and 12 heads per layer—projected onto 6 layers, with roughly 110 M parameters (Karmakar et al., 2021). The later technical report, focused on a “small” variant used for AST-aware enhancement, specifies 6 Transformer layers with multi-head self-attention and feed-forward sublayers, totaling 83.5 M parameters (Bartkowiak et al., 5 Jul 2025). Taken together, these descriptions identify CodeBERTa less as a single immutable checkpoint than as a compact RoBERTa-style code encoder family centered on 6-layer Transformer backbones.

A recurrent theme in both accounts is the trade-off between efficiency and structural expressiveness. The probing study states that CodeBERTa keeps the per-layer capacity of CodeBERT while reducing depth for faster inference and lighter weight (Karmakar et al., 2021). The later enhancement work, in turn, leaves the core Transformer stack unchanged and modifies only the embedding interface, which suggests that CodeBERTa is particularly amenable to structural augmentation without wholesale architectural redesign (Bartkowiak et al., 5 Jul 2025).

2. Baseline representation scheme

In the technical report’s baseline formulation, each input token is represented as the sum of three learned embeddings: a token embedding $e^{(w)} \in \mathbb{R}^d$, an absolute positional embedding $e^{(p)} \in \mathbb{R}^d$, and a token-type embedding $e^{(t)} \in \mathbb{R}^d$ (Bartkowiak et al., 5 Jul 2025). For the $i$-th token in a sequence of length $N$, the embedding is

$$E_i = e^{(w)}_i + e^{(p)}_i + e^{(t)}_i \in \mathbb{R}^d.$$

This representation supplies the self-attention stack with token identity, linear position, and type information. The report emphasizes, however, that such embeddings capture linear ordering only and do not reflect hierarchical relations in source code’s AST (Bartkowiak et al., 5 Jul 2025). That observation is important because, in code, sibling order, nesting depth, and parent-child structure often encode semantics or constrain valid continuations more strongly than mere token adjacency.

The probing study indirectly supports this distinction between linear and structured information. Its diagnostic tasks separate surface-level properties such as snippet length from syntactic, structural, and semantic properties, and CodeBERTa’s performance varies substantially across those categories (Karmakar et al., 2021). This suggests that standard pretrained token representations can encode some code properties strongly while leaving others only weakly linearly decodable.

3. Diagnostic evidence about what CodeBERTa encodes

Karmakar and Robbes evaluate CodeBERTa with four probing tasks built as single linear classifiers on frozen hidden-layer embeddings: length prediction (LEN), AST node tagging (AST), cyclomatic complexity prediction (CPX), and invalid type detection (TYP) (Karmakar et al., 2021). Because the probes are linear, the results are explicitly about how easily these properties can be read off the pretrained representations rather than about full task-specific fine-tuning.

Probe	CodeBERTa accuracy	Context
LEN	70.35%	Competitive, but behind GraphCodeBERT at 71.10% and BERT at 76.05%
AST	92.55%	Best among all four tested models
CPX	40.80%	Weaker than BERT, CodeBERT, and GraphCodeBERT
TYP	90.10%	Better than BERT, below CodeBERT and GraphCodeBERT

These results establish CodeBERTa’s strongest reported property as syntax. On AST node tagging, it leads all tested models at 92.55%, exceeding both CodeBERT and GraphCodeBERT (Karmakar et al., 2021). By contrast, on cyclomatic complexity it scores 40.80%, and the same paper states that it lags behind all others except the CodeBERT sibling. On invalid-type detection it reaches 90.10%, outperforming BERT but trailing CodeBERT and GraphCodeBERT. The authors summarize this profile by placing CodeBERTa “squarely in the middle of the pack,” strongest at syntax and weaker on structural complexity.

Two common misconceptions are directly challenged by these probe results. First, code pretraining does not automatically dominate generic language pretraining: the same study notes that plain English BERT performs surprisingly well on some code tasks, especially the surface and structural probes (Karmakar et al., 2021). Second, strong syntactic performance does not imply equally strong structural or semantic competence. CodeBERTa’s 92.55% AST score coexists with materially weaker CPX and non-leading TYP results, indicating that grammatical regularities are more readily linearly extractable than deeper program properties.

4. Layer-wise behavior and sample efficiency

The probing paper also analyzes CodeBERTa layer by layer, extracting embeddings from layer 0 through layer 6 and training probes separately on each representation level (Karmakar et al., 2021). No single layer “owns” all code properties. Surface cues such as LEN are easier to extract from lower layers, syntactic cues such as AST peak in middle layers, structural cues such as CPX are noisy but slightly stronger in mid-to-upper layers, and semantic cues such as TYP tend to improve toward the top.

This stratification is consistent with a layered representational pipeline in which local lexical and formatting regularities emerge earlier, while progressively transformed states support more abstract separability later. The authors also report irregular “spikes and dips” across layers that flatten as more probe data are provided (Karmakar et al., 2021). That observation matters methodologically: apparent layer specialization can partly reflect limited probe supervision rather than stable architectural roles.

In sample-efficiency experiments on TYP, the paper highlights a broader comparison in which GraphCodeBERT with 100 samples beats BERT with 10,000. For CodeBERTa specifically, the reported behavior is monotonic improvement as the probe data increase from 1% to 10% to 100%, and by approximately 1,000 samples it reaches within a few points of its full-data performance (Karmakar et al., 2021). A plausible implication is that CodeBERTa’s pretrained states make several code properties accessible with relatively modest linear supervision, even if not as efficiently as models pretrained with explicit structural bias such as data-flow.

5. Tree-based positional augmentation

The 2025 technical report extends CodeBERTa by introducing tree-based positional embeddings that encode AST hierarchy explicitly through node depth and sibling index (Bartkowiak et al., 5 Jul 2025). For an AST node $x$ with parent $f(x)$ and zero-based sibling index $i_x$, the hierarchical position is defined as

$F(x) = (d_x, s_x),$

with recursive specification

$e^{(p)} \in \mathbb{R}^d$0

Two learned embedding tables are then allocated:

$e^{(p)} \in \mathbb{R}^d$1

where row $e^{(p)} \in \mathbb{R}^d$2 of $e^{(p)} \in \mathbb{R}^d$3 yields the depth embedding $e^{(p)} \in \mathbb{R}^d$4 and row $e^{(p)} \in \mathbb{R}^d$5 of $e^{(p)} \in \mathbb{R}^d$6 yields the sibling embedding $e^{(p)} \in \mathbb{R}^d$7 (Bartkowiak et al., 5 Jul 2025). The structural embedding $e^{(p)} \in \mathbb{R}^d$8 is formed by aggregating these components. The report studies three aggregation strategies:

• Sum aggregation:

$e^{(p)} \in \mathbb{R}^d$9

• Weighted sum aggregation:

$e^{(t)} \in \mathbb{R}^d$0

• Concatenation + projection:

$e^{(t)} \in \mathbb{R}^d$1

Integration occurs at the embedding level. In the weighted-sum setting, for example, the final input becomes

$e^{(t)} \in \mathbb{R}^d$2

The report states that these $e^{(t)} \in \mathbb{R}^d$3 replace the standard embeddings at the bottom of each Transformer layer, after which self-attention, feed-forward sublayers, residual connections, and layer normalization remain unchanged (Bartkowiak et al., 5 Jul 2025). This is an embedding-space intervention rather than a graph encoder or message-passing redesign.

6. Training objectives, empirical gains, and interpretive significance

The AST-aware study evaluates baseline and Tree-Enhanced CodeBERTa on two tasks: masked language modeling pretraining on CodeSearchNet and clone detection fine-tuning on the PoolC clone detection dataset, described as approximately 600 K pairs (Bartkowiak et al., 5 Jul 2025). For clone detection, code pairs $e^{(t)} \in \mathbb{R}^d$4 receive labels $e^{(t)} \in \mathbb{R}^d$5 for nonclone or clone; pooled vector representations are concatenated, mapped to a probability with a sigmoid classifier, and optimized with binary cross-entropy. All reported runs use three seeds $e^{(t)} \in \mathbb{R}^d$6, AdamW with $e^{(t)} \in \mathbb{R}^d$7, batch size 32, and three epochs.

Task	Baseline CodeBERTa	Tree-Enhanced CodeBERTa (Weighted Sum)
MLM	loss 0.44388; Acc 0.8972; F1 0.8939; P 0.8953; R 0.8972	loss 0.41417; Acc 0.9029; F1 0.8999; P 0.9012; R 0.9029
Clone detection	loss 0.25836; Acc 0.9173; F1 0.9172; P 0.9180; R 0.9173	loss 0.21799; Acc 0.9187; F1 0.9186; P 0.9191; R 0.9187

The report concludes that the weighted-sum configuration gives the best trade-off, reducing loss by approximately 7–16% while improving F1, precision, and recall consistently across both generative and discriminative settings (Bartkowiak et al., 5 Jul 2025). It further reports t-SNE visualizations of final hidden states colored by AST depth: the Tree-Enhanced model forms clusters that correlate tightly with syntactic depth, with deeper-depth nodes in distinct peripheral clusters and shallow nodes toward the center, whereas the baseline mixes depths indiscriminately. The same section states that early in training the weights $e^{(t)} \in \mathbb{R}^d$8 and $e^{(t)} \in \mathbb{R}^d$9 start high and then gradually diminish, indicating that the model first emphasizes structure and later refines semantic distinctions.

These findings sharpen the broader diagnosis from the probing paper. The probing results already showed that CodeBERTa is particularly strong on syntax but weaker on structure that is not directly recoverable from linear token position (Karmakar et al., 2021). The tree-based extension addresses precisely that gap by supplying explicit AST-derived positional signals, and the observed gains in masked language modeling and clone detection suggest that some of CodeBERTa’s limitations are not due to the Transformer stack alone but to the insufficiency of conventional positional embeddings for hierarchical code (Bartkowiak et al., 5 Jul 2025). A plausible implication is that CodeBERTa’s baseline strength in grammatical regularities can be converted into stronger source-code representations when the embedding layer exposes depth and sibling-order information directly.