AntiBERTa: Antibody Language Model

• AntiBERTa is a self-supervised antibody language model designed to learn representations from immunoglobulin sequences, serving as a foundational benchmark for antibody design and affinity prediction.
• It employs a RoFormer-based architecture with 16 layers and 1024 hidden units, capturing biologically relevant features such as CDR regions, V-gene motifs, and somatic hypermutation patterns.
• AntiBERTa is used as a critical baseline in comparative studies, demonstrating competitive performance, clear inductive biases, and practical insights for antibody-antigen binding classification.

Searching arXiv for AntiBERTa and closely related antibody LLM papers to ground the article. AntiBERTa is an antibody-specific protein LLM developed for representation learning over immunoglobulin sequences. In the literature summarized here, it appears in two distinct but complementary roles. First, it is treated as prior foundational antibody language-model work in antibody design and affinity-prediction studies, where it is cited alongside AntiBERTy, IgFold, AbLang, and AbGPT as part of the antibody-language-model landscape (Hossain et al., 17 May 2025). Second, it is examined as an antibody-specialized sequence encoder whose architectural and pretraining biases appear to align with immunologically salient sequence structure, especially complementarity-determining regions (CDRs), V-gene-associated motifs, and somatic hypermutation (SHM)-related variation (Mengren et al., 10 Dec 2025). The original AntiBERTa publication is “Deciphering the language of antibodies using self-supervised learning” (Ruffolo et al., 2021).

1. Origins and position within antibody language modeling

AntiBERTa originated as a self-supervised LLM for antibody sequences and is explicitly cited in later work as an important early antibody design model (Ruffolo et al., 2021). In subsequent studies, it is grouped with AntiBERTy, IgFold, AbLang, and AbGPT as foundational antibody LLMs that preceded newer affinity-prediction systems (Hossain et al., 17 May 2025). Within that framing, AntiBERTa is not described as an architectural component of later models such as LlamaAffinity; rather, it serves as prior work and as an empirical comparison baseline (Hossain et al., 17 May 2025).

A later comparative study places AntiBERTa at the center of a broader inquiry into architecture-induced biases in protein LLMs for antibody comprehension (Mengren et al., 10 Dec 2025). There, AntiBERTa is explicitly framed as the most antibody-specific model among the evaluated encoders, in contrast to broader models such as ESM2, BioBERT, and GPT-2 (Mengren et al., 10 Dec 2025). This suggests that AntiBERTa’s significance in the literature is not limited to predictive performance; it is also used as a reference point for whether antibody-specialized pretraining produces qualitatively different internal representations.

The available papers therefore support a dual characterization. AntiBERTa is both a historically important antibody LLM and a methodological benchmark for testing whether antibody-native pretraining yields biologically aligned inductive bias (Hossain et al., 17 May 2025, Mengren et al., 10 Dec 2025).

2. Model identity and architectural characterization

The original AntiBERTa paper establishes the model as a self-supervised antibody LLM (Ruffolo et al., 2021). In the later architecture-bias study, the AntiBERTa configuration is reported in a concrete form: Architecture RoFormerForMaskedLM, Number of layers 16, Hidden size 1024, and Attention heads 16 (Mengren et al., 10 Dec 2025). In that same comparison, BioBERT is listed as a 12-layer BertModel with hidden size 768 and 12 heads, ESM2 as a 12-layer EsmForMaskedLM with hidden size 480 and 20 heads, and GPT-2 as a 12-layer GPT2LMHeadModel with hidden size 768 and 12 heads (Mengren et al., 10 Dec 2025).

That description permits only a limited but precise architectural statement. AntiBERTa is treated in these papers as a masked-language-model-style transformer encoder specialized for antibody sequences, and in the 2025 comparative study it is instantiated as a RoFormer-based model with 16 layers and 1024-dimensional hidden states (Mengren et al., 10 Dec 2025). No further tokenizer specification, pretraining-corpus breakdown, or mathematical definition of the original AntiBERTa pretraining objective is given in the supplied downstream studies, so those details cannot be expanded here without exceeding the evidentiary basis.

What can be said reliably is that later authors distinguish AntiBERTa from general protein or text models not merely by downstream accuracy but by domain specialization: AntiBERTa is described as pretrained or fine-tuned specifically for antibody sequences, whereas ESM2 is a broad protein model, BioBERT is a biomedical text model, and GPT-2 is a general text model (Mengren et al., 10 Dec 2025).

3. Experimental role in antibody-affinity classification

In “Llama-Affinity: A Predictive Antibody Antigen Binding Model Integrating Antibody Sequences with Llama3 Backbone Architecture” (Hossain et al., 17 May 2025), AntiBERTa is used as a direct baseline in an antibody-antigen binding affinity prediction benchmark. Despite the phrase “binding affinity prediction,” the task is operationalized as a binary classification task with labels low affinity (label 0) and binder (label 1) (Hossain et al., 17 May 2025). The input is antibody sequences only, specifically heavy and light chains concatenated into a single sequence; no explicit antigen sequence or structure is part of the described input (Hossain et al., 17 May 2025).

The dataset is the Observed Antibody Space (OAS), curated from the official AntiFormer GitHub repository, and evaluation uses five-fold cross-validation with StratifiedKFold (Hossain et al., 17 May 2025). The reported metrics are Accuracy, F1-score, Precision, Recall, and ROC AUC, with the paper giving the following formulas:

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

$$F1 = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

$\text{Precision} = \frac{TP}{TP + FP}$

$\text{Recall} = \frac{TP}{TP + FN}$

$\text{ROC AUC} = \int_0^1 TPR \, d(FPR)$

The paper also presents a loss expression,

$$\text{Loss} = - [y \cdot \log (p) + (1 - y) \cdot \log (1 - p)]$$

and labels it “Sparse Categorical Cross Entropy,” although the displayed equation is the standard binary cross-entropy form (Hossain et al., 17 May 2025).

AntiBERTa’s role in this study is limited but concrete: it is a serious antibody-language-model baseline against which a LLaMA-3-based classifier is compared. The study does not present AntiBERTa as influencing the internal design of LlamaAffinity beyond being part of the broader antibody-language-model context (Hossain et al., 17 May 2025).

4. Reported performance characteristics

The LlamaAffinity paper provides one explicit numerical block for AntiBERTa in its main comparison table (Hossain et al., 17 May 2025). AntiBERTa’s reported results are:

Model	Metric value	Training
AntiBERTa	Accuracy 0.8796	2.97 h
AntiBERTa	F1-score 0.8570	2.97 h
AntiBERTa	Precision 0.9080	2.97 h
AntiBERTa	Recall 0.9090	2.97 h
AntiBERTa	ROC AUC 0.9340	2.97 h

Within that benchmark, AntiBERTa is stronger than the simple 6-layer and 12-layer transformer baselines, weaker than AntiFormer across all listed metrics, and mixed relative to AntiBERTy (Hossain et al., 17 May 2025). Specifically, AntiBERTa exceeds AntiBERTy on Accuracy, F1-score, and Recall, but trails AntiBERTy on Precision and ROC AUC and requires more training time: 2.97 hours versus 1.46 hours (Hossain et al., 17 May 2025). This places AntiBERTa in a middle tier in that study: clearly better than generic transformers, competitive with antibody-specific baselines, but not the top performer (Hossain et al., 17 May 2025).

Against the proposed LlamaAffinity model, AntiBERTa is reported as worse on every listed predictive metric and much slower to train: Accuracy 0.8796 vs 0.9640, F1-score 0.8570 vs 0.9643, Precision 0.9080 vs 0.9702, Recall 0.9090 vs 0.9586, ROC AUC 0.9340 vs 0.9936, and training time 2.97 h vs 0.46 h (Hossain et al., 17 May 2025). The paper explicitly states that LlamaAffinity “outperformed AntiFormer (ROC AUC: 0.9660) and AntiBERTa (ROC AUC: 0.9340) while requiring significantly less training time (0.46 hrs vs. 0.76 hrs and 2.97 hrs, respectively)” (Hossain et al., 17 May 2025).

At the same time, the AntiBERTa comparison in that paper is methodologically narrow. There are no AntiBERTa-specific ROC curves, no confusion matrix, no per-fold breakdown, no significance testing, no confidence intervals, and no ablation explaining why AntiBERTa underperforms relative to LlamaAffinity (Hossain et al., 17 May 2025). The reported result is therefore informative as a practical benchmark outcome but not as a mechanistic diagnosis.

5. Antibody-specific inductive biases

The most detailed characterization of AntiBERTa’s internal behavior appears in “Exploring Protein LLM Architecture-Induced Biases for Antibody Comprehension” (Mengren et al., 10 Dec 2025). In that study, AntiBERTa is used to test whether antibody-specialized pretraining yields distinct biological biases relative to BioBERT, ESM2, and GPT-2. The paper’s central claim is that AntiBERTa appears naturally aligned with antibody biology, especially the regions and latent variables immunologists regard as functionally central (Mengren et al., 10 Dec 2025).

The study reports that AntiBERTa naturally focuses attention on the three CDRs, with increasing emphasis from CDR1 to CDR3 (Mengren et al., 10 Dec 2025). It also shows strong attention to the first 15 amino acids, which the authors connect to V-gene alignment (Mengren et al., 10 Dec 2025). In embedding space, AntiBERTa is reported to separate sequences not only by target specificity but also by V gene family, SHM count, and IgH isotype-associated structure (Mengren et al., 10 Dec 2025). Since only variable-region sequence is used, and constant-region sequence is absent, the inferred isotype sensitivity is interpreted as indirect, plausibly mediated by correlation between isotype and SHM-related sequence statistics (Mengren et al., 10 Dec 2025).

These observations matter because they distinguish AntiBERTa from being merely a competent classifier. The paper argues that its representations reflect biologically meaningful latent structure. AntiBERTa is thus portrayed as encoding both functional motifs associated with antigen recognition and broader immunological correlates such as lineage-associated V-gene usage and maturation-related SHM patterns (Mengren et al., 10 Dec 2025).

A plausible implication is that AntiBERTa’s usefulness arises partly from this antibody-centric inductive bias rather than only from model scale or generic transformer capacity. However, the paper also emphasizes that such bias can be double-edged: if target labels correlate with V-gene family, SHM burden, or isotype-associated maturation signatures, then some predictive success may rely on correlated biology rather than direct binding determinants (Mengren et al., 10 Dec 2025).

6. Interpretability analyses and CDR-focused pooling

The same 2025 study provides the clearest interpretability evidence regarding AntiBERTa (Mengren et al., 10 Dec 2025). The authors compute attention attribution scores from the [CLS] token and examine positionwise contributions in the last hidden layer across all 10,377 sequences in their dataset (Mengren et al., 10 Dec 2025). They report median attribution scores, along with 25th and 75th percentiles, across sequence positions. The exact attribution formula is not given in the paper, so the interpretive claims remain qualitative rather than formally specified (Mengren et al., 10 Dec 2025).

The reported AntiBERTa attention pattern has two salient features. First, there is strong attention to the first approximately 15 residues, which is consistent with V-gene-associated motifs. Second, there are distinct peaks at CDR1, CDR2, and CDR3, with attention increasing progressively toward CDR3, matching the conventional immunological view that CDR3 often plays the dominant role in antigen binding (Mengren et al., 10 Dec 2025). ESM2 and BioBERT also show some CDR-related signal, but the CDR peaks are described as less prominent than AntiBERTa’s (Mengren et al., 10 Dec 2025).

The study also performs a CDR3 mean pooling ablation, comparing full-sequence mean pooling against pooling restricted to the CDR3 region for BioBERT, ESM2, and AntiBERTa with a fully connected classifier (Mengren et al., 10 Dec 2025). For AntiBERTa, the qualitative result is that CDR3 pooling and full-sequence pooling yield almost identical validation-accuracy curves (Mengren et al., 10 Dec 2025). By contrast, ESM2 shows a significant boost in training efficiency under CDR3 pooling, and BioBERT also improves, though less dramatically (Mengren et al., 10 Dec 2025).

The authors interpret this near-invariance in AntiBERTa as evidence that the model already internalizes the privileged role of CDR3, so explicit downstream restriction to CDR3 adds little (Mengren et al., 10 Dec 2025). This is one of the strongest arguments in the current literature that AntiBERTa’s pretraining induces an antibody-specific functional prior rather than merely a generic sequence representation.

7. Limitations, reproducibility issues, and scope of interpretation

The downstream papers are informative about AntiBERTa’s comparative behavior, but they also expose substantial limitations in how AntiBERTa is documented and evaluated.

In the LlamaAffinity benchmark, AntiBERTa is severely under-specified as an implementation (Hossain et al., 17 May 2025). The paper does not state which pretrained AntiBERTa checkpoint was used; whether AntiBERTa was fine-tuned end-to-end or used as a frozen encoder; what classification head, pooling method, tokenization strategy, separator scheme, optimizer, learning rate, batch size, scheduler, epoch count, random seed, or hardware protocol were used specifically for AntiBERTa (Hossain et al., 17 May 2025). The dataset is described as pre-tokenized with the ProtBERT tokenizer and a vocabulary size of 30, but the paper does not clarify how that relates to AntiBERTa’s native tokenizer or checkpoint vocabulary (Hossain et al., 17 May 2025). Consequently, the numerical comparison is useful but not fully reproducible.

The architecture-bias study provides more AntiBERTa-specific setup information, including architecture, layer count, hidden size, attention heads, fine-tuning of the terminal two layers, batch size 128, training duration of approximately 30 epochs, and use of a Google Colab T4 GPU (Mengren et al., 10 Dec 2025). Even there, however, important details remain absent: learning rate, optimizer, weight decay, scheduler, dropout, exact downstream classifier dimensions, split methodology, random seeds, CDR extraction protocol, alignment or numbering convention, and variance across runs (Mengren et al., 10 Dec 2025). The study reports validation accuracy, not clearly defined held-out test performance, and does not provide exact AntiBERTa accuracies in tabular form (Mengren et al., 10 Dec 2025).

Both studies also impose scope limitations on interpretation. The LlamaAffinity benchmark uses concatenated heavy and light chains but no antigen features, so the task is closer to binder classification from antibody sequence patterns than explicit paired antibody-antigen interaction modeling (Hossain et al., 17 May 2025). The architecture-bias study uses only Rhesus macaque antibody heavy-chain variable region sequences, with no light-chain input and no constant-region input, while nevertheless drawing conclusions about target specificity, SHM, V-gene structure, and isotype-associated patterns (Mengren et al., 10 Dec 2025). This suggests that AntiBERTa’s observed success may partly reflect dataset-specific correlational structure.

Taken together, the literature supports a careful conclusion. AntiBERTa is a foundational antibody LLM whose downstream behavior is widely regarded as biologically meaningful, especially in its natural focus on CDRs and antibody-specific sequence correlates (Ruffolo et al., 2021, Mengren et al., 10 Dec 2025). It remains a credible and often strong baseline in antibody-sequence prediction tasks, but later studies also show it can be surpassed on particular benchmarks and that many published comparisons leave fairness, reproducibility, and causal interpretation incompletely resolved (Hossain et al., 17 May 2025, Mengren et al., 10 Dec 2025).