---
title: 'LEGAL-BERT: Legal NLP Transformers'
url: https://www.emergentmind.com/topics/legal-bert
type: topic
---

# LEGAL-BERT: Legal NLP Transformers

LEGAL-BERT is a family of BERT-based transformer models specifically designed for natural language processing (NLP) tasks in the legal domain. These models are trained or further pre-trained on large corpora of legal texts to capture the sublanguage, terminology, and stylistic conventions distinctive to statutes, case law, contracts, and judicial opinions. LEGAL-BERT provides a foundation for a variety of downstream applications, including legal document classification, named entity recognition (NER), and the automated analysis of judicial reasoning. Multiple research efforts have expanded upon and evaluated LEGAL-BERT models, establishing their empirical superiority over standard pre-trained architectures in legal NLP benchmarks [2010.02559][2310.18440][2103.11792].

## 1. Corpus Construction and Domain Adaptation

LEGAL-BERT models rely on legal-domain corpora that cover diverse jurisdictions and document types. The canonical LEGAL-BERT corpus comprises approximately 12 GB of English legal text (~355,000 documents) drawn from:

- EU legislation (EUR-Lex): 61,826 documents, 1.9 GB  
- UK legislation (legislation.gov.uk): 19,867 documents, 1.4 GB  
- ECJ cases: 19,867 documents, 0.6 GB  
- ECHR cases: 12,554 documents, 0.5 GB  
- US court cases: 164,141 documents, 3.2 GB  
- US contracts (SEC-EDGAR): 76,366 documents, 3.9 GB  

Legal corpora are often minimally pre-processed. For specialized tasks such as statutory-interpretation classification, filtering by keyword co-occurrence and up-sampling of paragraphs containing domain-typical terminology (e.g., “plain meaning,” “legislative history”) is used to target conceptually salient content [2310.18440][2010.02559].

## 2. Pre-training Strategies and Model Variants

Three principal domain adaptation strategies for BERT in LEGAL-BERT research are:

1. **Out-of-the-box BERT**: Direct fine-tuning of the original BERT model on legal tasks without domain adaptation.
2. **Further pre-training (FP)**: Continued pre-training of BERT-base on legal data, preserving the original architecture and vocabulary, but updating contextual knowledge for the legal sublanguage.
3. **Pre-training from scratch (SC)**: Initialization of a new BERT-base model trained exclusively on legal texts, often with a dedicated vocabulary learned via SentencePiece. For LEGAL-BERT-SC, the vocabulary size is 30,000 subwords, and the base architecture comprises 12 layers, 768 hidden size, 12 attention heads, totaling ~110 million parameters [2010.02559].

The standard pre-training objectives—Masked Language Modeling (MLM) and Next Sentence Prediction (NSP)—are retained:

$$
L_{MLM} = -\sum_{i \in M} \log P(x_i \mid x_{\setminus M}), \qquad
L_{NSP} = -[I_{IsNext} \log P(IsNext \mid A,B) + I_{NotNext} \log P(NotNext \mid A,B)],
$$

where $M$ denotes masked token positions, and $(A,B)$ is a candidate sentence pair.

## 3. Fine-tuning, Hyperparameters, and Vocabulary Integration

LEGAL-BERT fine-tuning departs from the “Devlin et al.” recipe by expanding the hyperparameter grid:

- Learning rate: $1\times10^{-5}$ to $5\times10^{-5}$
- Batch size: 4, 8, 16, 32
- Dropout: 0.1 or 0.2 (classification head sometimes up to 0.3)
- Early stopping on validation loss, no fixed epoch cap

Task-specific output heads are attached directly to the [CLS] token for classification tasks or token-level representations for NER (linear+CRF) [2010.02559][2103.11792].

Additional adaptations include **domain vocabulary injection**: expanding the WordPiece vocabulary with frequently observed legal terms (e.g., adding 555 new terms) and randomly initializing associated embeddings. This modification improves representation of multi-word legal phrases and, when coupled with domain pre-training, produces synergistic gains [2103.11792].

## 4. Empirical Performance on Legal NLP Tasks

LEGAL-BERT is consistently evaluated against general-purpose BERT, further domain-pretrained variants, and alternative architectures (e.g., XLNet, T5):

- **Legal Reasoning Classification**: On the three-way FORMAL/GRAND/NONE task for paragraph-level statutory-interpretation reasoning (U.S. Supreme Court), LEGAL-BERT-SC achieves macro-F1 = 0.70 (precision = 0.70, recall = 0.71), outperforming BERT-base (macro-F1 = 0.68), DistilBERT (0.67), T5-base (0.64), and all prompt-only LMs (GPT-4 with codebook prompt, macro-F1 = 0.22; few-shot GPT-4, 0.45; FLAN-T5 ≈0.19; Llama-2-chat ≈0.20) [2310.18440].

- **Legal Opinions Classification**: On majority-dissent classification, Legal-Vocab-BERT attains the best test accuracy (95.28%, +1.13 percentage points over BERT-Base). Domain-specific pretraining (Legal-BERT) yields a +0.93 point gain, while vocabulary augmentation alone yields +0.20 points. Combined, these techniques are mildly synergistic. XLNet achieves only +0.06 over BERT-Base and is 50% slower [2103.11792].

- **Legal NER and Multi-label Classification**: LEGAL-BERT-SC matches or exceeds all other variants, with observed micro-F1 improvements across EURLEX57K multi-label classification (+0.2%), ECHR binary (+0.8%) and multi-label (+2.5%) classification, and Contracts-NER (+1.1–1.8% across entity types) [2010.02559].

## 5. Specialized Applications: Judicial Reasoning and Legal Periodization

LEGAL-BERT-SC, fine-tuned on expert-annotated Supreme Court data, enables quantitative jurisprudence analysis:

- A dataset of 15,860 opinions (1870–2014) is processed to derive annual formality scores—the fraction of interpretive paragraphs labeled FORMAL or GRAND.
- Historical transitions captured by LEGAL-BERT-SC mirror canonical legal-historical periodizations:
  - 1870–1910: Prevalence of formalism
  - 1910–1937: Formalism declines
  - 1937–1980: Grand style dominance
  - Post-1980: Resurgence of formalism

These results align with existing qualitative accounts but afford more granular, data-driven assessments of doctrinal evolution. The paragraph-centric approach imposes constraints, as broader legal context beyond isolated text fragments is not modeled [2310.18440].

## 6. Limitations and Future Directions

Several limitations are identified:

- Inter-annotator agreement is moderate (Krippendorff’s $\alpha \approx 0.63$) even among domain experts, underscoring the inherent subjectivity of jurisprudential classification tasks.
- Current taxonomy (three labels) restricts granularity; future efforts could expand interpretive categories or integrate discourse-level features.
- Prompt engineering for generative LMs yields poor performance compared to fine-tuned LEGAL-BERT, indicating the necessity of annotation-intensive methods for highly specialized tasks.

Future directions include semi-supervised learning with larger silver-standard datasets, integration of cross-paragraph or document-level features, and development of richer legal label taxonomies [2310.18440].

## 7. Comparative Table: LEGAL-BERT Variants and Performance

| Model Variant          | Domain Adaptation Method           | Test Accuracy / Macro-F1           |
|-----------------------|------------------------------------|------------------------------------|
| BERT-Base-Cased       | General-domain, no further adapt.  | 94.15% (Legal Opinion Task) / 0.68 (Statutory Interpretation) |
| Legal-BERT            | Domain-specific pre-training       | 95.08% (+0.93 vs BERT-Base)        |
| Vocab-BERT            | Legal vocabulary injection         | 94.35% (+0.20)                     |
| Legal-Vocab-BERT      | Pre-training + vocab injection     | 95.28% (+1.13)                     |
| LEGAL-BERT-SC         | Pre-trained from scratch, +legal vocab (for some tasks) | macro-F1 = 0.70 (Legal Reasoning)  |
| XLNet-Base-Cased      | General-domain, autoregressive     | 94.21% (+0.06 vs BERT-Base)        |
| T5-base               | General-domain seq2seq             | Macro-F1 = 0.64                    |
| GPT-4 (prompt)        | Prompted LLM, no fine-tuning       | Macro-F1 = 0.22–0.45               |

LEGAL-BERT and its variants establish state-of-the-art benchmarks for multiple legal NLP tasks, demonstrating the necessity and efficacy of in-domain pre-training and legal vocabulary adaptation in the pursuit of robust, high-fidelity language understanding in the legal field [2010.02559][2103.11792][2310.18440].

Source: https://www.emergentmind.com/topics/legal-bert