HUKUKBERT: Domain-Specific Language Model for Turkish Law

Abstract: Recent advances in NLP have increasingly enabled LegalTech applications, yet existing studies specific to Turkish law have still been limited due to the scarcity of domain-specific data and models. Although extensive models like LEGAL-BERT have been developed for English legal texts, the Turkish legal domain lacks a domain-specific high-volume counterpart. In this paper, we introduce HukukBERT, the most comprehensive legal LLM for Turkish, trained on a 18 GB cleaned legal corpus using a hybrid Domain-Adaptive Pre-Training (DAPT) methodology integrating Whole-Word Masking, Token Span Masking, Word Span Masking, and targeted Keyword Masking. We systematically compared our 48K WordPiece tokenizer and DAPT approach against general-purpose and existing domain-specific Turkish models. Evaluated on a novel Legal Cloze Test benchmark -- a masked legal term prediction task designed for Turkish court decisions -- HukukBERT achieves state-of-the-art performance with 84.40\% Top-1 accuracy, substantially outperforming existing models. Furthermore, we evaluated HukukBERT in the downstream task of structural segmentation of official Turkish court decisions, where it achieves a 92.8\% document pass rate, establishing a new state-of-the-art. We release HukukBERT to support future research in Turkish legal NLP tasks, including recognition of named entities, prediction of judgment, and classification of legal documents.

• The paper introduces HukukBERT, a domain-specific language model that achieves 84.40% Top-1 accuracy on legal cloze tests and superior document segmentation.
• The paper employs a hybrid pre-training approach with a custom 48K WordPiece tokenizer, significantly reducing subword fragmentation compared to prior models.
• The paper demonstrates its practical impact on Turkish LegalTech by enabling accurate legal term prediction, robust segmentation of court decisions, and groundwork for advanced legal applications.

HukukBERT: Technical Summary and Implications for Turkish Legal NLP

Introduction and Motivation

HukukBERT addresses a critical gap in Turkish legal NLP: the lack of a large-scale, domain-specific pre-trained LLM. While domain-adaptive pre-training (DAPT) has yielded improvement in specialized English settings (e.g., LEGAL-BERT), the Turkish legal domain has suffered from data scarcity, absence of domain-specific tokenizers, and severe performance degradation of general-purpose Turkish LMs when applied to legal texts. General models fail due to lexical/semantic divergence (semantic shift), tokenization fragmentation, and inability to model the highly formal, syntactically complex structure of Turkish court decisions. The paper’s explicit premise is that legal text properties in Turkish, combined with morphological complexity, are not adequately captured by scaling data alone—targeted domain adaptation is required.

Corpus Development and Tokenization

The pre-training of HukukBERT is backed by an 18.93 GB balanced, deduplicated legal corpus, curated from over 2.3 million documents. Large-scale deduplication based on MinHash LSH eliminates repetitive templates and redundant appellate boilerplate, while explicit sub-domain balancing mitigates overrepresentation (notably, Yargıtay decisions). The corpus synthesizes case law from numerous judicial bodies, statutory texts, and academic legal literature, establishing full coverage of the Turkish legal linguistic spectrum.

A 48K WordPiece tokenizer is trained from scratch, seeded with Ministry of Justice legal dictionary entries to maximize OOV legal term inclusion. Comparative evaluation demonstrates that this tokenizer reduces mean subword fragmentation per line to 4.82, outperforming other Turkish tokenizers (TabiBERT: 7.38, BERTurk-Legal: 5.67). Reduced fragmentation results in enhanced semantic coherence, context retention, and more compact sequence representations for legal idioms and compounds. This is a measurable improvement over larger-vocabulary—but generic—tokenizers, which suffer from severe loss of coherence in legal phrase parsing.

Pre-Training and Domain Adaptation Methodology

HukukBERT utilizes a hybrid DAPT scheme over the legal corpus, with weights from a general model transferred for overlapping vocabulary—with mean initialization for novel tokens (23.3%). Masked language modeling (MLM) is executed using a composite masking pipeline: Whole Word Masking (20%), Token Span Masking (20%), Word Span Masking (30%), and targeted Keyword Masking (30%) over a curated list of 40,000+ legal tokens. This design compels the model to internalize conceptual legal units, reduces overfitting to grammatical artifacts, and trains the model to resolve semantic shift (e.g., predicting "tereke" rather than general "miras" in inheritance contexts). Training is conducted on high-end GPUs with batch size and learning rate schedules optimized for rapid convergence.

Evaluation: Legal Cloze Test and Structural Segmentation

Legal Cloze Test

A novel, intrinsic evaluation protocol—the Hukuki Cloze Test—comprises 750 queries across diverse sub-domains, targeting the model's ability to predict masked legal terms in context. HukukBERT achieves 84.40% Top-1 accuracy (95% CI: [81.63%, 86.82%]) and 98.80% Top-3 accuracy, a performance lead of +8.93 points over the strongest competitor (BERTurk-Legal, 75.47% Top-1). General-domain large-scale models such as TabiBERT (68.13%) and BERTurk-cased (63.7%) lag substantially, confirming that domain size and vocabulary scale do not compensate for lack of domain adaptation.

Qualitative error analysis shows that HukukBERT reliably resolves cases of semantic shift and legal procedural knowledge, consistently predicting legally correct terminology where general models default to colloquial or semantically imprecise alternatives.

Structural Segmentation of Court Decisions

For the practical downstream task of court decision segmentation (mapped as a BIO-based sequence labeling problem), HukukBERT achieves a stringent document-level pass rate of 92.8%, compared with 84.3% (general BERTurk-128k) and 81.9% (BERTurk-Legal), and yields boundary accuracy of 99.0%. Span-level analysis shows high F1 values, particularly on procedural segments (formal, ruling, footer). The persistent bottleneck exists for the header (F1: 68.6%) and dissent segments, primarily due to precision deficit and training data sparsity, respectively. Overfitting is mitigated by strict selection based on downstream metrics rather than validation loss.

Discussion and Implications

The paper establishes that for agglutinative, morphologically complex languages like Turkish, specialized pre-training and tokenizer adaptation are essential for high-fidelity legal NLP. Vocabulary size alone is not predictive of downstream performance; semantic alignment and deep coverage of domain-specific morphemes and legal phraseology are critical factors.

By releasing the pre-trained model, tokenizer, and Cloze Test benchmark, the work substantially lowers the barrier for research and development in Turkish LegalTech. HukukBERT’s embedding space, by virtue of explicit legal vocabulary and deep context modeling, provides a robust foundation for tasks such as document segmentation, legal named entity recognition, judgment prediction, and advanced retrieval.

The paper further suggests future directions: extending to architectures capable of encoding longer sequences (ModernBERT, FlashAttention, RoPE), exploring late-interaction models (ColBERT) for information retrieval, and establishing new fine-tuned baselines on currently unevaluated extrinsic legal tasks. The work recognizes the encoder’s inherent inability for generative modeling (i.e., non-applicability to contract drafting or summary generation), and the limitations imposed by the 512-token window.

Conclusion

HukukBERT represents a new state-of-the-art for Turkish legal language modeling, conclusively demonstrating that general-domain scaling is insufficient for domain-specific NLP in Turkish law (2604.04790). Through its balanced and deduplicated corpus, custom tokenization, and hybrid domain-adaptive pre-training, the model achieves substantial performance margins over prior approaches, successfully capturing legal concepts, resolving semantic drift, and segmenting complex judicial texts. This resource establishes a crucial building block for future research in Turkish computational law, facilitating rapid development and deployment of advanced LegalTech infrastructure.

---