KL3M: Knowledge, Law & Licensing for LLMs

• KL3M is a comprehensive protocol and tokenizer suite designed for legally compliant pretraining of LLMs in specialized domains.
• It aggregates over 132 million documents with stringent provenance checks to ensure copyright-clear and auditable content from government, legal, and financial sources.
• KL3M tokenizers empirically reduce token counts by up to 10-20%, improving cost efficiency and enabling long-context modeling in professional applications.

KL3M (Knowledge, Law, and Licensing for LLMs) refers to a data project and associated tokenizer family specifically engineered to support the development of LLMs using copyright-clean, auditable, and domain-optimized resources. KL3M addresses both the legal/ethical deficiencies in prevailing LLM pretraining corpora and the inefficiencies of generic tokenization when applied to legal, financial, and governmental text. Originated by the ALEA Institute, KL3M encompasses (1) a rigorously constructed corpus of over 132 million documents cleared under strict legal criteria, (2) a data engineering pipeline guaranteeing provenance and compliance at every stage, and (3) a suite of tokenizers that empirically outperform mainstream alternatives on professional-domain text (II et al., 10 Apr 2025, Bommarito et al., 21 Mar 2025).

1. Project Motivation and Conceptual Framework

The KL3M Data Project was established in response to the problematic legal status of most existing LLM pretraining datasets. Prior practices involved ingestion of copyrighted texts in violation or ambiguity of consent, leading to direct legal risk (e.g., high-profile litigation such as Thomson Reuters v. ROSS Intelligence Inc.). The project’s rationale is twofold:

• Ethical rectitude: pretraining solely on public-domain or permissively licensed sources, thereby honoring creator rights and avoiding uncompensated appropriation.
• Operational certainty: enabling downstream model builders, commercial users, and researchers to eliminate copyright and contractual ambiguity from pipeline design, model deployment, and model-derived outputs.

KL3M operationalizes this via the “KL3M Data Protocol”: a three-part test that, for every document, adjudicates its eligibility for inclusion based on (1) free-at-creation status (e.g., U.S. government works), (2) public-domain determination (expiration, explicit CC0 dedication), and (3) license terms permitting unencumbered reuse (exclusion of NC, ND, SA provisions, and requirement for scalable attribution if CC-BY).

2. Corpus Composition and Legal Compliance

KL3M’s corpus, as of April 2025, aggregates 16 source categories, yielding $132.3 \times 10^6$ documents and approximately $1.348 \times 10^{12}$ tokens. Core sources include SEC EDGAR filings (975.3 billion tokens), Court Listener (16.7 billion tokens), .gov websites, the EU Official Journal, legislative and regulatory texts, patent office datasets, and RECAP court archives. Each document is tracked in its original format and maintained with cryptographic hashing, full source URIs, explicit license fields, and granular metadata to ensure provenance (see Table 6 in (II et al., 10 Apr 2025)).

Every document undergoes the KL3M Data Protocol. This three-test sequence is defined as:

1. Test 1: Is the work uncopyrightable from creation (e.g., “works of the U.S. government or edicts of law”)?
2. Test 2: If not, has it definitively entered the public domain?
3. Test 3: If still under copyright, does its license grant unencumbered rights (disallowing CC-NC, CC-ND, or CC-SA; allowing only if machine learning is permitted and attribution is feasible)?

Documents only included if an unequivocal “include” is returned; ambiguous or encumbered materials are excluded. For Creative Commons, only CC0 and, if scalable attribution is possible, CC-BY are accepted.

3. Data Engineering Pipeline and Quality Assurance

The KL3M pipeline consists of three principal stages:

• Stage 1: Acquisition and archival of original documents (base64-encoded, compressed) with all source and legal metadata.
• Stage 2: Extraction of textual representations (text, Markdown, HTML) with standardized metadata and pre-tokenization via the KL3M BPE tokenizer (“alea-institute/kl3m-004-128k-cased” as default).
• Stage 3: Creation of training-optimized data, represented as Parquet files containing columnar token arrays, indexed and ready for machine learning frameworks.

Provenance is maintained throughout via persistent references between stages. Quality filtering incorporates both a composite document-quality score (structural, character-level, and token-level metrics) and an $L^2$-norm filter applied to token-frequency vectors: $\|f_d - f_{\mathrm{ctrl}}\|_2 < \tau$ where $f_d$ is the document frequency profile and $f_{\mathrm{ctrl}}$ is the control vector derived from curated legal corpora. This ensures lexical and stylistic integrity and mitigates downstream contamination from data artifacts.

4. KL3M Tokenizer Suite: Design and Empirical Performance

KL3M tokenizers are grouped into two main families:

• Domain-specific BPE tokenizers: “kl3m-003-64k (cased),” “kl3m-004-128k-cased,” “kl3m-004-128k-uncased.” These are trained on copyright-free legal, financial, and government corpora, with large vocabulary sizes (64 K, 128 K) and extensive custom token integration (domain-specific citations, Markdown, numeric ranges). Cased versions preserve distinctions for professional terms; uncased variants prioritize embedding efficiency.
• Character-level BPE tokenizers: “kl3m-004-char-4k-cased,” “kl3m-004-char-8k-cased,” “kl3m-004-char-16k-cased.” These enforce short maximal merge lengths, stabilizing token boundaries for OCR post-processing and correction applications. This aids model learning of text corrections by minimizing token segmentation drift.

Tokenization Efficacy

KL3M’s primary domain-specific tokenizer (kl3m-004-128k-cased) outperforms mainstream alternatives on key professional datasets, reducing token counts as follows (Bommarito et al., 21 Mar 2025):

Dataset	kl3m-004-128k-cased	GPT-4o	LLaMA-3
Congressional Hearings	0.2292	0.2482	0.2475
Court Documents	0.2741	0.2971	0.2986
US Code	0.3181	0.3716	0.3717
SEC Filings	0.1816	0.1976	0.1992
General Content	0.2057	0.2033	0.2066

Average tokens per character (TPC) are 9% lower than GPT-4o and 8.7% lower than LLaMA-3. This produces a direct cost reduction and increased effective context window (~10–20% more characters per fixed context).

Specialized terminology is substantially compressed: up to 83% fewer tokens for legal terms and 39% for financial terms compared to LLaMA-3. Custom tokens cover citation schemes, block-level document structure (Markdown, HTML elements), numeric expressions, and enumerations.

Character-level BPE variants address error-correction by constraining token merges so that both erroneous and corrected text share stable segmentation patterns, improving model alignment and error correction convergence.

5. Released Artifacts, Public Access, and Licensing

KL3M’s assets are released under CC-BY 4.0 for all data and MIT for all code. The range of downloadable and interoperable components includes:

1. Full source code for document acquisition, processing, and standardized metadata.
2. Original files and provenance metadata (JSON schema with source, licensing, hash, file size, and document-specific fields).
3. Extracted and normalized text content (text, Markdown, HTML, and JSON).
4. Pre-tokenized representations using the KL3M tokenizers.
5. Mid- and post-train benchmarks for supervised and generative tasks (QA, summarization, drafting, classification).
6. Domain samples (e.g., 500 K enterprise office/PDF files for benchmarking).
7. Structured databases (e.g., .gov document links).
8. Interactive tools such as the Data Gallery.

Endpoints include S3 buckets, the “alea-institute” Hugging Face organization, and public GitHub repositories for full transparency and reproducibility.

6. Primary Applications and Empirical Impact

KL3M is primarily intended for pretraining and fine-tuning of small- to medium-scale LLMs in domains where legal, regulatory, or high-fidelity government data are crucial. The dataset supports training of models (e.g., kl3m-002-170m, kl3m-003-1.7b) that exhibit competitive performance using only KL3M and minimal supplementary data (II et al., 10 Apr 2025). Instruction tuning tasks span legal Q&A, contract drafting, statutory summarization, and regulatory analytics.

A notable feature is the corpus’s suitability for long-context modeling: 17.5% of documents exceed 8 K tokens and 0.5% exceed 100 K tokens, supporting transformer architectures with extended input limits.

The project empirically reduces legal risk by grounding every token in a documented, statutory, or license-cleared provenance. The reliance on fact-checked, expert-reviewed government documents also yields a higher mean token-level entropy (7–8 bits) than web scrapings, supporting improved downstream factuality.

7. Limitations and Future Prospects

KL3M’s domain-specificity can reduce efficiency on corpora outside its primary legal, financial, and governmental focus (e.g., biomedical or conversational datasets). Highly emergent or outside-vocabulary terms will be decomposed into smaller subwords due to their absence from the initial training corpus.

Integration into workflows designed for standard tokenizers requires reinitialization or further model adaptation. As of the latest release, large-scale evaluation on generic downstream tasks remains incomplete.

Planned future work includes: (1) domain-adaptive tokenizer expansion, (2) downstream task evaluation (legal QA, financial sentiment), (3) multilingual support (e.g., EU law in German/French), (4) dynamic/incremental BPE to accommodate emergent terminology, and (5) advanced token adapter layers for seamless model interoperability.

All resources, evaluation suites, and experimental frameworks are available for public use under robust open licenses, facilitating both academic research and application development without risk of copyright or contractual liability (II et al., 10 Apr 2025, Bommarito et al., 21 Mar 2025).