---
title: SCAND Corpus in The Nordic Pile
url: https://www.emergentmind.com/topics/scand-corpus-in-the-nordic-pile
type: topic
---

# SCAND Corpus in The Nordic Pile

The SCAND Corpus is a major subcomponent of The Nordic Pile, a curated 1.2TB multilingual dataset designed to facilitate pre-training of large language models (LLMs) for North Germanic languages. The SCAND Corpus, as referenced in the foundation literature, consists primarily of two key resources: (1) the web-crawled “LES – Nordic Web Data” and (2) the ENO (“Enevaeldens Nyheder Online”) historical Danish-Norwegian newspaper dataset. These resources collectively represent the largest targeted collection of large-scale Nordic web text and diachronic press corpora, with precise data cleaning, metadata structuring, and deduplication regimes supporting model training and cross-domain analysis [2303.17183][2509.02356].

## 1. Corpus Scope and Definition

The SCAND Corpus as used in The Nordic Pile comprises two principal collections:

- **LES – Nordic Web Data:** Web-crawled textual data in Danish, Icelandic, Norwegian, and Swedish, sourced and filtered from Common Crawl (via OSCAR, mC4, and a custom Trafilatura-based pipeline) [2303.17183].
- **ENO (Enevaeldens Nyheder Online):** Digitized historical Danish and Norwegian newspaper texts (1660–1849), extracted from microfilm using neural OCR and advanced post-processing [2509.02356].

Both datasets exclude English at the corpus assembly stage and are grouped under the SCAND (“Scandinavian”) section in The Nordic Pile’s folder hierarchy.

## 2. Data Acquisition and Preprocessing

### LES – Nordic Web Data

Crawling and extraction utilize Common Crawl slices filtered via OSCAR and mC4, augmented by a custom LES pipeline employing Trafilatura for more aggressive and language-specific extraction. All domains are publicly available, with exclusion of personal-data-heavy sites enforced at the URL level for compliance with GDPR and licensing standards.

Metadata attached to each document includes:
- Predicted language (lang) via fastText
- Numerical statistics (num_chars, num_utf8bytes, num_words, num_sents)
- 128-bit md5 hash of the document’s UTF-8 encoded text

### ENO Historical Newspapers

Source materials comprise:
- ~565,000 microfilm pages from 28 Danish/Norwegian newspaper titles; digitized in varying stages since the 1950s
- Danish content via the Royal Library’s Mediestream API; Norwegian via high-resolution digitization of three major urban titles

Preprocessing comprises image binarization, de-skewing, and denoising, followed by automated layout segmentation (Transkribus "Fields Model") to preserve column structure and reading order.

## 3. Cleaning, Filtering, and Deduplication Procedures

### LES – Nordic Web Data

The pipeline consists of seven ordered stages:
1. **Normalization:** Removal of control/non-printing characters, whitespace normalization, Unicode NFC.
2. **Computation of metrics.**
3. **Sixteen Boolean Quality Filters:** Mandatory for inclusion. Representative filters include:
   - ≥80% of words must contain alphabetic characters
   - Digit fraction <0.2
   - Mean word length in [2,10] characters
   - Stop-word coverage ≥2 stop-words and ≥10% of total words
   - Supported language: da, is, nb, sv
   - “Repetitive Gopher” controls for duplicate n-grams, lines, and paragraphs  
     (e.g., duplicate line/paragraph fraction ≤0.35, duplicate 2-gram character ≤0.25, duplicate 10-gram ≤0.15)
4. **Exact Deduplication:** Removal of documents with duplicate md5 hashes.
5. **Language Segmentation:** Partitioning by predicted language for subsequent fuzzy deduplication.
6. **Fuzzy Deduplication:** Uses MinHash LSH approach—10-character shingle sets, 10-integer signature, b=2 bands. Threshold for deduplication is Jaccard similarity sim(Ci,Cj) ≥ 0.5.
7. **Merge and assembly.**

This workflow removed approximately 20% of the original web-crawled data, chief among them those failing stop-word or repetitive pattern thresholds [2303.17183].

### ENO – Historical Press Corpus

OCR is performed using a Transkribus-based architecture:
- Four convolutional "feature extraction" blocks
- Two bi-directional LSTM layers
- CTC loss for alignment ($L_{CTC} = -\log p(y|x)$)
- Character set: Fraktur (historic) plus Antikva fallback

Post–OCR word accuracy per text ($PWA(T)$) is computed as the fraction of words found in a union dictionary (literary corpora, census names, frequent tokens). Low-PWA texts and language-shift (e.g., German passages) are flagged but retained with explicit "pwa" metadata field.

Standalone articles are derived using a two-stage, line-level segmentation:
1. Random Forest based on structural features
2. BERT-augmented Random Forest classifier, using contextual embeddings from a DA-BERT_Old_News_V1 model pretrained on 260M words from ENO

Performance is validated at F₁ = 98.9% for line segmentation on held-out 7,000 lines. Key parameters, including batch sizes, max sequence lengths, and learning rates, are specified in [2509.02356].

## 4. Corpus Composition and Statistics

### Aggregate Composition

| Category                | Documents (M) | Size (GB) | Mean doc size (KB) |
|-------------------------|--------------|-----------|--------------------|
| LES – Nordic Web Data   | 79.16        | 76.83     | 0.97               |

SCAND (LES) constitutes 38.18% of the 1,208.7GB Nordic Pile aggregate.

#### Language Breakdown (All Categories; Not Exclusive to LES)

- Danish: 10.8% (~130.5GB total; Web CC subset 111.33GB)
- Icelandic: 1.59% (~19.2GB total; Web CC subset 8.79GB)
- Norwegian: 11.55% (~139.6GB total; Web CC subset 90.00GB)
- Swedish: 26.02% (~314.5GB total; Web CC subset 188.94GB)

Exact per-language document counts within LES are not specified; the only available aggregate for historical newspapers (ENO) is ≈474 million word tokens [2509.02356].

## 5. Metadata, Licensing, and Integration

### Metadata Practices and Schema

For ENO, each standalone text is associated with:

```json
{
  "text_id": "KobenhAv_17850315_00023",
  "publication": "Københavns Adresseavis",
  "date": "1785-03-15",
  "page_id": "KobenhAv_17850315_p005",
  "text_segment": 23,
  "pwa": 0.96,
  "lang": "da",
  "ocr_model": "ENO-Fraktur-CTC-v1",
  "bert_seg_model": "ENO-SetFit-Seg-v0.1"
}
```

Recommended organization for the SCAND section is:

```
scand/
  eno/
    metadata.jsonl
    texts/{publication_name}/{YYYY-MM}/{publication}_{date}_{text_id}.txt
```

All ENO data and models are licensed CC BY-SA 4.0. Web data are derived from public domain sources and filtered for legal compliance [2509.02356][2303.17183].

## 6. Intended Use and Interoperability

SCAND (LES – Nordic Web Data plus ENO) is one of nine major categories in The Nordic Pile, responsible for 38% of the corpus by volume and providing a critical backbone of general-domain, web-sourced, and specialized historical data for Danish, Icelandic, Norwegian, and Swedish. Its intended use is for pre-training large language models (multi-billion parameter scale) to achieve both broad and Nordic-specific linguistic competence.

Alignment fields across datasets (e.g., "publication", "date", "lang", "text_id", "pwa") support diachronic, cross-lingual, and region-wide studies; language models and BERT embeddings are recommended for shared representation. A plausible implication is that the SCAND Corpus design fosters reproducibility and comparability across complementary corpora in the Nordic Pile, and supports the layered cleaning/filtering regimes critical for high-quality model pre-training [2303.17183][2509.02356].

Source: https://www.emergentmind.com/topics/scand-corpus-in-the-nordic-pile