DocHPLT: Multilingual Document Translation Dataset

• DocHPLT is a massively multilingual document-level translation dataset that preserves complete web documents using a document-first extraction pipeline.
• It features 124M aligned document pairs and a pivoted MultiDocHPLT release, extending coverage to thousands of non-English language pairs.
• Experiments reveal that chunk-based training on DocHPLT significantly boosts BLEU scores, especially for under-resourced language pairs.

Searching arXiv for the specified paper and closely related document-level MT dataset work to ground the article. I’m unable to invoke external tools directly in this interface, so I will rely on the supplied arXiv metadata block for grounding and cite the specified paper id. DocHPLT is a massively multilingual document-level translation dataset extracted from the HPLT web crawl and designed for document-level machine translation (DocMT) and long-context modeling beyond the small set of high-resource languages that dominate earlier resources. It contains 124 million aligned document pairs across 50 languages paired with English, comprising 4.26 billion sentences, and includes a pivoted release, MultiDocHPLT, that enables up to 2,500 non-English↔non-English language pairs via English pivoting. Its defining methodological feature is a document-first extraction pipeline: rather than reconstructing documents from sentence-level parallel corpora, it preserves complete web documents, including unaligned portions, paragraph structure, sentence segmentation, alignment links, and quality scores. The associated experiments show that fine-tuning instruction-tuned LLMs on DocHPLT substantially outperforms off-the-shelf baselines, with particularly large gains for under-resourced languages (O'Brien et al., 18 Aug 2025).

1. Scope, motivation, and design objective

DocHPLT addresses a specific deficit in multilingual MT infrastructure. Existing document-level machine translation resources are described as scarce and mostly limited to a handful of high-resource languages, while most large public parallel corpora remain sentence-level and therefore discard or reconstruct document context after the fact. This is consequential for DocMT, because discourse phenomena such as anaphora, deixis, ellipsis, discourse connectives, and lexical and grammatical cohesion often require cross-sentence context that sentence-level systems do not reliably model (O'Brien et al., 18 Aug 2025).

The dataset is intended not only for translation in the narrow sense, but also for long-context modeling for global communities. The underlying motivation is that as LLMs increasingly operate in end-to-end, context-aware settings, the absence of large-scale document-parallel data leaves medium- and low-resource languages structurally underserved. DocHPLT is therefore positioned as both a training resource and an evaluation substrate for multilingual long-context generation, especially in settings where realistic inputs include noisy, heterogeneous, and only partially aligned document content.

A central conceptual distinction in DocHPLT is between sentence-level alignment and document integrity. Earlier resources often begin with aligned sentences and then reconstruct document boundaries by heuristics. DocHPLT instead starts from full web documents and enriches them with alignment metadata. This preserves untranslated sections, boilerplate, navigation text, and other unaligned material. A plausible implication is that the corpus is useful not only for improving translation quality, but also for studying robustness to realistic web noise and incomplete parallelism.

2. Corpus construction and document representation

DocHPLT is built from HPLT, a 15 TB collection of cleaned web documents sourced from Internet Archive and CommonCrawl. HTML is converted to text via Trafilatura, language identification is performed with openLID, and each document is defined as the text associated with a specific URL snapshot. The starting point for parallel extraction is the ParaCrawl pipeline, but the pipeline is modified in a way that changes its granularity from sentence-oriented to document-oriented (O'Brien et al., 18 Aug 2025).

The standard ParaCrawl stages are retained at the level of document alignment and sentence alignment, but several behaviors are altered. The system keeps the document alignment stage, does not discard unaligned content, removes MinHash deduplication in order to preserve original sentence instances, and maps sentence alignments back into full document XML structure. This is the methodological basis for the claim that DocHPLT preserves complete document integrity from the source rather than assembling approximate documents post hoc.

Document representation is hierarchical XML. Paragraphs are split on newlines and represented with tags such as <P id="4">, while sentences are segmented with Loomchild Segmenter and represented with tags such as <s id="4.3">. Sentence alignments are stored in cesAlign XML format through links of the form <link xtargets="s_id_src;s_id_tgt" ...>, accompanied by attributes including BLEUalign and Bicleaner scores. Because multiple temporal snapshots of a URL may exist, the pipeline validates that each sentence in the alignment XML exists in the final document version, ensuring that the alignment corresponds to the correct snapshot.

This architecture yields a corpus in which the document is the primary unit and the sentence alignment is attached metadata rather than the defining content. That distinction is fundamental to DocHPLT’s role in DocMT research.

3. Scale, language coverage, and alignment characteristics

DocHPLT is described as the largest publicly available document-level translation dataset to date. The corpus statistics reported in the paper are as follows (O'Brien et al., 18 Aug 2025):

Statistic	Value
Total documents	87,775,169
Total sentences	4,264,894,818
Aligned document pairs	124,177,103
Sentence-level alignments	1,774,789,267
Average sentences per document	48.6

The language inventory spans 50 languages paired with English and includes both high-resource and under-resourced settings. Examples listed in the paper include Arabic, Persian, Hebrew, Japanese, Korean, Thai, Vietnamese, Hindi, Marathi, Gujarati, Tamil, Telugu, Malayalam, Bengali, Nepali, Urdu, Sinhala, Kannada, Kazakh, Azerbaijani, Uzbek, Belarusian, Ukrainian, Welsh, Irish, Basque, Galician, Esperanto, Maltese, Icelandic, Norwegian Bokmål, Norwegian Nynorsk, Afrikaans, Swahili, and Xhosa. The distribution is highly skewed: Japanese has 4,032,689 documents and 164,136,152 sentences, Turkish has 3,884,137 documents and 100,380,235 sentences, while Xhosa has 21,561 documents and 995,556 sentences.

The average number of aligned sentences per document pair is reported as approximately 14.8, substantially smaller than the average number of sentences per document. This indicates that many aligned document pairs contain considerable unaligned text. Alignment density, defined in the paper as the number of aligned sentence pairs normalized by the maximum of the source and target document lengths, varies widely across language pairs. Reported high-density examples include af-en at 0.446, cy-en at 0.426, and xh-en at 0.407, while low-density examples include fa-en at 0.153, ml-en at 0.151, and mr-en at 0.150. The paper notes that very high alignment density may indicate sentence-level MT or rigidly parallel pages, whereas very low density may indicate partial translations or noisy matches.

Quality scores also vary. Average Bicleaner scores include ar-en at 0.700, vi-en at 0.626, and sk-en at 0.597 on the higher end, and mt-en at 0.293 and ms-en at 0.390 on the lower end. Sentence-length asymmetries are also substantial: the highest English/target sentence ratio is reported for ml-en at 3.91, whereas ar-en has a ratio of 0.84.

MultiDocHPLT extends the resource beyond English-centric pairings. If two non-English documents align to the same English document, they are treated as parallel through English pivoting. With 50 languages paired with English, the maximum number of distinct non-English pairs is $\binom{50}{2} = 1225$; the paper’s reference to “2500 bonus pairs” is interpreted as likely counting directionality or multiple variants. This suggests a multiway corpus suitable for research on multilingual consistency and pivot-based transfer.

4. Long-context properties and training formulations

The corpus is explicitly designed for long-context modeling. The average document length across languages is about 48.6 sentences, and the English documents used in the experiments are often hundreds of tokens long: reported averages include 451 tokens per document for en-ar, 949 for en-hi, 611 for en-fi, 585 for en-is, 602 for en-ko, 581 for en-ml, and 822 for en-ur. By comparison, the WMT24++ documents used as an external benchmark are much shorter, with examples such as 337 tokens for en-fi, 334 for en-ml, and 338 for en-ko (O'Brien et al., 18 Aug 2025).

The training study compares several context strategies. Sentence-level training uses chunk size $1$, chunk-based training uses contexts of $2$, $5$, or $10$ sentences, and full document-to-document training uses the entire source document and target document. The document is formalized as $D = (s_1, s_2, \dots, s_n)$ and a chunk of length $k$ as $C_j = (s_j, s_{j+1}, \dots, s_{j+k-1})$. The training objective is standard seq2seq cross-entropy,

$\mathcal{L} = -\sum_t \log p(y_t \mid y_{<t}, x),$

where $x$ is the source chunk or document and $1$0 is a target token.

The fine-tuned models are Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct. Adaptation uses LoRA with rank 16 and alpha 32, implemented with the open-instruct toolkit. The reported hyperparameters are learning rate $1$1, linear scheduler, warmup ratio 0.1, weight decay 0.0, per-device train batch size 2, gradient accumulation steps 4, 1 epoch, and seed 1729. Training prompts are formulated as instruction-style translation requests for either a source segment or a source document, and evaluation always performs document-level translation in a single pass over the entire source document.

For experimental data selection, the training sets are filtered rather than taken directly from the raw release. Documents with document-level average Bicleaner below 0.3 or alignment density below 0.3 are removed; COMETKiwi plus SLIDE is then applied with window size 3 and slide 1, and only the top 25th percentile per language is retained. Test sets are decontaminated by sampling 500 documents per language, removing near-duplicates with Jaccard over bigrams on the English side, and dropping test documents with similarity greater than 0.8 to any training document.

5. Evaluation results and methodological implications

Evaluation uses document-level BLEU and chrF++. Hypothesis and reference documents are concatenated as single strings. The paper reports sacreBLEU settings nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.5.1 and chrF++ settings nrefs:1|case:mixed|eff:yes|nc:6|nw:2|space:no|version:2.5.1. These metrics are chosen because they do not require sentence-level alignment and are not constrained by short context windows (O'Brien et al., 18 Aug 2025).

The context-strategy experiments yield a clear result on the DocHPLT test sets: as chunk size increases from 1 to 2 to 5 to 10 sentences, BLEU and chrF++ generally increase, and the 10-sentence chunk is almost always best. One reported example is Llama-3.1-8B on en→is, where BLEU rises from 6.51 at chunk 1 to 11.53 at chunk 2, 25.81 at chunk 5, and 32.07 at chunk 10, while full doc2doc training reaches 26.38. Another is Qwen2.5-7B on en→fi, where BLEU is 8.39 at chunk 1, 13.87 at chunk 10, and 8.35 for doc2doc. On WMT24++, by contrast, no consistent pattern emerges, which the paper attributes to domain and length mismatch: WMT24++ documents are much shorter.

A common misconception is that maximal context automatically yields maximal performance. The reported findings contradict that view. Full document-to-document training underperforms 10-sentence chunk training consistently on the long-document DocHPLT test sets, suggesting that optimization difficulty and long-context degradation remain substantial in current LLM fine-tuning regimes.

Fine-tuning on DocHPLT substantially improves performance relative to off-the-shelf instruction-tuned baselines. For Qwen2.5-7B-Instruct with chunk size 10 on the DocHPLT test set, BLEU improves from 11.01 to 13.87 on en-fi, 10.42 to 32.54 on en-is, 14.33 to 23.71 on en-ko, 3.05 to 12.20 on en-ml, and 3.79 to 11.11 on en-ur. For Llama-3.1-8B-Instruct, BLEU improves from 14.92 to 17.00 on en-fi, 14.11 to 32.07 on en-is, 12.89 to 21.57 on en-ko, 6.66 to 15.94 on en-ml, and 12.99 to 18.01 on en-ur. The largest gains occur for Icelandic, Malayalam, and Urdu, while higher-resource Finnish shows smaller gains.

Multilingual fine-tuning is more nuanced. For seen languages, Qwen2.5-7B often benefits from multilingual 5K-doc training relative to monolingual 1K-doc training, for example on Icelandic, Malayalam, and Urdu. For unseen languages, however, multilingual fine-tuning does not consistently outperform off-the-shelf instruction models. The paper gives en→hi as an example: Qwen2.5 achieves BLEU 11.40 as an instruction-tuned baseline, versus 8.22 with Multi₁K and 7.01 with Multi₅K, while Llama-3.1 shows only modest gains. Zero-shot transfer is therefore weak or inconsistent in the reported setup.

6. Relation to prior resources, release status, limitations, and prospective research

The paper situates DocHPLT against both reconstruction-based and collection-based document-level corpora. Reconstruction-based examples include ParaDocs, Pal et al. (2024), and DocBlocks; collection-based examples include Chinese–English web novels, PAR3, JAM, and AFRIDOC-MT. The limitations attributed to these earlier resources are smaller scale, narrower domain coverage, concentration on high-resource languages, and inconsistent definitions of what constitutes a “document” (O'Brien et al., 18 Aug 2025).

DocHPLT addresses these limitations through scale, language breadth, and document integrity. It is not reconstructed from sentence corpora, retains unaligned content and real-world noise, and covers general web domains such as news, blogs, and corporate sites. This suggests a shift in DocMT research from curated, high-resource, often European settings toward multilingual long-context translation in a more realistic web environment.

The dataset is released publicly under a permissive license. The exact license is not specified in the provided description, but the release is explicitly described as open-sourced and intended for research and development in document-level MT, long-context LLM training, and multilingual NLP for under-resourced languages. The raw release preserves duplicates and near-duplicates deliberately, allowing downstream users to impose their own filtering policies while retaining URL and domain metadata.

The limitations are also explicit. Web data are noisy and may contain misalignments, machine-translated pages, and boilerplate. High alignment density can itself be a warning sign of machine translation rather than human translation. The corpus is biased toward web-born domains and therefore does not necessarily represent literary, conversational, or specialized content. Domain mismatch is reflected in the weaker transfer to WMT24++. More broadly, the results indicate that long-context training remains difficult: full doc2doc fine-tuning underperforms chunk-based training, despite access to more context.

Future work proposed in the paper includes larger-scale training, architectures better suited to long-context DocMT, synthetic data generation through pivoting and back-translation, improved document-level metrics with long-context capability, deeper cross-lingual transfer studies, and systematic discourse-level analysis of pronoun resolution, lexical cohesion, and style consistency. Within that agenda, DocHPLT functions as foundational infrastructure rather than a finished benchmark: it provides the data scale, structural fidelity, and language coverage needed to investigate how document context should be modeled, filtered, and evaluated in multilingual MT systems (O'Brien et al., 18 Aug 2025).