Document Encoding Biases Overview

• Document encoding biases are systematic effects in document representations, spanning both unintended regularities (e.g., first-position effects) and deliberate structural inductive biases.
• They manifest as distortions in embedding stability, where alterations like beginning insertions can reduce cosine similarity by up to 12.3%, highlighting challenges in evaluation.
• Deliberate biases, such as hierarchy-aware attention and document-aware positional encoding, are leveraged to better encode document structure and improve retrieval fairness.

Document encoding biases are systematic effects introduced when documents are represented for machine processing. In recent work, the phrase has at least two distinct technical uses. In one use, it denotes undesirable regularities embedded in document representations, such as first-position preference in text embeddings, source or language confounding in similarity space, and low-resource-script underperformance in multilingual document understanding (Goel et al., 2024, Fan et al., 1 Jul 2025, Heakl et al., 12 May 2026). In another use, it denotes deliberate inductive biases that encode document structure into models, such as hierarchy-aware attention, document-aware positional encoding, and record-structured decoders for graphs, sets, or sequences (Cao et al., 2022, Ma et al., 2022, Meyer et al., 11 Jul 2025). The literature therefore spans both diagnostic and design-oriented interpretations of how documents are encoded.

1. Conceptual scope and terminological distinctions

A central distinction in recent retrieval work separates query-document interaction biases from document encoding biases. Query-document interaction biases arise from how queries and documents are compared and are more amenable to query-side interventions; literal matching is the canonical example. Document encoding biases, by contrast, “appear embedded in document representations and persist regardless of query formulation,” with position bias named as the clearest case (Goyal et al., 7 Apr 2026). This usage treats bias as a property of the learned geometry of document vectors rather than of any single query.

A different strand of work uses “bias” in the classical ML sense of an inductive bias. HIBRIDS injects hierarchical biases into attention scores through a document tree; HDT hard-codes a sparse hierarchy with [DOC], [SEC], and [SENT] anchors; and document-aware positional encoding for multi-document summarization adds a document-level signal to token positions (Cao et al., 2022, He et al., 2024, Ma et al., 2022). In this literature, the bias is desirable because it aligns computation with document structure.

A further terminological caution comes from work on bias in documents themselves. “It’s All Relative” studies inappropriate subjectivity expressed through document text and learns an interpretable scalar bias score from pairwise comparisons of Wikipedia revisions, but it is not a fairness analysis of embedding models or retrievers (Suresh et al., 2023). Doc-BiasO, in turn, uses “document” in the sense of machine-readable documentation of bias findings rather than document-vector behavior (Russo et al., 2024). The topic therefore includes representational skews in encoders, structural priors in architectures, and formal encodings of bias knowledge, but these are not interchangeable senses.

2. Positional asymmetry in text embedding models

The most explicit recent account of document encoding bias in text embeddings is the first-position effect quantified in “Quantifying Positional Biases in Text Embedding Models” (Goel et al., 2024). The paper studies a systematic tendency for document embeddings to depend more on content near the beginning of the input than on equally informative content later. It evaluates BGE-m3, E5-Large-V2, Nomic-Embed-Text-v1.5, E5-RoPE-base, Jina-Embeddings-v2-Base, Mosaic-Bert-Base, and Cohere Embed-English-v3.0 on PubMed publication abstracts, Paul Graham essays, Amazon Reviews, ArguAna argumentative texts, and Reddit posts. For inputs longer than a model’s context limit, examples are truncated from the end.

Two perturbation families are used. The first inserts semantically irrelevant Lorem Ipsum text at the beginning, middle, or end, with needle lengths of 5%, 10%, 25%, 50%, and 100% of the original token count. The second removes 10%, 25%, or 50% of document sentences from the beginning, middle, or end. Stability is measured by cosine similarity between original and perturbed embeddings:

$$\cos(\mathbf{x}, \mathbf{y}) = \frac{\mathbf{x} \cdot \mathbf{y}}{\|\mathbf{x}\|\|\mathbf{y}\|}$$

The aggregate finding is a consistent early-position bias: insertion of irrelevant text or removal at the start of a document reduces cosine similarity “by up to 12.3\% more than ablations at the end,” and beginning insertion reduces cosine similarity “by up to 8.5\% more than when inserted in the middle, and 12.3\% more than when inserted at the end” (Goel et al., 2024).

The paper reports finer-grained examples for a 20% insertion. For APE models, average cosine similarity is $0.885$ for beginning insertion versus $0.963$ for end insertion, a relative decrease of about 8%. For RoPE-based models, the corresponding values are $0.819$ and $0.968$, described as a 15.4% decrease. For ALiBi models, beginning insertion yields $0.981$ versus $0.999$ at the end, only a 1.8% decrease. Deletion ablations are weaker but directionally similar: removing half the sentences from the beginning gives a median cosine similarity 10.6% lower than removing half from the end, while even 50% middle deletion leaves median similarity at 95%. The asymmetry indicates that later and mid-document content can often be heavily altered without strongly changing the embedding.

The paper also reconstructs a full document embedding from sentence embeddings with ordinary least squares,

$$\mathbf{d} \approx \sum_{i=1}^{n} \beta_i \mathbf{s}_i,$$

after sentence splitting and L2 normalization. Across eight models and five datasets, train $R^2$ ranges from 0.75 to 0.99, with average $R^2 = 0.876$. Using normalized regression coefficients as sentence-importance weights, the study finds a monotonic decline in importance with later position. After randomly shuffling sentence order to remove human rhetorical structure, negative correlations remain statistically significant: APE $0.885$0, RoPE $0.885$1, and ALiBi $0.885$2. This is the clearest evidence that the effect is not just a byproduct of front-loaded human writing.

The proposed mechanism is not pooling alone, since mean or max pooling is position-invariant in theory. The paper instead points to learned positional information together with training methodology, especially truncation and context packing. If $0.885$3 is the number of non-padding tokens seen at position $0.885$4, the paper models positional importance as $0.885$5 and posits

$0.885$6

This suggests that later positions receive fewer effective updates during training and that long-document embedding behavior is partly a consequence of how encoders are exposed to context.

3. Retrieval geometry, confounders, and the limits of query-side debiasing

The retrieval literature sharpens the distinction between document-side and interaction-side failure modes. “Masking or Mitigating? Deconstructing the Impact of Query Rewriting on Retriever Biases in RAG” studies brevity, literal, position, and repetition biases with the paired preference score

$0.885$7

where $0.885$8 contains a bias-inducing feature and $0.885$9 is a factual control (Goyal et al., 7 Apr 2026). Among these biases, the paper explicitly treats literal matching as a query-document interaction bias and position bias as a document encoding bias. The reason is mechanistic: changing the query can materially alter lexical overlap, but it often fails to remove dependence on answer position within the document.

This distinction is clearest in the contrast between aggregate $0.963$0-statistics and feature-score decorrelation. Query2Doc reduces position-bias mean $0.963$1 from 5.28 to 2.81, a 47% decrease, but the mechanistic analysis shows that several rewriting methods do not genuinely decorrelate retrieval scores from normalized answer position. The paper highlights a paradoxical case where simple rewriting achieves strong $0.963$2-statistic reduction while increasing position sensitivity, interpreting this as variance-based masking rather than principled mitigation (Goyal et al., 7 Apr 2026). A plausible implication is that position bias is upstream of query phrasing, in the learned document representation or retriever-level geometry.

A complementary document-side account appears in “The Medium Is Not the Message: Deconfounding Text Embeddings via Linear Concept Erasure” (Fan et al., 1 Jul 2025). There, embeddings are decomposed as

$0.963$3

with $0.963$4 semantic content, $0.963$5 observed confounders such as source or language, and $0.963$6 unobserved confounders. Similarity between unit-normalized embeddings then contains semantic and confounder terms:

$0.963$7

The paper adapts LEACE to remove linear information about an observed confounder from frozen document embeddings via an affine map $0.963$8. The practical effect is large: multilingual Recall@1 can improve from $0.963$9 to $0.819$0 in Swiss case-summary retrieval, and source-matching Recall@1 improves across all evaluated embedding variants and source pairings, while out-of-distribution MTEB retrieval and STS scores remain essentially unchanged (Fan et al., 1 Jul 2025).

Taken together, these results suggest that many failures of document retrieval are not reducible to query wording. Position, source, and language can be encoded directly in document vectors, and query-side interventions may lower measured bias without removing the underlying sensitivity.

4. Information representation fairness across positions, languages, and scripts

Long-document embedding work has recast document encoding bias as a fairness problem. “Information Representation Fairness in Long-Document Embeddings” defines segment discoverability for a document $0.819$1 with permuted segments by

$0.819$2

and calls a model positionally fair if expected discoverability is equal across positions for every fixed segment set (Schuhmacher et al., 23 Jan 2026). The same paper defines information retention for mean-pooled models through the cosine similarity between a segment’s standalone embedding and its contextualized embedding inside the full document. This framework isolates representation unfairness from content differences by holding segment content fixed and varying only order.

Using Wikipedia articles in English, Chinese, German, Italian, Korean, and Hindi, the paper studies mGTE and jina-embeddings-v3 on documents of 3–6 segments and up to 8,192 tokens. Both models exhibit a strong first-position bias: segment-representation profiles are “L-shaped,” with the first segment much more similar to the global document embedding than later segments. In multilingual settings, a second axis appears: segments in higher-resource languages, especially English, and for mGTE often Chinese, are better represented than segments in lower-resource languages such as Hindi or Korean (Schuhmacher et al., 23 Jan 2026). The interaction is nontrivial: a later English segment can remain well represented, while a later lower-resource-language segment is doubly penalized by both position and language.

The mechanistic explanation is attention-based. In mGTE, the document representation comes from the $0.819$3 token, and the paper shows that the $0.819$4 query has a front-loaded attention distribution: among content-bearing baskets, the first basket covering tokens 2–129 receives the most attention. This makes the pooled document representation disproportionately reflect early tokens. The proposed mitigation is an inference-time attention calibration that equalizes the $0.819$5-token’s attention mass across position baskets while preserving within-basket relative proportions. It reduces first-position dominance and increases later-segment discoverability, but it does not remove language bias; in some mixed-language settings, reducing positional uplift makes English or Chinese preference more visible (Schuhmacher et al., 23 Jan 2026).

Bias in document encoding is also visible at the benchmark-construction level. DocAtlas constructs OCR and document-understanding data for 82 languages and 9 evaluation tasks using two model-free pipelines: differential rendering of native DOCX files and synthetic LaTeX generation for right-to-left scripts (Heakl et al., 12 May 2026). Core annotations are serialized in DocTag, an XML-like schema encoding component type, geometry, and text content. The motivation is explicit: model-based annotation pipelines create a circular dependency in which English-centric models generate the labels used to train the next generation of English-centric models. Evaluating 16 models, the paper finds persistent gaps in low-resource scripts: high-resource languages maintain 80–95% accuracy, while low-resource scripts fall into a 20–85% range, with 40–60% drops highlighted in the abstract (Heakl et al., 12 May 2026).

DocAtlas also shows that the source of supervision matters. Direct Preference Optimization using rendering-derived ground truth as the positive signal improves both in-domain (+1.9%) and out-of-domain (+1.8%) accuracy without measurable base-language degradation, whereas supervised fine-tuning can degrade out-of-domain performance by up to 21% (Heakl et al., 12 May 2026). The paper attributes failures of model-generated positives to systematic biases on low-resource scripts, hallucinated diacritics, and RTL column misordering. This extends document encoding bias beyond embedding geometry to the pipelines that create multilingual document labels in the first place.

5. Deliberate structural and positional inductive biases

Not all document encoding biases are pathologies. A substantial line of work introduces them intentionally to encode document structure into attention, positional signals, or output factorization. In multi-document summarization, document-aware positional encoding augments token position with a document-level signal:

$0.819$6

where the paper requires document encodings to be unique, bounded, and not remarkably larger than token positional encodings (Ma et al., 2022). The preferred document function is $0.819$7, which outperforms linear alternatives such as $0.819$8 or $0.819$9. The same model adds a linguistic-guided dependency mask that reweights self-attention with learned dependency-relation signals, reflecting a second structural bias toward syntactic organization rather than only document membership (Ma et al., 2022).

HIBRIDS injects hierarchy directly into attention logits by defining a per-head lookup table indexed by section-tree relations:

$0.968$0

The resulting attention is the standard compatibility score plus a structural bias induced by the section hierarchy, not by linear distance alone (Cao et al., 2022). The full model uses both path length and level difference; the reduced HIBRIDS$0.968$1 keeps selected salient relation types, which account for 21.6% of all relation occurrences in the data. Empirically, removing level difference, path length, or path-length asymmetry degrades performance, indicating that these structural biases encode usable document priors rather than incidental metadata.

HDT pushes this farther by building a sparse hierarchical attention graph over tokens, sentence anchors, section anchors, and a document root. A document is represented with auxiliary [DOC], [SEC], and [SENT] anchor tokens, and the hierarchical mask allows attention among siblings and along parent-child paths (He et al., 2024). The paper states that this reduces theoretical attention complexity from $0.968$2 to $0.968$3, where $0.968$4 is the length of the longest sentence, and reports that HDT pre-training can require 5–20× fewer steps than baselines to reach better or comparable downstream performance. In this design, the “bias” is a constituency prior: useful interactions are assumed to be hierarchical rather than uniformly global.

A still broader structural view appears in “A document is worth a structured record,” which reframes document recognition as transcription from a document to a structured record $0.968$5, with convention-bound documents satisfying

$0.968$6

The paper groups document types by the structure of the record they encode—sequence, set, or graph—and introduces corresponding decoder biases, losses, and masking patterns (Meyer et al., 11 Jul 2025). Monophonic sheet music uses a sequence bias, shape drawings a set bias, and engineering drawings an unrestricted graph bias via relationship nodes. The reported ablations are large: on sheet music, using a set bias instead of a sequence bias drops transcription accuracy from 96.6% to 19.4%; on shape drawings, using a sequence bias instead of a set bias drops accuracy from 74.9% to 12.0% (Meyer et al., 11 Jul 2025). Here the document encoding bias is a principled inductive bias derived from the structure the document conventionally encodes.

6. Related concepts: bias in documents, audited document-trained models, and formal documentation

Some of the closest neighboring literatures study bias expressed by documents rather than bias of document encoders. “It’s All Relative” formulates document bias as inappropriate subjectivity in textual content and learns a scalar score $0.968$7 from additive lexical-semantic features under a Bradley–Terry pairwise comparison model,

$0.968$8

The score decomposes into word-level contributions $0.968$9, and the WAND dataset contributes 895,957 Wikipedia revision pairs across 358,941 articles (Suresh et al., 2023). This is directly relevant to documents, but the paper is explicit that the target is subjective language in the document, not bias in embedding architectures.

A different but related issue is what happens when document-trained LLMs internalize associations from institutional text. “A Computational Audit of Demographic Association Encoding in ClinicalBERT Language Predictions” probes Bio_ClinicalBERT with demographic substitutions in 98 real MIMIC-III sentence templates and finds 32 statistically significant demographic effects (Soetan, 12 Jun 2026). Of these, 65.6% contradict corpus frequency direction; for Black patients the contradiction rate rises to 80.0%, and for agency attribution under the MLM probe it rises to 87.5%. The clearest pattern is suppression of active agency terms such as requested, agreed, and declined for Black Male and Black Female descriptors, alongside increased presented for Black Female. The paper interprets this as evidence that representational bias is operating predominantly through model-internal amplification rather than raw training-data inheritance (Soetan, 12 Jun 2026).

At the documentation layer, Doc-BiasO provides an ontology for formally encoding bias findings in ML pipelines. It contains 389 classes, 72 object properties, and 28 data properties, with bias:BiasMeasure as the central concept linking a bias measure to the bias it measures, the document where it is defined, the dataset where it is evaluated, and the ML task for which it is used (Russo et al., 2024). This is not a model of document-vector behavior, but it is part of the broader landscape: it formalizes how biases are represented, queried, and related to harms, applications, and datasets.

A final general lesson comes from multilingual word-vector auditing. Bias effects in distributional spaces vary substantially across embedding model, training corpus, language, and bilingual alignment; they can be emphasized or downplayed by the representation learner itself (Lauscher et al., 2019). This suggests that document encoding bias is best treated as a property of the full encoding pipeline—data, architecture, training dynamics, alignment procedure, and downstream serialization—not of “the data” alone.