KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates

Abstract: Standard LLM pre-training typically treats corpora as flattened token sequences, often overlooking the real-world context that humans naturally rely on to contextualize information. To bridge this gap, we introduce Knowledge Coordinate Conditioning (KoCo), a simple method that maps every document into a three-dimensional semantic coordinate. By prepending these coordinates as textual prefixes for pre-training, we aim to equip the model with explicit contextual awareness to learn the documents within the real-world knowledge structure. Experiment results demonstrate that KoCo significantly enhances performance across 10 downstream tasks and accelerates pre-training convergence by approximately 30\%. Furthermore, our analysis indicates that explicitly modeling knowledge coordinates helps the model distinguish stable facts from noise, effectively mitigating hallucination in generated outputs.

• The paper introduces a novel method that conditions LLM pre-training on explicit, multi-dimensional knowledge coordinates (Source, Content, Stability).
• It demonstrates accelerated convergence (~30%) and improved downstream generalization with gains up to 5.52 points over baseline models.
• KoCo enables controllable inference by steering outputs with task-specific knowledge parameters, effectively reducing hallucination rates.

KoCo: Conditioning LLM Pre-training on Knowledge Coordinates

Motivation and Problem Statement

Standard LLM pre-training disregards the higher-order contextual signals underpinning human-like knowledge acquisition by treating corpora as undifferentiated token streams. This paradigm ignores the source reliability, content function, and temporal stability of documents, compressing heterogeneous web data into flattened statistical signals. Such flattening impairs data efficiency and amplifies hallucination, as token statistics from ephemeral forum discussions and peer-reviewed theorems are given indiscriminate treatment through the NLL objective. Prior interventions, notably URL-based context methods and scalar-quality data selection, partially mitigate these issues but lack an explicit, multi-dimensional contextual framework. Binary filtering in data selection disregards the nuanced utility of noisy data, while URL prefixing fails to abstract document roles and temporal signatures. The absence of cognitive abstraction hinders alignment with human knowledge organization.

Figure 1: KoCo transforms DIKW into an objective representation of the knowledge contained in the corpus.

KoCo Framework: Knowledge Coordinates as Explicit Context

KoCo introduces a methodology wherein each training document is mapped into a three-dimensional knowledge coordinate $\mathcal{T}$ covering Source (origin/channel), Content (textual function/style), and Stability (temporal persistence). The taxonomy deliberately abstracts away from topic or domain, focusing instead on objective annotation. It spans labels such as Academic, Media, Pedagogical, Opinion, Evergreen, and Ephemeral, effectively simulating the meta-observational signals humans derive when consuming information. A lightweight LLM-based classifier tags each document, and the coordinate is prepended as a structured textual prefix before the document text during pre-training. The MLM loss only supervises prediction over the document, not its prefix, thus enforcing conditioning without leakage of trivial cues.

Figure 2: Overview of Knowledge Coordinate Conditioning (KoCo). Different from standard pre-training (a), KoCo (b) conditions the model on the context tag, facilitating discrimination of knowledge properties.

KoCo generalizes beyond prior metadata or data selection paradigms by providing continuous, structured conditioning instead of binary selection or hard identifier mapping. The coordinate space covers over 99.5% of web-scale DCLM documents, validating its sufficiency for real-world data.

Experimental Results: Convergence, Generalization, and Hallucination Mitigation

KoCo is evaluated on LLaMA-like architectures up to 1.6B parameters, using MeCo (a metadata-URL-pretrained baseline) and fastText-selected data as comparators. KoCo shows clear, consistent advantages in downstream generalization (OLMES suite: COPA, ARC, CSQA, PIQA, SIQA, LogiQA, TruthfulQA, etc.) with absolute increases of up to 5.52 points (IFEval) over leading baselines. More importantly, KoCo accelerates convergence by ~30% in pre-training from scratch, confirming both its regularization and informativeness across both 0.3B and 0.6B model scales.

Figure 3: Pre-training from scratch results comparing KoCo (red) with standard paradigm (blue); KoCo consistently yields lower training loss and higher downstream task scores, exhibiting accelerated convergence.

Ablation shows the orthogonality and complementarity of the Source, Content, and Stability axes: removing any dimension consistently degrades performance, confirming the multi-dimensional coordinate's necessity for maximal effect.

Knowledge Steering and Hallucination Control

Because the coordinate prefix is exposed at inference, LLM answers can be explicitly steered toward desired knowledge modes. Applying task-specific prefixes (e.g., {Source: Academic; Content: Pedagogical; Stability: Evergreen} for logic tasks) further enhances accuracy, with KoCo realizing an average 0.97% gain from conditional inference versus generic prompting. Critically, conditioning on reliable (or unreliable) source coordinates on TruthfulQA demonstrably modulates hallucination rates (absolute truthfulness delta of +5% between low- and high-reliability contexts), enabling user-controllable truthfulness—a significant operational advantage in safety-sensitive deployments.

Representation Dynamics and Internal Mechanisms

KoCo's effect on internal representations is measured by embedding-space analysis via PCA of token-level activations for factual versus opinion statements. Standard LLMs show substantial cluster overlap; KoCo-trained models effectuate strong separation in this latent space, indicating semantically disentangled representations tied to objective knowledge properties.

Figure 4: PCA visualization of hidden states for factual (blue) vs. opinion-based (red) statements; KoCo models learn clear cluster separation, unlike standard pre-training which yields substantial overlap.

Per-document perplexity is reduced when using the correct coordinate, confirming that the model has internalized the mapping between coordinate and content statistics, not merely memorizing prefixes.

Corpus Analysis, Tagger Robustness, and Distillation Control

Analysis of the DCLM corpus shows media/community-derived, discussion/opinion content is dominant, with academic/government slices sparse. KoCo's objective coordinates facilitate robust modeling even amidst such imbalance. Tagger reliability is confirmed by high agreement with SOTA LLMs (Gemini-3, GPT-5.1) on test set coordinate annotation. Importantly, using a weak 110M BERT-base tagger for coordinate generation yields indistinguishably strong KoCo performance compared to large-Qwen3-4B tagging, nullifying arguments that apparent gains are due to distillation from a powerful teacher.

Figure 5: Distribution of 10,000 documents from DCLM in knowledge coordinate space; media and community sources dominate, highlighting the need for explicit contextualization signals.

Figure 6: KoCo performance using different taggers; BERT-base-based coordinates yield results on par with Qwen3-4B, confirming gains stem from coordinate abstraction, not teacher distillation.

Implications and Prospects

KoCo demonstrates that explicit multi-dimensional knowledge contextualization at the pre-training stage (rather than post-hoc SFT/RLHF) yields more data-efficient, controllable, and hallucination-robust LLMs. The framework supports scale-impartial knowledge ingestion, optimal use of "noisy" data, and operational steerability at inference with minimal added computational complexity. KoCo also suggests a pathway toward upstreaming alignment controls into base model training, moving certain behavioral guarantees from downstream SFT/RLHF into the pre-training loop itself.

Practically, KoCo can be integrated with standard data selection pipelines, boosting both efficiency and quality, and may further reduce the resource/environmental footprint required for strong generalization.

Conclusion

KoCo advances the LLM pre-training methodology by bridging corpus-level metadata and conditional data selection into an objective, structured, and operationally impactful coordinate system. It consistently yields efficiency, accuracy, and controllability benefits, and opens new lines for integrating pre-training alignment into foundational stages, with significant implications for LLM engineering at scale. Future research should verify scaling to ≥7B parameters and test taxonomy refinements for further improvements in inference behavioral control.