---
title: Native Pre-training Paradigm Overview
url: https://www.emergentmind.com/topics/native-pre-training-paradigm
type: topic
---

# Native Pre-training Paradigm Overview

In recent literature, the expression **native pre-training paradigm** denotes a set of closely related attempts to make pre-training faithful to the structure of the eventual task, modality, data source, or deployment stack, rather than relying on indirect proxy objectives, externally pretrained modules, or purely post-hoc correction. In NLP, this includes formulating pre-training as meta-learning of an initialization optimized for rapid downstream adaptation [2004.05568], analyzing how example chunking induces an in-context bias [2110.04541], and describing pre-training as a symmetry-breaking process that forms strong-match token clusters and higher-order language structure [2509.03407]. In other areas, the same label is used for synthetic pre-pre-training before natural language [2603.10055], domain-native corpus design in antibodies and information retrieval [2211.08406] [2108.05652], task-specific visual pre-training for detection, grounding, and autonomous driving [2206.03657] [2406.07091] [2411.14716], monolithic native multimodal backbones [2510.14979] [2510.08565], and alignment performed during pre-training itself [2412.03253].

## 1. Direct optimization of downstream adaptability

A central reframing appears in "Pre-training Text Representations as Meta Learning" [2004.05568]. Conventional pre-training objectives such as negative log likelihood for language modeling, masked language modeling, next sentence prediction, translation, and discourse prediction are treated there as **proxy tasks** rather than the true goal of pre-training. The true goal is to learn text representations that transfer well to downstream tasks after fine-tuning. The proposed formulation therefore learns an initialization $\theta_0$ that is explicitly optimized for post-adaptation performance:
$$
\theta_0 = \arg\min_{\theta_0}\mathcal{L}_{\mathcal{T}_i}(\theta_k; D_{\mathcal{T}_i}^{test}),
$$
where $\theta_k$ is obtained after multiple gradient steps on task-specific training data. Within this view, standard BERT-style multi-task pre-training is a special case with meta-train depth $k=0$ [2004.05568].

The same conceptual move appears in the study of pre-training example construction. "The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design" proves that text segments appearing in the same training example can be modeled with much stronger dependencies than segments appearing in different examples, and quantifies the gap using separation rank and $\varepsilon$-separation rank [2110.04541]. Under the boundedness assumptions used there, the sequential case incurs an effective loss of depth of roughly $0.5\log_3(\eta^{-1})$ layers; for typical learning rates $\eta\in[10^{-6},10^{-4}]$, this corresponds to roughly **6 missing layers** [2110.04541]. This makes example topology itself part of the pre-training objective.

A mechanistic interpretation is provided in "Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning" [2509.03407]. In that work, pre-training on Wikipedia with BERT-6 is characterized as a process that breaks token-level symmetry, raises the average **accuracy per token** $\langle \mathrm{APT} \rangle$ along transformer depth, and organizes tokens into finite, small, strong-match token clusters visible in the token confusion matrix. The authors argue that higher-order language structures emerge even though the loss is directed only at single-token reconstruction [2509.03407].

## 2. Native objectives and the topology of training examples

Native pre-training often modifies not only the corpus but also the predictive factorization. "Meet in the Middle: A New Pre-training Paradigm" keeps a standard left-to-right decoder-only LM as a deployable component, but jointly trains a right-to-left LM on the reversed sequence and regularizes agreement between their token distributions with total variation distance [2303.07295]:
$$
\sum_{x\in S} \sum_{i=1}^{|x|} -\log\left(\pf(x_i\mid x_{<i})\right) -\log\left(\pb(x_i\mid x_{>i})\right) + \beta D_{i,x}^{TV}(\pf||\pb).
$$
The paper uses $\beta=0.1$, trains both directions with shared parameters, and couples training to a bidirectional infilling procedure that uses **4-gram matching** and parallel verification [2303.07295]. The stated aim is simultaneously better training data efficiency and stronger infilling.

The design of training examples is treated as an independent degree of freedom in kNN-Pretraining. Instead of relying only on contiguous text segments, semantically related non-neighboring sentences are appended into the same example using **RoBERTa-large sentence representations**, **FAISS**, and a **256-token** cap [2110.04541]. The reported ablation is decisive: placing related sentences in the same batch but separate examples is weaker than placing them in the same example. On SentEval, **kNN-TAPT (neighbors, in-context)** reaches an average of **61.0**, compared with **58.8** for **kNN-TAPT neighbors, in-batch** and **59.8** for TAPT [2110.04541].

These formulations suggest that a native paradigm is not reducible to “better data” in a generic sense. It also concerns where dependency is permitted to exist: inside the loss, inside the factorization, and inside the example boundary.

## 3. Domain-native corpora, supervision, and synthetic priors

A second major interpretation of native pre-training is **domain fidelity**. In antibody design, "Incorporating Pre-training Paradigm for Antibody Sequence-Structure Co-design" does not pre-train on arbitrary protein sequences; it introduces **AbBERT**, pretrained on heavy-chain variable domains $V_H$ from the **Observed Antibody Space (OAS)**, with masking restricted to **CDR positions** while the framework region is retained as context [2211.08406]. After preprocessing, **118,825,825 sequences** remain and **50 million** are sampled for pre-training; the masking rate is intentionally high, with $p_m\in\{0.5,1.0\}$, and fully masking the CDRs works best [2211.08406]. The native choice is therefore both corpus-specific and objective-specific.

A related logic governs retrieval. "Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm" argues that relevance has a **user’s view** and a **system’s view** [2108.05652]. Pre-Rank pre-trains initial query-document representations on large-scale click logs from **ORCAS** to capture the user’s view, then fine-tunes on expert-labeled relevance datasets while concatenating handcrafted learning-to-rank features in a wide-and-deep architecture to model the system’s view [2108.05652]. In this setting, native pre-training means relevance-aware pre-training rather than generic text pre-training.

The strongest challenge to natural-language-only pre-training appears in "Training Language Models via Neural Cellular Automata" [2603.10055]. That work proposes a **pre-pre-training** pipeline in which a transformer is first trained autoregressively on synthetic trajectories generated by neural cellular automata and only later transferred to web text, math, or code. The NCA generator is a **2D discrete cellular automaton on a $12\times 12$ grid with periodic boundaries and a 10-state alphabet**, and the main experiments pre-pre-train a **1.6B-parameter** Llama-style transformer on **164M NCA tokens** from the **$50\%+$ compressibility band** [2603.10055]. Reported gains include up to **6%** improvement in downstream language modeling, up to **1.6x** faster convergence, and the striking result that **160M NCA tokens beat 1.6B C4 tokens**, even when the C4 baseline is given more compute [2603.10055]. This directly contests the assumption that native pre-training must begin with human language.

A data-centric version of nativeity is advanced in "Alignment at Pre-training! Towards Native Alignment for Arabic LLMs" [2412.03253]. Native alignment is defined there as alignment during pre-training rather than only during instruction tuning or RLHF-style post-training. The workflow consists of **deduplication**, **annotation**, **training alignment workers**, and **rewriting**, with rewriting explicitly targeting format issues, values issues, content moderation, and knowledge preservation [2412.03253]. In the reported Arabic case study, **10 billion tokens** sampled from **ArabicText2022** are rewritten using **10k expert alignment data** produced by **GPT-4**, with **Qwen1.5-4B-Chat** trained as the alignment worker [2412.03253].

## 4. Monolithic multimodality and task-specific visual pre-training

In multimodal modeling, native pre-training is often defined in opposition to modular pipelines that separately pretrain a visual encoder and an LLM and connect them later. "From Pixels to Words -- Towards Native Vision-Language Primitives at Scale" argues that native VLMs should build a **shared representation from the outset** within a monolithic decoder-only backbone, using a **pre-Buffer**, a **post-LLM**, **Multi-Head Native Attention**, and **Native-RoPE** [2510.14979]. NEO pre-trains on **345M** web-scale and synthetic image-text pairs, with a **3:7 language to multimodal ratio**, and learns visual perception from scratch while the LLM weights remain frozen during the initial pre-training stage [2510.14979].

"NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints" studies the same question under explicit data constraints [2510.08565]. Its staged recipe uses **500M image-text pairs** for multimodal generative pretraining, then **185M** high-quality multimodal plus pure language data for continued pretraining, and **68M** higher-quality multimodal samples for supervised fine-tuning, all under a unified **next-token prediction** objective [2510.08565]. The paper reports that pretrained LLM initialization is crucial under this regime: the scratch-trained model needs roughly **10× more multimodal data** to approach the initialized model’s loss [2510.08565]. It also finds a positive scaling relationship between visual encoder size and LLM size.

Outside general multimodality, task-native visual pre-training has been developed for highly structured downstream problems. "Delving into the Pre-training Paradigm of Monocular 3D Object Detection" proposes **DEPT**, which imitates the downstream representation by combining depth estimation and 2D object detection, then refines the baseline with target-guided semi-dense depth estimation, keypoint-aware 2D object detection, and class-level loss adjustment [2206.03657]. With a DLA34 backbone on KITTI-3D, the moderate ${\rm AP}_{3D}70$ score of Car is boosted by **18.71\%**, and on nuScenes the **NDS** score improves by **40.41\%** relatively [2206.03657]. "Cheaper Pre-training Lunch" follows a related logic for 2D detection: **Montage pre-training** uses only the target detection dataset, assembles four extracted regions into one Montage image, and uses **ERF-adaptive dense classification**, while requiring only **1/4** of the computational resources of ImageNet pre-training [2004.12178].

Temporal grounding and autonomous driving extend the same pattern. "AutoTVG" replaces generic vision-language pre-training with pre-training on **untrimmed videos** using automatically generated **captioned moments**, so that the model learns semantic alignment and temporal boundary regression and can be evaluated **zero-shot** on TVG datasets [2406.07091]. "VisionPAD" makes pre-training native to camera-only autonomous driving by reconstructing multi-view representations with **3D Gaussian Splatting**, adding self-supervised **voxel velocity estimation**, and imposing **multi-frame photometric consistency** under pure image supervision [2411.14716]. On Occ3D, **UVTR + VisionPAD** improves **mIoU** from **30.1** to **35.4**, and for map segmentation the lane **IoU** rises from **15.0** to **20.4** [2411.14716].

## 5. Systems-native pre-training and the software stack

A distinct but increasingly important usage of *native* is software-native pre-training. "TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training" defines PyTorch-native pre-training as building the entire large-scale training stack around **DTensor**, **DeviceMesh**, **FSDP2**, **tensor parallelism**, **pipeline parallelism**, checkpointing, and compilation within PyTorch itself [2410.06511]. The design supports scaling from **8 GPUs to 512 GPUs** and from **8B to 405B** Llama 3.1 models. The paper reports accelerations of **65.08%** with 1D parallelism at the 128-GPU scale for Llama 3.1 8B, an additional **12.59%** with 2D parallelism at the 256-GPU scale for Llama 3.1 70B, and an additional **30%** with 3D parallelism at the 512-GPU scale for Llama 3.1 405B [2410.06511].

"TorchAO: PyTorch-Native Training-to-Serving Model Optimization" extends this notion by preserving low-precision numerics across the full lifecycle from pre-training to serving [2507.16099]. In the canonical workflow, pre-training is performed with **TorchTitan** and TorchAO’s **FP8** support, which composes with **`torch.compile`**, **autograd**, **FSDP2**, and **tensor parallelism** [2507.16099]. The reported result is **throughput gains of about 1.5x at 405B scale** with “virtually no change in model quality” [2507.16099]. The core mechanism is a tensor-subclass abstraction representing backend-agnostic low-precision types including **INT4**, **INT8**, **FP8**, **MXFP4**, **MXFP6**, and **MXFP8** [2507.16099]. In this systems sense, native pre-training means that pre-training, fine-tuning, quantization, sparsity, and serving share one numerical and software representation.

## 6. Empirical regularities, misconceptions, and unresolved questions

Across these works, the most recurrent empirical claim is that native pre-training yields **better initializations** rather than merely better terminal scores. In the meta-learning formulation, models with meta-train depth $k\ge 1$ perform better earlier in fine-tuning, particularly at the first epoch, and performance improves as $k$ increases from **1** to around **5 or 10** before degrading at **$k=20$** [2004.05568]. In antibody design, the pre-trained prior improves both sequence recovery and structure prediction, while one-shot decoding is reported to be both better and cheaper than iterative autoregressive decoding [2211.08406]. In retrieval, click-based pre-training captures user relevance directly and becomes more effective when combined with expert labels and handcrafted features [2108.05652].

Several common misconceptions are directly contradicted by the literature. Native does **not** mean natural-language-only pre-training: the NCA work explicitly argues that natural language is not the only path to transferable computational priors [2603.10055]. Native does **not** mean that later alignment becomes unnecessary: the Arabic native-alignment study concludes that native alignment and post-alignment are **orthogonal**, **complementary**, and can be applied simultaneously in the same model [2412.03253]. Native also does **not** simply mean “end-to-end” in a trivial sense. In multimodal work, the distinction is specifically between grafting pretrained modules together and learning a shared vision-language representation inside one backbone from the outset [2510.14979] [2510.08565].

The remaining open questions are largely about **matching**. Synthetic pre-pre-training raises the question of how the complexity of the synthetic generator should be matched to the target domain; the NCA study reports that web text and math prefer more complex trajectories, whereas code prefers intermediate complexity [2603.10055]. Native multimodal scaling raises the question of how visual encoder size should track LLM size under limited data [2510.08565]. Native alignment raises the question of how much aligned data is needed and how seed-data selection should be performed [2412.03253]. Task-native video grounding remains sensitive to out-of-distribution bias introduced by automatically generated moments and captions [2406.07091]. Taken together, these results suggest that native pre-training is best understood not as a single recipe, but as a design principle: pre-training is made “native” when its data, supervision, objective, architecture, and systems substrate are reorganized to resemble the structure of the capability one ultimately wants.

Source: https://www.emergentmind.com/topics/native-pre-training-paradigm