Native Pre-training Paradigm Overview

• The paradigm redefines pre-training by directly optimizing for downstream task adaptation using meta-learning and tailored example construction.
• It leverages native objectives across modalities, such as domain-specific corpus design and synthetic pre-pre-training, to closely match target task structures.
• Empirical evidence shows that native pre-training yields better initializations, faster fine-tuning, and enhanced performance while aligning data, supervision, and system design.

In recent literature, the expression native pre-training paradigm denotes a set of closely related attempts to make pre-training faithful to the structure of the eventual task, modality, data source, or deployment stack, rather than relying on indirect proxy objectives, externally pretrained modules, or purely post-hoc correction. In NLP, this includes formulating pre-training as meta-learning of an initialization optimized for rapid downstream adaptation (Lv et al., 2020), analyzing how example chunking induces an in-context bias (Levine et al., 2021), and describing pre-training as a symmetry-breaking process that forms strong-match token clusters and higher-order language structure (Tzach et al., 3 Sep 2025). In other areas, the same label is used for synthetic pre-pre-training before natural language (Lee et al., 9 Mar 2026), domain-native corpus design in antibodies and information retrieval (Gao et al., 2022, Bo et al., 2021), task-specific visual pre-training for detection, grounding, and autonomous driving (Li et al., 2022, Zhang et al., 2024, Zhang et al., 2024), monolithic native multimodal backbones (Diao et al., 16 Oct 2025, Tian et al., 9 Oct 2025), and alignment performed during pre-training itself (Liang et al., 2024).

1. Direct optimization of downstream adaptability

A central reframing appears in "Pre-training Text Representations as Meta Learning" (Lv et al., 2020). Conventional pre-training objectives such as negative log likelihood for language modeling, masked language modeling, next sentence prediction, translation, and discourse prediction are treated there as proxy tasks rather than the true goal of pre-training. The true goal is to learn text representations that transfer well to downstream tasks after fine-tuning. The proposed formulation therefore learns an initialization $\theta_0$ that is explicitly optimized for post-adaptation performance:

$$\theta_0 = \arg\min_{\theta_0}\mathcal{L}_{\mathcal{T}_i}(\theta_k; D_{\mathcal{T}_i}^{test}),$$

where $\theta_k$ is obtained after multiple gradient steps on task-specific training data. Within this view, standard BERT-style multi-task pre-training is a special case with meta-train depth $k=0$ (Lv et al., 2020).

The same conceptual move appears in the study of pre-training example construction. "The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design" proves that text segments appearing in the same training example can be modeled with much stronger dependencies than segments appearing in different examples, and quantifies the gap using separation rank and $\varepsilon$-separation rank (Levine et al., 2021). Under the boundedness assumptions used there, the sequential case incurs an effective loss of depth of roughly $0.5\log_3(\eta^{-1})$ layers; for typical learning rates $\eta\in[10^{-6},10^{-4}]$, this corresponds to roughly 6 missing layers (Levine et al., 2021). This makes example topology itself part of the pre-training objective.

A mechanistic interpretation is provided in "Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning" (Tzach et al., 3 Sep 2025). In that work, pre-training on Wikipedia with BERT-6 is characterized as a process that breaks token-level symmetry, raises the average accuracy per token $\langle \mathrm{APT} \rangle$ along transformer depth, and organizes tokens into finite, small, strong-match token clusters visible in the token confusion matrix. The authors argue that higher-order language structures emerge even though the loss is directed only at single-token reconstruction (Tzach et al., 3 Sep 2025).

2. Native objectives and the topology of training examples

Native pre-training often modifies not only the corpus but also the predictive factorization. "Meet in the Middle: A New Pre-training Paradigm" keeps a standard left-to-right decoder-only LM as a deployable component, but jointly trains a right-to-left LM on the reversed sequence and regularizes agreement between their token distributions with total variation distance (Nguyen et al., 2023):

$\sum_{x\in S} \sum_{i=1}^{|x|} -\log\left(\pf(x_i\mid x_{<i})\right) -\log\left(\pb(x_i\mid x_{>i})\right) + \beta D_{i,x}^{TV}(\pf||\pb).$

The paper uses $\beta=0.1$, trains both directions with shared parameters, and couples training to a bidirectional infilling procedure that uses 4-gram matching and parallel verification (Nguyen et al., 2023). The stated aim is simultaneously better training data efficiency and stronger infilling.

The design of training examples is treated as an independent degree of freedom in kNN-Pretraining. Instead of relying only on contiguous text segments, semantically related non-neighboring sentences are appended into the same example using RoBERTa-large sentence representations, FAISS, and a 256-token cap (Levine et al., 2021). The reported ablation is decisive: placing related sentences in the same batch but separate examples is weaker than placing them in the same example. On SentEval, kNN-TAPT (neighbors, in-context) reaches an average of 61.0, compared with 58.8 for kNN-TAPT neighbors, in-batch and 59.8 for TAPT (Levine et al., 2021).

These formulations suggest that a native paradigm is not reducible to “better data” in a generic sense. It also concerns where dependency is permitted to exist: inside the loss, inside the factorization, and inside the example boundary.

3. Domain-native corpora, supervision, and synthetic priors

A second major interpretation of native pre-training is domain fidelity. In antibody design, "Incorporating Pre-training Paradigm for Antibody Sequence-Structure Co-design" does not pre-train on arbitrary protein sequences; it introduces AbBERT, pretrained on heavy-chain variable domains $$\theta_0 = \arg\min_{\theta_0}\mathcal{L}_{\mathcal{T}_i}(\theta_k; D_{\mathcal{T}_i}^{test}),$$0 from the Observed Antibody Space (OAS), with masking restricted to CDR positions while the framework region is retained as context (Gao et al., 2022). After preprocessing, 118,825,825 sequences remain and 50 million are sampled for pre-training; the masking rate is intentionally high, with $$\theta_0 = \arg\min_{\theta_0}\mathcal{L}_{\mathcal{T}_i}(\theta_k; D_{\mathcal{T}_i}^{test}),$$1, and fully masking the CDRs works best (Gao et al., 2022). The native choice is therefore both corpus-specific and objective-specific.

A related logic governs retrieval. "Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm" argues that relevance has a user’s view and a system’s view (Bo et al., 2021). Pre-Rank pre-trains initial query-document representations on large-scale click logs from ORCAS to capture the user’s view, then fine-tunes on expert-labeled relevance datasets while concatenating handcrafted learning-to-rank features in a wide-and-deep architecture to model the system’s view (Bo et al., 2021). In this setting, native pre-training means relevance-aware pre-training rather than generic text pre-training.

The strongest challenge to natural-language-only pre-training appears in "Training LLMs via Neural Cellular Automata" (Lee et al., 9 Mar 2026). That work proposes a pre-pre-training pipeline in which a transformer is first trained autoregressively on synthetic trajectories generated by neural cellular automata and only later transferred to web text, math, or code. The NCA generator is a 2D discrete cellular automaton on a $$\theta_0 = \arg\min_{\theta_0}\mathcal{L}_{\mathcal{T}_i}(\theta_k; D_{\mathcal{T}_i}^{test}),$$2 grid with periodic boundaries and a 10-state alphabet, and the main experiments pre-pre-train a 1.6B-parameter Llama-style transformer on 164M NCA tokens from the $$\theta_0 = \arg\min_{\theta_0}\mathcal{L}_{\mathcal{T}_i}(\theta_k; D_{\mathcal{T}_i}^{test}),$$3 compressibility band (Lee et al., 9 Mar 2026). Reported gains include up to 6% improvement in downstream language modeling, up to 1.6x faster convergence, and the striking result that 160M NCA tokens beat 1.6B C4 tokens, even when the C4 baseline is given more compute (Lee et al., 9 Mar 2026). This directly contests the assumption that native pre-training must begin with human language.

A data-centric version of nativeity is advanced in "Alignment at Pre-training! Towards Native Alignment for Arabic LLMs" (Liang et al., 2024). Native alignment is defined there as alignment during pre-training rather than only during instruction tuning or RLHF-style post-training. The workflow consists of deduplication, annotation, training alignment workers, and rewriting, with rewriting explicitly targeting format issues, values issues, content moderation, and knowledge preservation (Liang et al., 2024). In the reported Arabic case study, 10 billion tokens sampled from ArabicText2022 are rewritten using 10k expert alignment data produced by GPT-4, with Qwen1.5-4B-Chat trained as the alignment worker (Liang et al., 2024).

4. Monolithic multimodality and task-specific visual pre-training

In multimodal modeling, native pre-training is often defined in opposition to modular pipelines that separately pretrain a visual encoder and an LLM and connect them later. "From Pixels to Words -- Towards Native Vision-Language Primitives at Scale" argues that native VLMs should build a shared representation from the outset within a monolithic decoder-only backbone, using a pre-Buffer, a post-LLM, Multi-Head Native Attention, and Native-RoPE (Diao et al., 16 Oct 2025). NEO pre-trains on 345M web-scale and synthetic image-text pairs, with a 3:7 language to multimodal ratio, and learns visual perception from scratch while the LLM weights remain frozen during the initial pre-training stage (Diao et al., 16 Oct 2025).

"NaViL: Rethinking Scaling Properties of Native Multimodal LLMs under Data Constraints" studies the same question under explicit data constraints (Tian et al., 9 Oct 2025). Its staged recipe uses 500M image-text pairs for multimodal generative pretraining, then 185M high-quality multimodal plus pure language data for continued pretraining, and 68M higher-quality multimodal samples for supervised fine-tuning, all under a unified next-token prediction objective (Tian et al., 9 Oct 2025). The paper reports that pretrained LLM initialization is crucial under this regime: the scratch-trained model needs roughly 10× more multimodal data to approach the initialized model’s loss (Tian et al., 9 Oct 2025). It also finds a positive scaling relationship between visual encoder size and LLM size.

Outside general multimodality, task-native visual pre-training has been developed for highly structured downstream problems. "Delving into the Pre-training Paradigm of Monocular 3D Object Detection" proposes DEPT, which imitates the downstream representation by combining depth estimation and 2D object detection, then refines the baseline with target-guided semi-dense depth estimation, keypoint-aware 2D object detection, and class-level loss adjustment (Li et al., 2022). With a DLA34 backbone on KITTI-3D, the moderate $$\theta_0 = \arg\min_{\theta_0}\mathcal{L}_{\mathcal{T}_i}(\theta_k; D_{\mathcal{T}_i}^{test}),$$4 score of Car is boosted by 18.71\%, and on nuScenes the NDS score improves by 40.41\% relatively (Li et al., 2022). "Cheaper Pre-training Lunch" follows a related logic for 2D detection: Montage pre-training uses only the target detection dataset, assembles four extracted regions into one Montage image, and uses ERF-adaptive dense classification, while requiring only 1/4 of the computational resources of ImageNet pre-training (Zhou et al., 2020).

Temporal grounding and autonomous driving extend the same pattern. "AutoTVG" replaces generic vision-language pre-training with pre-training on untrimmed videos using automatically generated captioned moments, so that the model learns semantic alignment and temporal boundary regression and can be evaluated zero-shot on TVG datasets (Zhang et al., 2024). "VisionPAD" makes pre-training native to camera-only autonomous driving by reconstructing multi-view representations with 3D Gaussian Splatting, adding self-supervised voxel velocity estimation, and imposing multi-frame photometric consistency under pure image supervision (Zhang et al., 2024). On Occ3D, UVTR + VisionPAD improves mIoU from 30.1 to 35.4, and for map segmentation the lane IoU rises from 15.0 to 20.4 (Zhang et al., 2024).

5. Systems-native pre-training and the software stack

A distinct but increasingly important usage of native is software-native pre-training. "TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training" defines PyTorch-native pre-training as building the entire large-scale training stack around DTensor, DeviceMesh, FSDP2, tensor parallelism, pipeline parallelism, checkpointing, and compilation within PyTorch itself (Liang et al., 2024). The design supports scaling from 8 GPUs to 512 GPUs and from 8B to 405B Llama 3.1 models. The paper reports accelerations of 65.08% with 1D parallelism at the 128-GPU scale for Llama 3.1 8B, an additional 12.59% with 2D parallelism at the 256-GPU scale for Llama 3.1 70B, and an additional 30% with 3D parallelism at the 512-GPU scale for Llama 3.1 405B (Liang et al., 2024).

"TorchAO: PyTorch-Native Training-to-Serving Model Optimization" extends this notion by preserving low-precision numerics across the full lifecycle from pre-training to serving (Or et al., 21 Jul 2025). In the canonical workflow, pre-training is performed with TorchTitan and TorchAO’s FP8 support, which composes with torch.compile, autograd, FSDP2, and tensor parallelism (Or et al., 21 Jul 2025). The reported result is throughput gains of about 1.5x at 405B scale with “virtually no change in model quality” (Or et al., 21 Jul 2025). The core mechanism is a tensor-subclass abstraction representing backend-agnostic low-precision types including INT4, INT8, FP8, MXFP4, MXFP6, and MXFP8 (Or et al., 21 Jul 2025). In this systems sense, native pre-training means that pre-training, fine-tuning, quantization, sparsity, and serving share one numerical and software representation.

6. Empirical regularities, misconceptions, and unresolved questions

Across these works, the most recurrent empirical claim is that native pre-training yields better initializations rather than merely better terminal scores. In the meta-learning formulation, models with meta-train depth $$\theta_0 = \arg\min_{\theta_0}\mathcal{L}_{\mathcal{T}_i}(\theta_k; D_{\mathcal{T}_i}^{test}),$$5 perform better earlier in fine-tuning, particularly at the first epoch, and performance improves as $$\theta_0 = \arg\min_{\theta_0}\mathcal{L}_{\mathcal{T}_i}(\theta_k; D_{\mathcal{T}_i}^{test}),$$6 increases from 1 to around 5 or 10 before degrading at $$\theta_0 = \arg\min_{\theta_0}\mathcal{L}_{\mathcal{T}_i}(\theta_k; D_{\mathcal{T}_i}^{test}),$$7 (Lv et al., 2020). In antibody design, the pre-trained prior improves both sequence recovery and structure prediction, while one-shot decoding is reported to be both better and cheaper than iterative autoregressive decoding (Gao et al., 2022). In retrieval, click-based pre-training captures user relevance directly and becomes more effective when combined with expert labels and handcrafted features (Bo et al., 2021).

Several common misconceptions are directly contradicted by the literature. Native does not mean natural-language-only pre-training: the NCA work explicitly argues that natural language is not the only path to transferable computational priors (Lee et al., 9 Mar 2026). Native does not mean that later alignment becomes unnecessary: the Arabic native-alignment study concludes that native alignment and post-alignment are orthogonal, complementary, and can be applied simultaneously in the same model (Liang et al., 2024). Native also does not simply mean “end-to-end” in a trivial sense. In multimodal work, the distinction is specifically between grafting pretrained modules together and learning a shared vision-language representation inside one backbone from the outset (Diao et al., 16 Oct 2025, Tian et al., 9 Oct 2025).

The remaining open questions are largely about matching. Synthetic pre-pre-training raises the question of how the complexity of the synthetic generator should be matched to the target domain; the NCA study reports that web text and math prefer more complex trajectories, whereas code prefers intermediate complexity (Lee et al., 9 Mar 2026). Native multimodal scaling raises the question of how visual encoder size should track LLM size under limited data (Tian et al., 9 Oct 2025). Native alignment raises the question of how much aligned data is needed and how seed-data selection should be performed (Liang et al., 2024). Task-native video grounding remains sensitive to out-of-distribution bias introduced by automatically generated moments and captions (Zhang et al., 2024). Taken together, these results suggest that native pre-training is best understood not as a single recipe, but as a design principle: pre-training is made “native” when its data, supervision, objective, architecture, and systems substrate are reorganized to resemble the structure of the capability one ultimately wants.