Self-Pretraining (SPT): Task-Domain Pretraining

• Self-Pretraining (SPT) is a training paradigm that uses unlabeled or pseudo-labeled data from the target domain to shape model initialization before supervised adaptation.
• It employs various self-supervised objectives—such as masked token prediction, inpainting, and contrastive modeling—to extract structure from task-specific data.
• SPT has shown practical benefits across NLP, genomics, aerial segmentation, and control tasks, offering improved sample efficiency and performance under limited label conditions.

Searching arXiv for the cited works and related usage of “Self-Pretraining (SPT)” to ground the article. Self-Pretraining (SPT) denotes a family of pretrain-then-finetune procedures in which model parameters are first optimized on unlabeled, pseudo-labeled, or otherwise self-derived data drawn from the target task distribution or a closely related domain, and are then reused for supervised adaptation. Across the recent literature, the term is not fully standardized. In some works it refers narrowly to self-supervised pretraining on the downstream dataset itself, with no external corpus (Krishna et al., 2022); in others it denotes task-specific pretraining on unlabeled in-domain sequences or images before standard fine-tuning (Mupparapu et al., 21 Jun 2025, Polley et al., 31 Mar 2025); and in semi-supervised settings it can refer to iterative pseudo-label-based pretraining of a student model before refinement on gold labels (Karisani et al., 2021). Despite these variations, the unifying idea is consistent: SPT seeks to improve supervised learning by first shaping the initialization with structure extracted from the target distribution itself, rather than relying only on random initialization or generic upstream corpora.

1. Conceptual scope and historical positioning

SPT occupies an intermediate position between training from scratch and large-scale generic pretraining. In the downstream-corpus NLP setting, the model is pretrained on the same training texts later used for supervised fine-tuning, but with labels ignored during pretraining (Krishna et al., 2022). In genomics, the encoder is pretrained on task-specific unlabeled sequences rather than the full human genome, then fine-tuned on BEND tasks such as gene finding, chromatin accessibility, histone modification, and CpG methylation (Mupparapu et al., 21 Jun 2025). In aerial road extraction, a segmentation network is first trained on abundant unlabeled aerial imagery with an inpainting objective, then steered toward road structure through guided inpainting, and finally fine-tuned for segmentation (Polley et al., 31 Mar 2025). In sequential decision-making, reward-free offline trajectories provide the self-supervised signal for pretraining a Control Transformer before downstream imitation learning or reinforcement learning (Sun et al., 2023).

This family resemblance is stronger than a shared acronym. What recurs is a two-stage logic: first learn reusable features from data that are already available in the task domain, then apply supervised adaptation under limited labels, limited compute, or both. The empirical motivation is also recurrent. Full-scale upstream pretraining is often expensive or impractical, whereas unlabeled task-domain data are relatively easy to obtain in remote sensing, genomics, pathology, speech, and many industrial text settings (Mupparapu et al., 21 Jun 2025, Polley et al., 31 Mar 2025, &&&10&&&, Chen et al., 2021).

A narrower but influential formulation treats SPT as a diagnostic tool for understanding pretraining itself. In that formulation, using the downstream corpus for both pretraining and fine-tuning isolates the effect of the self-supervised objective from the effect of massive external data (Krishna et al., 2022). A later sequence-classification study sharpened this interpretation by arguing that SPT helps Transformers not primarily because it adds information, but because masked reconstruction learns attention patterns that supervised labels often fail to induce from random initialization (Coser et al., 20 May 2026).

2. Canonical training pattern and objective families

Across domains, SPT is defined more by training structure than by any single loss. The pretraining objective may be masked token prediction, masked reconstruction, inpainting, contrastive trajectory modeling, or pseudo-label distillation. The downstream stage typically restores the task head and fine-tunes end-to-end.

Domain	Pretraining source	Pretraining target
NLP sequence classification	Downstream training text	MLM or RTD on the same corpus
Genomics	Task-specific unlabeled DNA sequences	MLM on nucleotide sequences
Aerial segmentation	Unlabeled aerial images	Inpainting, then guided inpainting
Sequential control	Offline multi-task trajectories	Forward, inverse, and masked hindsight-control prediction

In genomics, SPT uses masked language modeling over one-hot encoded DNA tokens from the vocabulary $\{A, C, G, T, N, [MASK], [PAD]\}$, with mask probability $0.15$ and the standard 80/10/10 corruption rule. The loss is

$$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$

and the pretrained encoder is then reused with a task-specific predictor for supervised fine-tuning (Mupparapu et al., 21 Jun 2025).

In aerial imagery, the pretraining objective is image reconstruction under masking. With original image $X$, network output $O$, and binary mask $M$, the first-stage loss is split into visible-pixel and missing-pixel terms,

$$L_{id}^0 = MSE(O \odot M, X \odot M), \qquad L_{fill}^0 = MSE(O \odot (1-M), X \odot (1-M)),$$

with weighted total loss

$$L_{total}^0 = w_{id}^0 \cdot L_{id}^0 + w_{fill}^0 \cdot L_{fill}^0.$$

A second guided-inpainting stage restricts the reconstruction loss to road pixels using the road mask $S$, before final segmentation fine-tuning (Polley et al., 31 Mar 2025).

In sequence classification, the self-pretraining stage may itself be formulated as masked reconstruction on the downstream inputs. For a masked-position set $\mathcal M_b$, the pretraining loss is

$0.15$0

with either $0.15$1 for continuous targets or token cross-entropy for discrete targets (Coser et al., 20 May 2026).

Control-oriented SPT departs from token masking and instead derives supervision from transition structure. SMART combines forward dynamics prediction, inverse dynamics prediction, and random masked hindsight control, using offline trajectories and no external semantic labels (Sun et al., 2023). This suggests that within SPT, the decisive design variable is not modality but alignment between the self-supervised signal and the structure of the downstream task.

3. Representative instantiations across application domains

In natural language processing, large-scale evidence for downstream-corpus SPT shows that it can rival conventional BookCorpus-plus-Wikipedia pretraining despite using roughly $0.15$2--$0.15$3 less data (Krishna et al., 2022). With ELECTRA-small, self-pretraining outperformed standard BookWiki pretraining on 7 of 10 classification datasets; with RoBERTa-base, it did so on 5 of 10. On structured tasks, it still recovered substantial fractions of the usual pretraining gain: for ELECTRA-small, SPT achieved benefit values of $0.15$4 on SQuAD, $0.15$5 on SWAG, $0.15$6 on HellaSwag, and $0.15$7 on CoNLL-2012 NER (Krishna et al., 2022).

In genomics, task-specific SPT was evaluated under matched compute against scratch training. The core comparison reported gene finding, chromatin accessibility, histone modification, and CpG methylation scores of $0.15$8 for scratch and $0.15$9 for SPT, with SPT + CRF reaching $$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$0 MCC on gene finding (Mupparapu et al., 21 Jun 2025). A particularly strong sample-efficiency result was that the 25% labeled-data SPT model on CpG methylation outperformed the 100% scratch model (Mupparapu et al., 21 Jun 2025).

In aerial road extraction, the strongest gains appeared in low-label and domain-shifted settings. On DeepGlobe with SPIN RoadMapper, the smallest labeled split $$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$1 improved from $$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$2 Road IoU to $$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$3, while the half-sized split $$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$4 improved from $$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$5 to $$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$6 (Polley et al., 31 Mar 2025). On CITY-OSM with EmekU-Net, the road IoU on the smallest split $$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$7 rose from $$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$8 to $$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i: m_i=1} \log P_{\theta}(x_i \mid \widetilde{x}),$$9, and on the Berlin-held-out split $X$0 from $X$1 to $X$2 (Polley et al., 31 Mar 2025). The ablation isolating guided inpainting showed that inpainting alone already helped, but the second road-focused stage added a further boost.

In sequential control, SMART reported consistent improvements in learning efficiency across seen tasks, unseen tasks, and unseen domains in DeepMind Control Suite, and remained useful even when pretrained on random trajectories rather than higher-quality exploratory replay (Sun et al., 2023). Although the paper’s strongest aggregate comparisons are qualitative rather than tabulated numerically, its importance for SPT lies in showing that pretrain-then-adapt logic extends beyond perception and sequence modeling into reward-free control.

Related self-pretraining frameworks appear in computational pathology and speech. Whole-slide self-supervision with a frozen patch encoder and a trainable slide transformer improved slide-level representations across diagnostic tasks, with the best self-supervised model reaching $X$3 mean class accuracy on SRH and $X$4 on H&E molecular classification (Hou et al., 2024). In speech, tts4pretrain injected text into self-supervised speech pretraining through synthesized speech plus auxiliary sequence losses, yielding relative WER reductions of 10% on LibriSpeech and up to 15% on Voice Search over speech-only pretraining (Chen et al., 2021). These frameworks are not always labeled “SPT,” but they instantiate the same pretrain-on-task-domain-structure principle.

4. Mechanistic interpretations

Several papers move beyond performance reporting and propose explicit mechanisms. In sequence classification, the central claim is that label supervision can be locally blind to useful attention-score directions, whereas masked reconstruction is sensitive to them (Coser et al., 20 May 2026). In the paper’s simplified theory, supervised mean-pooled loss satisfies

$X$5

for certain attention-score directions at uniform attention, while the reconstruction objective has nonzero derivative

$X$6

Empirically, this was linked to the ability of SPT to learn query-key structure that converts absolute positional encodings into proximity-biased attention scores. In one-layer models, initializing only $X$7 from an SPT checkpoint largely reproduced the full SPT gain on CIFAR10 and PathFinder (Coser et al., 20 May 2026).

In aerial imagery, the proposed mechanism is geometric and scene-structural rather than attentional. Large contiguous masking clusters force the network to infer road continuity, intersections, road markings, and scene layout from broader spatial context rather than local interpolation (Polley et al., 31 Mar 2025). Guided inpainting then reduces the objective mismatch between generic reconstruction and the downstream semantic target by focusing the loss on road pixels.

In genomics, the interpretation is that task-relevant unlabeled sequences provide a more useful inductive bias than random initialization under limited compute, especially for long-context structured tasks such as gene finding (Mupparapu et al., 21 Jun 2025). The strongest supporting evidence is optimization-dynamic rather than purely endpoint-based: extending scratch training to 50 epochs still plateaued below SPT on gene finding, and on CpG methylation the SPT model held a validation advantage from the first fine-tuning epoch onward (Mupparapu et al., 21 Jun 2025).

In control, the mechanistic emphasis shifts to controllability. SMART argues that generic visual SSL is insufficient because downstream policies require representations encoding one-step dynamics, action-state coupling, and longer-horizon action consistency. Its combined forward, inverse, and masked-hindsight losses are therefore designed to make the latent representation control-centric rather than merely perceptual (Sun et al., 2023).

5. Relation to adjacent paradigms and terminological ambiguities

SPT overlaps with, but should not be collapsed into, several neighboring paradigms. It differs from training from scratch because the self-supervised or self-derived stage changes the initialization before any gold-label optimization (Krishna et al., 2022, Mupparapu et al., 21 Jun 2025). It differs from large-scale generic pretraining because the pretraining corpus is drawn from the downstream task distribution or a closely related domain rather than a broad upstream source (Krishna et al., 2022, Mupparapu et al., 21 Jun 2025). It differs from task-adaptive pretraining in the usual sense because some SPT formulations start from random initialization rather than from an already pretrained foundation model (Krishna et al., 2022). It also differs from classic self-training because pseudo-labels need not be permanently added to the gold set, and because what is transferred across iterations may be a hypothesis rather than a fixed pseudo-labeled dataset (Karisani et al., 2021).

A semi-supervised text-classification method explicitly called “Self-Pretraining” illustrates this neighboring usage. There, two classifiers alternate teacher and student roles; one labels a random subset of unlabeled documents, and the other is pretrained on those pseudo-labels before fine-tuning on gold labels. The method is threshold-free, iterative, and designed to revise pseudo-label beliefs over time while mitigating semantic drift (Karisani et al., 2021). This is recognizably part of the SPT family, but its mechanics are pseudo-label-driven rather than purely self-supervised.

Another adjacent line is detector pretraining through self-training. SimDETR constructs clustering-based pseudo-boxes and pseudo-classes, trains a detector with a standard DETR loss, then refreshes the pseudo-labels with detector-generated predictions for a second pretraining stage (Metaxas et al., 2023). This is SPT in a broader bootstrapping sense: the model learns from self-generated supervision that improves over time.

The acronym itself is overloaded. In “General Intelligence Requires Reward-based Pretraining,” SPT means supervised pretraining by next-token prediction, contrasted with reward-based pretraining (Han et al., 26 Feb 2025). In visual adaptation, SPT may instead mean “Self-Prompt Tuning” (Wang et al., 2024) or “Sensitivity-aware visual Parameter-efficient fine-Tuning” (He et al., 2023). In whole-slide pathology, the relevant method is called S3L rather than SPT, even though it is conceptually a transformer slide-pretraining framework (Hou et al., 2024). This lexical instability is a common source of confusion and makes disambiguation essential in survey writing.

6. Limitations, reproducibility, and research directions

The strongest shared limitation is scope. Many SPT papers show convincing gains over scratch training, but do not fully position themselves against the broader SSL landscape. The aerial road-extraction study, for example, compares only against training from scratch and does not evaluate contrastive SSL, masked autoencoders, supervised ImageNet pretraining, or remote-sensing-specific alternatives (Polley et al., 31 Mar 2025). The genomics study similarly argues compute efficiency through matched scratch comparisons, but does not provide direct FLOP or wall-clock accounting against genome-scale pretraining (Mupparapu et al., 21 Jun 2025).

A second limitation is that “task-specific” often remains narrower in experiment than in rhetoric. The genomics paper frames SPT as task-specific self-pretraining, yet uses one shared encoder pretrained only on gene-finding sequences and transfers it across all downstream tasks (Mupparapu et al., 21 Jun 2025). The sequence-classification study identifies a compelling attention-learning bottleneck, but its most detailed mechanistic evidence comes from Long-Range Arena and small-scale UCR datasets, with larger multi-seed analyses concentrated on CIFAR10 and PathFinder (Coser et al., 20 May 2026). SMART demonstrates cross-task and cross-domain transfer in DeepMind Control Suite, but its generality beyond image-based continuous control remains open (Sun et al., 2023).

A third limitation is operational. SPT typically increases training cost while leaving inference cost unchanged. The aerial paper states this trade-off explicitly: training time increases, but the final deployed architecture is not altered (Polley et al., 31 Mar 2025). Sequence-classification SPT likewise raises total optimization cost because the reported setup uses 200 epochs of self-pretraining followed by 100 epochs of supervised fine-tuning, versus 100 epochs from scratch (Coser et al., 20 May 2026).

The strongest practical implication is therefore selective rather than universal. SPT is most compelling when labels are scarce, unlabeled in-domain data are plentiful, and a full generic-pretraining pipeline is unjustified or unavailable. The evidence is especially strong in low-label segmentation (Polley et al., 31 Mar 2025), compute-constrained genomics (Mupparapu et al., 21 Jun 2025), and downstream-corpus NLP where large fractions of the usual pretraining benefit can be recovered without external corpora (Krishna et al., 2022). A plausible implication is that future work will increasingly treat SPT not as a single algorithm, but as a design principle: choose a self-derived pretraining objective whose inductive bias is closely matched to the structure of the eventual supervised task.