---
title: Zero-Pretraining Deep Learning Methods
url: https://www.emergentmind.com/topics/zero-pretraining-deep-learning-methods
type: topic
---

# Zero-Pretraining Deep Learning Methods

Zero-pretraining deep learning methods are a class of algorithms and frameworks that eschew reliance on pretrained weights, external large-scale datasets, or gradient information obtained from previous optimization stages. Instead, they either begin with random initialization, use architectural inductive biases, or utilize function-value queries alone. This paradigm is gaining traction in several domains—including network-reparameterized full waveform inversion, neural field reconstruction, incremental learning, and optimization—largely due to its workflow simplicity, robustness to data gaps, and ability to circumvent negative transfer commonly observed when pretraining and downstream objectives misalign.

## 1. Key Principles and Formal Definitions

Zero-pretraining refers to the complete avoidance of any pretrained parameters, dataset-dependent priors, or first-order gradient information in deep learning initialization and training. Formally, given a deep model $f_\Theta$ parametrized by $\Theta$, zero-pretraining workflow entails:
- Random initialization of $\Theta$;
- Absence of any supervised or unsupervised fitting stage before deployment;
- No incorporation of external large-scale pretraining datasets or hand-crafted regularizers;
- Training is guided strictly by the task-specific loss function.

In network-reparameterized FWI, this is realized by a denormalization operator $\mathcal{D}_2(f_\Theta(I), m_{\rm init}) = f_\Theta(I)\odot S + m_{\rm init}$, imposing the initial model only as a fixed architectural bias rather than fitting the network to it [2506.05484]. In sparse-view 3D reconstruction, all inductive bias is provided solely by generator architecture (e.g., Deep Image Prior) without pretrained features [2312.09249]. In incremental learning, task progression is managed by Adaptive Prompt Generation rather than retrieval from a pretrained pool [2308.10445].

## 2. Methodologies: Model Construction and Training Strategies

Zero-pretraining deep learning is instantiated in several research directions:

**Denorm-DRFWI for Full Waveform Inversion** [2506.05484]:
- The model learns perturbations about a fixed initial model $m_{\rm init}$, eliminating any pretraining or fine-tuning stage.
- Training proceeds via a single-stage physics-guided inversion loss:
  $$
  \mathcal{L}(\Theta) = \frac12\left\|d_{\mathrm{obs}} - {\cal M}\odot{\cal F}\left(f_\Theta(I)\odot S + m_{\rm init}\right)\right\|_F^2
  $$
- Variants include Static Denorm (non-updating $m_{\rm init}$) and Adaptive Denorm (gradient-driven updates to $m_{\rm init}$).
- Advantages over two-stage pretraining include preserved network plasticity, mitigation of spectral bias, and reduction of workflow hyperparameter sensitivity.

**ZeroRF for Sparse View 360° Reconstruction** [2312.09249]:
- Neural radiance field factors are generated via convolutional generator $G_\phi$ fed with fixed Gaussian noise, enforcing a Deep Image Prior effect.
- No hand-crafted nor dataset-trained priors are used; generator weights are randomly initialized.
- Fitting is per-scene; optimization addresses photometric reconstruction loss only.

**Incremental Learning with Adaptive Prompt Generator** [2308.10445]:
- Vision Transformer backbone is randomly initialized and trained solely on first task, then frozen.
- Task-specific prompts are produced dynamically via cross-attention to a growing pool of learned prompt candidates, not by retrieval from a static pretraining source.
- A knowledge pool regularizes the prompt generator and classifier by maintaining statistical summaries per class across tasks, ensuring resistance to catastrophic forgetting.

**Zeroth-Order Deep Optimization (DeepZero)** [2310.02025]:
- All optimization is conducted via function-value queries alone; gradients are estimated by finite-difference coordinatewise perturbations.
- Sparsity-inducing pruning techniques reduce computation by targeting coordinates with significant gradient signal.
- Feature reuse and forward parallelization techniques accelerate training time and allow scalability for deep architectures.

## 3. Empirical Performance and Comparative Results

Zero-pretraining frameworks have demonstrated state-of-the-art convergence, robustness, and fidelity across several domains:

| Task/Setting            | Method           | Acc./MSE/SSIM          | Comments                                  |
|------------------------|------------------|------------------------|-------------------------------------------|
| FWI (Marmousi, smooth) | Pretrain-DRFWI   | MSE=0.1839, SSIM=0.8016| Two-stage, highest error, slow convergence|
| FWI (Marmousi, smooth) | S-Denorm         | MSE=0.1605, SSIM=0.8268| Faster ($\sim2\times$), improved accuracy |
| FWI (Marmousi, smooth) | A-Denorm         | MSE=0.1517, SSIM=0.8363| Fastest, best accuracy                    |
| NeRF-Synthetic (4 view)| ZeroRF           | PSNR=21.94, SSIM=0.856 | Outperforms baselines in $\ll$ comp. time |
| CIFAR-100 B50-T10      | L2P (no pretrain)| 36.55%                 | Prompt-retrieval (severely degraded)      |
| CIFAR-100 B50-T10      | APG              | 66.68%                 | Prompt-generation (zero pretraining)      |
| ResNet-20 (CIFAR-10)   | DeepZero         | 86.94%                 | ZO optimization ($\approx$FO in 50 epochs)|

Denorm-DRFWI excels in both speed and high-frequency model recovery, ZeroRF provides SOTA 3D reconstruction in timeframes (minute-scale) unattainable by alternatives, APG achieves nearly double the accuracy of prompt-retrieval on sequential CIFAR-100 splits, and DeepZero delivers deep CNN convergence within $6\%$ of FO baselines, with unique applicability in black-box certified adversarial defense [2310.02025] and PDE error correction.

## 4. Architectural and Algorithmic Inductive Biases

Zero-pretraining leverages implicit architectural biases in lieu of data-dependent or handcrafted priors:

- **Deep Image Prior** [2312.09249]: Convolutional generator architectures (e.g., ResNet-style blocks, bilinear upsampling) naturally suppress high-frequency "speckle," regularizing solution search space. This suppresses noise without external regularization.
- **Denormalization Operator** [2506.05484]: Forces the network to model only the high-wavenumber perturbation. This targets frequency bias and accelerates high-frequency detail recovery.
- **Feature Reuse** [2310.02025]: In ZO optimization, intermediate network activations are cached, minimizing redundant computation for parameter perturbations.
- **Prompt Generation via Cross-Attention** [2308.10445]: Dynamically adapts prompt vectors to intermediate features, instead of static selection by cosine similarity, bridging domain gaps and distribution shifts inherent in incremental tasks.

A plausible implication is that architectural choices, such as choice of generator in ZeroRF, MLP depth and activation function (e.g., sine with $\omega\approx30$ in Denorm-DRFWI), and prompt composition in APG, directly dictate the attainable inductive bias and thus final model fidelity.

## 5. Workflow Simplification, Pitfalls, and Best Practices

Zero-pretraining eliminates mismatched objectives, hyperparameter sensitivity, and negative transfer observed in pretraining regimes [2506.05484]. Recommended protocols include:

- Single-stage training with strictly task-driven loss.
- Avoidance of additional supervised fitting; let domain constraints (physics, photometric, cross-entropy) guide solution search.
- Adam/AdamW optimization (\( \eta \sim 10^{-4} \)), batch/layer normalization for stability [2109.02752].
- Monitoring both error-based and structural metrics (e.g., $R^2$, SSIM in FWI/neural fields) for convergence and overfitting detection.
- Ensemble or test-time augmentation to mitigate data sample perturbations.
- Careful data preprocessing and augmentation for scarce, noisy, or imbalanced regimes.

A plausible implication is that, while zero-pretraining theoretically affords maximal plasticity, architectural instabilities (e.g., badly calibrated initialization, lack of inductive bias) must still be constrained through thoughtful design and regularization-by-architecture.

## 6. Applications, Extensions, and Limitations

Zero-pretraining methods have been validated in:

- Unsupervised physics inversion (FWI) with robust recovery from inaccurate starters.
- Sparse-view 3D reconstruction and editing, mesh texturing, and text/image-to-3D applications [2312.09249].
- Black-box optimization and certified model robustness against adversarial attacks (DeepZero).
- Class-incremental learning with domain-gap resilience [2308.10445].

Limitations include propagation of structural grid biases (ZeroRF), potential architectural instability in unbounded domains (ZeroRF), and requirement for strong task-specific inductive biases in the absence of external priors. In particular, for incremental learning, APG’s effectiveness stems from its dynamic prompt generation by cross-attention, whereas standard prompt retrieval collapses when pretraining-domain alignment is poor.

## 7. Context and Future Directions

Zero-pretraining is not merely an engineering convenience—it offers a systematic avenue for model robustness, plasticity, and rapid deployment when external data or gradients are unavailable or unreliable. Workflows such as denormalization-based parameterization, Deep Image Prior generators, cross-attentive prompt generators, and ZO finite-difference methods all represent concrete operationalizations of this paradigm.

Research directions include further scalability of zeroth-order optimization for large-scale models [2310.02025], architectural adaptations for unbounded or highly non-Euclidean domains, and new forms of regularization-by-structure that further reduce reliance on external data. In incremental and continual learning, adaptive architectures such as APG may continue to close the distribution-gap previously untraversable without strong pretraining.

Zero-pretraining approaches mark a foundational shift toward direct, architecture-driven learning, circumventing the pitfalls and mismatches introduced by prior-dependent deep learning workflows.

Source: https://www.emergentmind.com/topics/zero-pretraining-deep-learning-methods