Task-Dependent Initialization (TDI)

• TDI is a family of strategies that derive initialization parameters from target-task data, prior task modules, or feature statistics instead of random assignment.
• Methods include closed-form solutions for logistic regression, representation similarity for model selection, and adapter or soft-prompt transfers in continual learning.
• Empirical results show TDI accelerates convergence, reduces overfitting, and improves metrics in tasks ranging from image recognition to state space modeling.

Task-Dependent Initialization (TDI) denotes a family of initialization strategies in which model parameters, prompts, adapters, or the choice of a pre-trained source model are set using information from the target task, target-task data, or previously learned task-specific parameters, rather than by task-agnostic randomization. In the literature summarized here, TDI appears as a closed-form initializer for multinomial logistic regression in transfer learning (Cheng et al., 2018), a representation-similarity criterion for selecting source models and layers (Dwivedi et al., 2020), an adapter-initialization procedure based on distilling prior task adapters (Srinivasan et al., 2023), a soft-prompt transfer mechanism based on task prompt vectors (Belanec et al., 2024), and a power-spectrum-matching method that aligns the inductive bias of state space models with task spectra (Chen et al., 25 Sep 2025). Across these settings, the recurring objective is to begin optimization from an initialization that is closer to a useful solution, more trainable, or better aligned with the task structure.

1. Conceptual scope and variants

TDI is not a single algorithmic template. The papers instead instantiate a common principle: the initialization should depend on a task-relevant signal that is available before ordinary end-to-end training. That signal may be labeled feature statistics, target-task images, previously trained task modules, or an estimated task spectrum. A nearby but distinct regime is represented by automatic critical initialization methods that are architecture-dependent but task-agnostic. In particular, AutoInit explicitly contrasts itself with TDI, defining TDI as schemes that train a network on the target task to discover a good initialization, often requiring a full training/evaluation loop multiple times (He et al., 2022).

A compact way to organize the literature is by the source of task information and the object being initialized.

Setting	Source of task information	Object of initialization or selection
RGC for classification	Class means and shared covariance of target-task features	Last linear layer
DDS for transfer learning	Target-task images and deep representations	Source model or layer
I2I for continual learning	Previously learned adapters and new-task data	Incoming task adapter
Task Prompt Vectors	Tuned soft-prompts from source tasks	New soft-prompt initialization
Power spectrum matching for SSMs	Estimated task spectrum	SSM parameters before full training

This taxonomy suggests that “task dependence” can enter at different levels of abstraction. In some formulations, it means directly computing parameter values from target-task statistics; in others, it means selecting among available initializations; in still others, it means composing information from previously solved tasks into a new initialization (Cheng et al., 2018, Dwivedi et al., 2020, Srinivasan et al., 2023, Belanec et al., 2024, Chen et al., 25 Sep 2025).

2. Closed-form data-dependent initialization for discriminative heads

A canonical TDI instance is the initialization of the multinomial logistic regression layer that typically forms the last stage of a classifier. The paper “Revisit Multinomial Logistic Regression in Deep Learning: Data Dependent Model Initialization for Image Recognition” studies the last layer of a DNN, modeled as a fully connected layer followed by softmax and cross-entropy loss, and argues that random initialization is especially suboptimal in transfer learning, where the feature extractor is pre-trained but the task-specific last layer is newly introduced (Cheng et al., 2018).

The proposed initializer is a closed-form approximate solution named the regularized Gaussian classifier (RGC). The construction begins by extracting penultimate-layer features $\mathbf{x}_i$ from the pre-trained network. For each class $k$, the class mean is

$$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$

and the shared covariance is

$$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$

To stabilize the covariance in few-sample or high-dimensional regimes, the paper adds regularization,

$\Sigma_{\text{reg}} = \Sigma + \epsilon I,$

and sets the classifier parameters to

$$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$

In practice, the recommended implementation solves $(\Sigma + \epsilon I)\mathbf{w}_k = \mu_k$ rather than explicitly inverting the matrix. The paper also describes an optional calibration step,

$$\hat{\mathbf{w}}_k = \alpha \mathbf{w}_k + v,\qquad \hat{b}_k = \alpha b_k + \beta,$$

with $\alpha$, $v$, and $k$0 chosen so that the first and second moments match those of the original classifier weights.

The motivation is that features from pre-trained DNNs are often approximately linearly separable, so a Gaussian-based linear classifier can serve as a practical proxy for the eventual logistic-regression optimum. The paper attributes several deficiencies to random initialization in this setting: slow convergence speed, the possibility of becoming stuck in local minimum, and the risk of over-fitting (Cheng et al., 2018).

The reported empirical effects are substantial. For image classification, the method can reduce the training time by 10 times and achieve a 3.2% gain in accuracy for Flickr-style classification. On the Flickr-style dataset with AlexNet, the reported accuracy after initialization is 37.96% for RGC and 0% for random initialization; after 3,000 iterations, RGC reaches 39.2%, matching the random-initialized baseline at 100,000 iterations; after 10,000 iterations, RGC reaches 42.39%. On Oxford Flower-102 and Caltech-256, RGC achieves higher accuracy in 25–50% of the iterations required for random initialization. For object detection, the reported improvements include 67% mAP after 1000 iterations for YOLO v2 on VOC2007 versus a baseline requiring 10,000 iterations, with final mAP 5% higher; 60% mAP in 2,000 iterations for Faster R-CNN on VOC2007 versus a baseline’s 35,000 iterations, with final mAP 4% higher; and a consistent 2x–4x speedup with 2–2.4% higher final mAP on Flickr-Logo32. The paper further reports a smaller gap between training and test loss, which it interprets as reduced overfitting (Cheng et al., 2018).

Within the broader TDI landscape, RGC is notable because it computes parameters directly from target-task feature statistics, rather than merely selecting among pre-trained alternatives. It is therefore a strong example of TDI as a closed-form, data-dependent parameter construction.

3. Initialization selection in transfer learning

A second TDI formulation treats initialization as a selection problem: given multiple pre-trained models, which one should be used as the starting point for a new task? “Duality Diagram Similarity: a generic framework for initialization selection in task transfer learning” addresses this question by ranking source models through representation similarity, without retraining them on the new task (Dwivedi et al., 2020).

The method represents features from a source model and a target task as matrices $k$1 and $k$2, and associates each with a duality diagram. Features are transformed as

$k$3

after which pairwise similarity or distance matrices are computed,

$k$4

and finally compared by

$k$5

The source model with the highest similarity score is predicted to transfer best. The framework is explicitly described as a generalization of representational similarity analysis and centered kernel alignment (Dwivedi et al., 2020).

The practical workflow uses a small set of images from the target task. These images are fed through each candidate source model and through a task-specific target model to obtain features, and the resulting similarity scores induce a ranking of source models. The same machinery can be applied at the layer level, so DDS can also select the layer location within a DNN from which transfer should branch (Dwivedi et al., 2020).

The paper reports high correspondence between predicted rankings and actual transfer-learning rankings on Taskonomy. Computing DDS for all $k$6 transfers requires less than 2 minutes and yields a mean Spearman correlation of $k$7, outperforming the prior best by 10–12% absolute improvement. The reported previous best correlations are RSA at $k$8, DeepLIFT at $k$9, and saliency at $$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$0. The computational asymmetry with brute-force fine-tuning is pronounced: the paper contrasts under-2-minute DDS computation with $$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$1 seconds, approximately 4,500 GPU hours, for actual transfer learning on the same grid (Dwivedi et al., 2020).

The method is also reported to be data-dependent in a strong sense: high correlation on Pascal VOC semantic segmentation is achieved only when the images used to compute DDS are sampled from the same distribution as the target task. For Pascal VOC images, the reported correlations are $$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$2 with cosine and $$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$3 with Laplacian similarity, whereas using NYUv2 or Taskonomy images yields lower correlations. For layer selection in a ResNet50 encoder, DDS identifies block 1 as matched to 2D tasks, block 3 to 3D tasks, and block 4 to semantic tasks; the paper states that the highest transfer performance is achieved at the layers predicted by DDS (Dwivedi et al., 2020).

This line of work broadens the meaning of TDI. The initialization is not synthesized from scratch, but selected by a task-dependent criterion derived from the target-task representation geometry.

4. Continual learning, adapters, and soft-prompt transfer

In continual learning and parameter-efficient fine-tuning, TDI often applies not to the full model but to small task modules. “I2I: Initializing Adapters with Improvised Knowledge” studies frozen-backbone transformers with task-specific adapters, arguing that training independent adapters for each task prevents cross-task knowledge transfer (Srinivasan et al., 2023).

I2I introduces a three-phase procedure for a new task $$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$4. In the “Improvise” phase, prior adapters $$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$5 are fused through a trainable fusion layer $$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$6, and $$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$7 together with a new task-specific head $$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$8 is trained on the new task data $$\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} \mathbf{x}_i,$$9:

$$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$0

For $$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$1, the paper uses the single previous adapter without fusion. In the “Initialize” phase, a new adapter $$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$2 is trained to mimic the representations of the fusion model by minimizing a distillation loss

$$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$3

with

$$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$4

For $$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$5, adapter parameters are directly copied from the previous task. In the final phase, the new adapter and head are fine-tuned on the task:

$$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$6

After distillation, the temporary fusion layer is discarded, so inference retains only ordinary adapter parameters (Srinivasan et al., 2023).

The experiments use CLiMB, specifically a five-task sequence of visual question answering tasks—VQAv2, Visual7W, VQA-Abstract, VizWiz, and DAQUAR—with a CLIP-BART model in which CLIP is a frozen image encoder and BART is a frozen text encoder-decoder with adapters in BART layers. The reported knowledge-transfer score is

$$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$7

On this benchmark, I2I-FF reports 4.03 overall transfer, I2I-FL 1.94, I2I-LL 1.17, ClosestTaskInit 1.90, AdapterFusion $$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$8, and vanilla adapters 0. The paper further states that AdapterFusion can exceed 20–40% of the base transformer size for each task, whereas I2I discards the fusion layer and avoids this inference-time parametric cost (Srinivasan et al., 2023).

A related prompt-based formulation appears in “Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer” (Belanec et al., 2024). Here, the initialized object is a soft prompt. For a task $$\Sigma = \frac{1}{N} \sum_{k=1}^K \sum_{i\in C_k} (\mathbf{x}_i - \mu_k)(\mathbf{x}_i - \mu_k)^T.$$9, the task prompt vector is defined as the element-wise difference between tuned soft-prompt weights and their random initialization:

$\Sigma_{\text{reg}} = \Sigma + \epsilon I,$0

A new initialization is obtained by

$\Sigma_{\text{reg}} = \Sigma + \epsilon I,$1

with $\Sigma_{\text{reg}} = \Sigma + \epsilon I,$2, and multiple task prompt vectors can be added,

$\Sigma_{\text{reg}} = \Sigma + \epsilon I,$3

The experiments use T5-base on 12 NLU datasets across natural language inference, topic classification, and sentiment analysis, with soft prompts of length 100 tokens and size $\Sigma_{\text{reg}} = \Sigma + \epsilon I,$4. The paper reports that task prompt vectors are independent of the random initialization of prompt tuning on 2 different LLM architectures. Example numbers given for random-initialization independence are QNLI at 93.3% versus 93.2% and MNLI at 85.4% versus 85.3%. For multi-task compositional transfer, the paper reports a SciTail F1 of 54.9% with random initialization, 70.4% with MNLI SPoT, and 71.5% with the QNLI+MNLI task-prompt-vector sum, described as a statistically significant improvement. It also states that adding two task prompt vectors usually preserves more than 90% task accuracy on both source tasks (Belanec et al., 2024).

These adapter and prompt methods show that TDI can be highly modular. Rather than reinitializing or retraining the full backbone, they initialize compact task-specific components by reusing structure from earlier tasks.

5. Trainability, criticality, and the boundary of TDI

A theoretical motivation for TDI comes from trainability analysis. “Trainability of ReLU networks and Data-dependent Initialization” studies dead ReLU neurons, distinguishing tentative death from permanent death, and defines trainability as the probability that the number of permanently dead neurons is sufficiently small for a learning task (Shin et al., 2019). The paper states that successful training is only possible if the network is trainable and that trainability serves as an upper bound on successful training rates.

The proposed data-dependent initialization is designed so that, for each neuron,

$\Sigma_{\text{reg}} = \Sigma + \epsilon I,$5

The paper describes informal constructions such as drawing $\Sigma_{\text{reg}} = \Sigma + \epsilon I,$6 and setting $\Sigma_{\text{reg}} = \Sigma + \epsilon I,$7 for some $\Sigma_{\text{reg}} = \Sigma + \epsilon I,$8 and small $\Sigma_{\text{reg}} = \Sigma + \epsilon I,$9, or using rejection sampling until at least one training sample activates the neuron. In the summary provided, this is interpreted as guaranteeing that each neuron is tentatively active and making the probability of permanent death zero at initialization, unless the data are degenerate (Shin et al., 2019).

The same paper links initialization to over-parameterization. Over-parameterization is stated to be both necessary and sufficient for minimizing the training loss, because with enough neurons the probability that all neurons are dead decays and trainability approaches one. This suggests a general TDI principle: using target-data geometry at initialization can substitute, at least partially, for brute-force reliance on over-parameterization (Shin et al., 2019).

A useful boundary case is AutoInit, which is not TDI in the ordinary task-dependent sense but clarifies what TDI is not. AutoInit tunes auxiliary scale parameters $$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$0 so that the Average Partial Jacobian Norm between adjacent blocks is close to unity, thereby achieving critical initialization for feed-forward DNNs (He et al., 2022). Its preferred objective is the Jacobian Log Loss,

$$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$1

with an optional Jacobian-Kernel Log Loss that also controls forward signal variance. The main weights are frozen during tuning, and only the scale factors are optimized; the paper states that a small fixed number of steps, approximately 500, suffices. It reports that Jacobian log loss is faster and more robust than Jacobian square loss, that APJN stabilizes for batch sizes $$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$2 with BatchNorm, and that on CIFAR-10 AutoInit yields training and validation accuracy on par with original ResMLP manual initialization, while on ImageNet with ResMLP-S12, 15M parameters, it yields 74% top-1 accuracy versus a 76.6% reference under a different training regimen (He et al., 2022).

The contrast is conceptually important. AutoInit is task-agnostic and uses architecture and random inputs rather than task labels or target-task statistics. The paper explicitly describes it as a zero-shot, architecture-dependent counterpart to TDI (He et al., 2022). This indicates that the literature distinguishes between initialization aligned to optimization dynamics in general and initialization aligned to the particular structure of a task.

6. Spectral alignment and inductive bias in state space models

A recent formulation places TDI at the level of inductive bias rather than local parameter heuristics. “Aligning Inductive Bias for Data-Efficient Generalization in State Space Models” formalizes the inductive bias of linear time-invariant state space models through an SSM-induced kernel and then proposes TDI by power spectrum matching (Chen et al., 25 Sep 2025).

For sequences of length $$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$3, the SSM-induced kernel is defined as

$$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$4

where $$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$5 is the Toeplitz matrix associated with the SSM impulse response. If $$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$6, then the kernel eigenvalues are $$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$7. The paper’s critical spectral claim is that, as $$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$8, the singular values correspond directly to the magnitude of the frequency response,

$$\mathbf{w}_k = \Sigma_\text{reg}^{-1} \mu_k,\qquad b_k = -\frac{1}{2} \mathbf{w}_k^T \mu_k.$$9

This means the inductive bias of the SSM is a frequency bias: frequencies with large $(\Sigma + \epsilon I)\mathbf{w}_k = \mu_k$0 correspond to dominant kernel modes and are learned most readily (Chen et al., 25 Sep 2025).

The TDI algorithm estimates the task spectrum through the input-output cross-power spectral density,

$(\Sigma + \epsilon I)\mathbf{w}_k = \mu_k$1

where $(\Sigma + \epsilon I)\mathbf{w}_k = \mu_k$2 is the empirical cross-covariance matrix. It then compares this task spectrum to the model spectrum $(\Sigma + \epsilon I)\mathbf{w}_k = \mu_k$3 using the normalized loss

$(\Sigma + \epsilon I)\mathbf{w}_k = \mu_k$4

and applies gradient descent, such as Adam, to update $(\Sigma + \epsilon I)\mathbf{w}_k = \mu_k$5 before full task training. The theoretical argument is that minimizing $(\Sigma + \epsilon I)\mathbf{w}_k = \mu_k$6 increases the cumulative power

$(\Sigma + \epsilon I)\mathbf{w}_k = \mu_k$7

thereby shifting more target-function energy onto top kernel modes and improving generalization from fewer samples (Chen et al., 25 Sep 2025).

The empirical pattern is explicitly conditional. On synthetic tasks, when the SSM’s initial bias already matches the task spectrum, power spectrum matching produces little change and no generalization benefit; when there is mismatch, especially on high-frequency tasks, it yields a faster rise in cumulative power and lower generalization error in data-scarce regimes. On real-world benchmarks using deep SSMs such as S4D, the paper reports significant improvements in generalization on Pathfinder and Speech Commands, particularly in low-data regimes, and marginal benefits on CIFAR-10 and ListOps, where the default bias already aligns well with the task. The paper also states that TDI helps reduce overfitting. At the same time, it limits the present theory to LTI SSMs, notes that experiments applied TDI only to the first layer of deep SSM networks, and identifies extensions to input-dependent or selective SSMs such as Mamba, as well as to higher-dimensional input spaces, as future work (Chen et al., 25 Sep 2025).

This spectral formulation makes explicit a broader interpretation of TDI: initialization can be understood as pre-aligning a model’s inductive bias with measurable structure in the task. A plausible implication is that TDI is most useful not when optimization is merely difficult, but when the model’s default prior is measurably mismatched to the task.