Soft Injection of Task Embeddings

• The paper demonstrates that injecting low-dimensional task embeddings via adapters and activation-space interpolation enhances multitask model performance without full fine-tuning.
• It details diverse methodologies including learned, predicted, and retrieval-based embeddings across NLP, audio, and diffusion models to condition computations.
• Empirical results show improved accuracy and efficiency, with optimized injection loci proving crucial for maintaining task-specific activation in frozen backbones.

Searching arXiv for the specified papers and closely related work on soft task embedding injection. arxiv_search(query="(Moon et al., 5 Mar 2026)", max_results=5) Soft injection of task embeddings denotes a family of conditioning strategies in which task information is encoded as continuous vectors and introduced into a model’s computation without full parameter updating or long discrete prompts. Across NLP multitask transformers, general-purpose LLMs, neural audio LLMs, and diffusion-based speech enhancement, the central pattern is to represent task structure, demonstrations, or degradation profiles as low-dimensional embeddings and to inject those embeddings into adapters, attention-head activations, token streams, or learned timestep embeddings so that downstream computation becomes task-aware (Sileo et al., 2021). The technical significance of the paradigm lies in the choice of injection locus: the cited works show that continuous conditioning can be beneficial when it is propagated through relevant depth or modules, but can underperform or be diluted when it is attached only at the input (Moon et al., 5 Mar 2026).

1. Conceptual scope and problem setting

In the cited literature, “task embeddings” are low-dimensional representations trained or constructed to capture task properties, while “soft injection” refers to mixing or conditioning with continuous vectors rather than using only discrete demonstrations or separate per-task model copies (Sileo et al., 2021). Sileo and Moens formulate task embeddings as learned vectors $z_i \in \mathbb{R}^{d_z}$ attached to tasks in a 101-task multitask benchmark, with the frozen pretrained transformer modulated by conditional adapters rather than full fine-tuning. SITE reinterprets the same general idea at activation level: task information is first distilled from a small set of few-shot ICL prompts into attention-head outputs and is then injected into a frozen LLM through learned per-head interpolation weights, eliminating the need for in-prompt demonstrations at inference (Park et al., 28 Jul 2025).

The same conditioning logic appears outside text. LiSTEN uses a prompt-pool of learnable key-value pairs to retrieve soft token embeddings for audio-language tasks and prepends the selected embeddings to the LLM input sequence (Mousavi et al., 24 May 2025). SLICE uses a frozen WavLM-Base encoder plus three specialized heads to construct a unified degradation embedding and adds it to the timestep embedding of an NCSN++ score network so that every residual block becomes degradation-aware without architectural changes (Moon et al., 5 Mar 2026). Taken together, these papers define soft injection not as a single algorithm but as a design family centered on continuous task-conditioned modulation.

2. Task-embedding construction across modalities

The construction of the embedding varies with the problem definition. In MetaEval, each of the 101 tasks carries its own 32-dimensional vector, with $d_z = 32$, and the vectors are jointly learned with conditional-adapter parameters and per-task classifier heads while the base transformer remains frozen (Sileo et al., 2021). The same work also describes prediction of embeddings for unseen tasks from task metadata such as domain cluster, number of training examples, number of text fields, median input length, and task type. This allows zero-shot inference through a predicted $\hat z_t$ rather than a learned task-specific vector.

SITE constructs its task embedding from few-shot prompts rather than an explicit task identifier. Given $M$ distinct 10-shot prompts, it runs each prompt once through the frozen LLM, extracts every attention head’s output at the last token, and averages across prompts to obtain $t^{(l,h)} \in \mathbb{R}^{d_v}$ for each layer-head pair (Park et al., 28 Jul 2025). The resulting object is not a single global vector but a structured collection $\{t^{(l,h)}\}_{l,h}$ aligned to the model’s internal head layout.

LiSTEN uses a prompt-pool size $P = 400$ with learnable keys $K \in \mathbb{R}^{P \times d}$ and values $V \in \mathbb{R}^{P \times d'}$, initialized at random and updated end-to-end (Mousavi et al., 24 May 2025). A query vector $q$ is formed by mean-pooling token embeddings that include both audio-derived tokens from the Q-Former and task instruction tokens from the LLM tokenizer. Prompt values are then retrieved by similarity-based, attention-based, or residual-based selection, optionally with stochastic prompt length.

SLICE constructs a multi-degradation embedding from a degraded waveform using WavLM-Base as a frozen feature extractor. Frame-level features $d_z = 32$0 with $d_z = 32$1 are projected and temporally pooled into $d_z = 32$2 with $d_z = 32$3, then mapped by three auxiliary heads into $d_z = 32$4 for noise, reverberation, and distortion, with $d_z = 32$5 (Moon et al., 5 Mar 2026). At inference, those head outputs are combined by a small MLP into a unified condition vector $d_z = 32$6 with $d_z = 32$7.

3. Injection loci and architectural realizations

The architectural locus of injection is the major axis of variation in this literature.

Method	Embedding source	Injection point
MetaEval	Learned or predicted task vector $d_z = 32$8	Conditional adapter in each transformer layer
SITE	Averaged head outputs from 10-shot prompts	Last-token attention-head activations
LiSTEN	Retrieved prompt values from key-value pool	Prepended token embeddings before layer 1
SLICE	Unified degradation embedding $d_z = 32$9	Timestep embedding of NCSN++

In MetaEval, the higher-performing parameterization described from Pilault et al. 2021 applies FiLM-style modulation inside a conditional adapter:

$\hat z_t$0

An alternative formulation, following Mahabadi et al. 2021, generates LayerNorm gains and biases from $\hat z_t$1 for an adapter MLP (Sileo et al., 2021). In both cases, the task embedding modulates every transformer layer through the adapter stack.

SITE performs direct activation-space interpolation. For head $\hat z_t$2 in layer $\hat z_t$3 at token position $\hat z_t$4,

$\hat z_t$5

with one scalar $\hat z_t$6 per head across all layers and intervention at the last token of the zero-shot prompt (Park et al., 28 Jul 2025). This shifts conditioning from prompt space to activation space.

LiSTEN uses a more conventional prompt-tuning locus. If $\hat z_t$7 are ordinary token embeddings and $\hat z_t$8 are selected prompt embeddings, the actual input is

$\hat z_t$9

No other layers are modified (Mousavi et al., 24 May 2025). This is still a soft injection scheme because the injected objects are learned continuous embeddings rather than discrete exemplars.

SLICE replaces the native timestep embedding $M$0 by

$M$1

Because the original NCSN++ backbone already adds the timestep embedding to features in every one of its approximately 37 residual blocks, the added degradation vector propagates through all residual blocks without additional FiLM layers or per-layer gates (Moon et al., 5 Mar 2026). The paper contrasts this with input-level conditioning, which simply adds $M$2 to the input spectrogram and is described as architecturally intrusive and progressively diluted over dozens of blocks.

4. Optimization regimes and inference procedures

The training regimes are likewise heterogeneous but share a frozen-backbone logic. In MetaEval, the base transformer is frozen; only the conditional-adapter parameters, the $M$3 values, and the per-task classifier heads are trained. Tasks are sampled with probability proportional to $M$4, and each task’s cross-entropy loss is capped at $M$5 to keep gradients balanced (Sileo et al., 2021). For unseen tasks, embeddings can be produced by offline regression $M$6 or by the feature-aware formulation $M$7.

SITE freezes the LLM and optimizes only the head-selection parameters. Each $M$8 is parameterized as $M$9, with $t^{(l,h)} \in \mathbb{R}^{d_v}$0 initialized to $t^{(l,h)} \in \mathbb{R}^{d_v}$1, and Adam with learning rate $t^{(l,h)} \in \mathbb{R}^{d_v}$2 is run for 400 steps on next-token cross-entropy over a small validation set (Park et al., 28 Jul 2025). After optimization, inference stores the task embeddings $t^{(l,h)} \in \mathbb{R}^{d_v}$3 and the optimized $t^{(l,h)} \in \mathbb{R}^{d_v}$4 and performs one interpolation step at the last prompt token, or once at the beginning if KV-cache is enabled.

LiSTEN updates the Q-Former and the key-value pool while keeping the LLM frozen. Training uses the standard next-token loss plus an auxiliary key-loss, with the exact form depending on whether selection is similarity-based, attention-based, or residual-based (Mousavi et al., 24 May 2025). Inference can choose prompt length $t^{(l,h)} \in \mathbb{R}^{d_v}$5 to trade off compute versus accuracy.

SLICE trains the speech enhancement objective jointly with auxiliary losses for the three degradation heads:

$t^{(l,h)} \in \mathbb{R}^{d_v}$6

with $t^{(l,h)} \in \mathbb{R}^{d_v}$7 (Moon et al., 5 Mar 2026). At inference, the auxiliary heads are discarded after producing the unified condition vector. The conditioning path itself remains a single vector addition into the existing timestep-embedding path.

5. Empirical behavior, gains, and failure modes

The empirical results show that soft injection is not merely a compression of prompting or fine-tuning; in several settings it changes the attainable performance regime. On MetaEval, conditional adapters with learned $t^{(l,h)} \in \mathbb{R}^{d_v}$8 reach 79.7% unweighted average accuracy on held-out test splits, compared with 76.9% for full fine-tuning each task separately, 67.8% for single-task adapters, and 75.6% for conditional adapters in the Mahabadi et al. formulation (Sileo et al., 2021). On held-out GLUE tasks, offline regression from task aspects to $t^{(l,h)} \in \mathbb{R}^{d_v}$9 reaches 70.2% average accuracy and the jointly learned feature-aware formulation reaches 70.4%, versus 69.4% for fine-tuning on same-type MetaEval tasks only.

SITE reports an extensive evaluation across 57 tasks and 12 LLMs spanning four model families from 4B to 70B, with average gains over 10-shot ICL of 10.2%–14.3% across 12 LLMs (Park et al., 28 Jul 2025). The condensed summary reports, for example, 76.7% to 90.0% on Llama-3.1-8B, 72.0% to 85.5% on Qwen3-32B, and 79.5% to 89.5% on Llama-3.1-70B. The same paper also reports lower runtime and memory than 10-shot ICL on Llama-3.1-8B over 10 k test prompts on AG_News: 248.4 min for 10-shot ICL versus 172.0 min for SITE, with memory figures of approximately 4 MB for the 10-shot KV-cache versus approximately 0.5 MB added by SITE.

SLICE provides a particularly explicit failure case for poorly placed conditioning. In a controlled ablation on a 2,472-file multi-degradation test set, “No encoder” yields PESQ 2.60, ESTOI 0.77, SI-SDR 2.3 dB, and UTMOS 3.70; “Input-add” yields 2.49, 0.73, 1.4 dB, and 3.62; and SLICE yields 2.60, 0.80, 3.7 dB, and 3.71 (Moon et al., 5 Mar 2026). The paper states that input-addition hurts over no encoder at all, while layer-wise injection achieves the best results. On standard noise-only test data of 824 utterances, SLICE matches the unconditioned SGMSE+ in SDR at 17.4 dB and PESQ at approximately 2.83, while outperforming in UTMOS at 3.93 versus 3.83.

LiSTEN’s reported ablations emphasize prompt selection rather than absolute task-level scores. The authors settle on $\{t^{(l,h)}\}_{l,h}$0 and $\{t^{(l,h)}\}_{l,h}$1 for most experiments, report that even $\{t^{(l,h)}\}_{l,h}$2 with stochastic training still outperforms LoRA, and state that similarity-based DPS gives the best or second-best score on 6/8 metrics, with residual-based retrieval second and attention-based retrieval weaker (Mousavi et al., 24 May 2025).

6. Interpretability, transfer structure, and broader implications

A recurring result is that soft injection exposes task structure in the model’s internal organization. MetaEval treats the learned task embeddings as an object of analysis in their own right, arguing that the resulting embedding space enables analysis of the space of tasks and that task aspects can be mapped to task embeddings for new tasks without annotated examples (Sileo et al., 2021). This suggests that task embeddings can function simultaneously as control signals and as latent summaries of task similarity.

SITE makes this interpretability claim more mechanistic. After optimization, roughly 80–90% of heads have $\{t^{(l,h)}\}_{l,h}$3 or $\{t^{(l,h)}\}_{l,h}$4, while a smaller fraction retain intermediate values; binarizing $\{t^{(l,h)}\}_{l,h}$5 at $\{t^{(l,h)}\}_{l,h}$6 still yields approximately 88.9% average accuracy versus 90.0% for the soft version (Park et al., 28 Jul 2025). Cross-task transfer of the $\{t^{(l,h)}\}_{l,h}$7 matrix is reported to remain high for similar tasks but to collapse for dissimilar ones, and a single shared task-agnostic $\{t^{(l,h)}\}_{l,h}$8 across all 57 tasks drops performance to approximately 75.3% average versus 90.0% for task-specific selection. The paper therefore argues that each task’s best injection locations are unique.

LiSTEN reports an analogous pattern in prompt space: when each test instance picks $\{t^{(l,h)}\}_{l,h}$9 prompts, the total distinct tokens used across the test set is very large, and similar tasks such as ASR and QA share many of the same prompt indices whereas dissimilar tasks such as Audio Captioning and Emotion Recognition use largely disjoint sets (Mousavi et al., 24 May 2025). SLICE, in turn, generalizes the pattern beyond “task” narrowly construed: it shows that a soft embedding of co-occurring degradations can be injected into an existing learned embedding path and thereby propagated to every residual block without changing the backbone (Moon et al., 5 Mar 2026).

A plausible implication is that the decisive design variable is not only the content of the conditioning vector but also the depth-wise route by which it enters the model. The cited works collectively support three related conclusions: continuous task embeddings can be predicted, retrieved, or distilled; the injection point determines whether conditioning is preserved, diluted, or task-specific; and soft injection can reduce reliance on long prompts or intrusive architecture changes while retaining a frozen or nearly frozen backbone (Park et al., 28 Jul 2025).